Lazy allocating sbcl, part II

After working on this in my spare time for a long time, I have something that almost works. The lazy sbcl 0.8.20.5 patches will with the following customize-target-features.lisp: :sb-lazy-linkage-table-space, :sb-lazy-dynamic-space, :sb-lazy-read-only-space, :sb-lazy-static-space, :sb-linkage-table-space. They change the specific allocations for that space so the memory is only allocated when it is needed. The net effect is:

$ diff -u /proc/13546/status /proc/13598/status
--- /proc/13546/status  2005-03-18 07:14:16.085383120 +0100
+++ /proc/13598/status  2005-03-18 07:14:16.085383120 +0100
@@ -1,18 +1,18 @@
 Name:  sbcl
 State: T (stopped)
 SleepAVG:      88%
-Tgid:  13546
-Pid:   13546
+Tgid:  13598
+Pid:   13598
 PPid:  8211
 TracerPid:     0
 Uid:   1000    1000    1000    1000
 Gid:   1000    1000    1000    1000
 FDSize:        256
 Groups:        4 6 20 24 27 29 44 109 111 114 1000
-VmSize:          886848 kB
+VmSize:           33624 kB
 VmLck:        0 kB
-VmRSS:     9160 kB
-VmData:          860116 kB
+VmRSS:     8804 kB
+VmData:            6900 kB
 VmStk:        8 kB
 VmExe:      104 kB
 VmLib:     1416 kB
@@ -22,7 +22,7 @@
 ShdPnd:        0000000000000000
 SigBlk:        0000000000000000
 SigIgn:        0000000000000000
-SigCgt:        0000000b000034fe
+SigCgt:        0000000f000034fe
 CapInh:        0000000000000000
 CapPrm:        0000000000000000
 CapEff:        0000000000000000

There are 2 gotchas:

• The patch is at the wrong place (linux-os.c) because there is no abstraction layer for 'get memory' and 'handle memory fault'.
• There is voodoo going on with the dynamic space: if I just allocate one page on every fault the system gets strange errors like 'file is not there' when renaming. If I allocate 10 pages on every fault (like in the patch) it works. This seems to be that sometimes the system needs the memory without first touching it. With the cmucl lazy patches this was due to system calls pointing to unallocated memory, but this does not seems the case here. I should investigate this further.

The good news is that the system build itself a dozen times without problems, and passes all the tests I've thrown it.

So Alexander I would translate "Beutelratte" as "pack rat".

new sbcl package

I've uploaded a split sbcl package. There is now a sbcl-common package that contains the architecture-independent parts of sbcl. You can grab a copy at the new-sbcl directory on people.

What a difference a few days make

Before

No. Maintainer  Source Binary Active Bugs Bug Rate   Archived Bugs Resolve Rate
263 Peter Van Eynde   9(263) 20(209) 24(387)    2.667(693) 230(209)      90.551%(291)

After

No. Maintainer  Source Binary Active Bugs Bug Rate   Archived Bugs Resolve Rate
6 Peter Van Eynde  78(6)  95(15)  32(309)   0.41(1170) 365(126)    91.94%(254)

In other news, a few people interested in packaging Common Lisp stuff for debian came out of the woodwork. I created the cl-debian mailing list to coordinate any work.

Interesting future packages?

While investigating the packages I adopted. I ran across a few interesting unpackaged ones.

• Common Lisp GNU Scientific Library
• elephant: an object database for Common Lisp
• AspectL is a library that provides aspect-oriented extensions for Common Lisp / CLOS
• CL-PREVALENCE is an implementation of Object Prevalence for Common Lisp
• A high-level Common Lisp framework for parallel and distributed programming.
• A Portable Common Lisp Localization Package
• A System Management Substrate for Common Lisp
• UnCommon Web is a Common Lisp web application development framework.

But before I package yet another set of Common Lisp libraries I would like to know if there would be users of these packages. If so, speak up.

I'm insane

I agreed to adopt 70 packages from kmr. Most of them are unlikely to ever get updated again, but just rebuilding them will already take quite a while.

I'm still uncertain if I should go to LSM2005 or Debconf-5, I'm leaning towards LSM2005 because it is in car-range and I would like to meet the other lispers. But their website is rather bad... On the other hand: Helsinki is rather nice...

fosdem 2005

I could manage to catch a few debian talks at fosdem today. As traditional Matteo had a bad evening and just after I left there was another little crisis. Still I managed to see 'Porting to Debian GNU variants: Targetting POSIX instead of GNU/Linux' which was interesting to see how people deal with a slightly broken standard that is not ANSI X3J13. Then the composition of the audience changed for 'The Debian Women Project'. It was a nice talk that gave some interesting ideas and opinions about a movement that seems to deserve my full support. Then came 'Free as in DFSG: an exploration of Debian's views on free software'. I actually feared this talk, maybe due to the harshness of debian-legal. But in the end it turned out that in real life debian developers seem less incendiary then in email and I think a fairly balanced view was presented. This gives me some degree of hope that we will exit the current issues without traumatic events.
Then I joined an old colleague for a talk about creating a stable 2.6 kernel from Alan Cox. This was an interesting talk, not only because he is a good speaker and liked the subject, but mainly because it confirmed my general idea that the new 2.6.x kernels are not for general consumption anymore. The kernel developers depend on the fact that most people will use kernels from distributions. This is not good as it will increase the fragmentation of the kernel. It also means that the debian-kernel group is getting more and more important.