[9fans] lisp

[9fans] lisp

[Bakul Shah]

[Questions in the third para below.]
CMUCL "initializes" its state essentialy by loading a
previously dumped core image file.  This is slow the first
time around but once the ~25MB core image is cached,
execution is really fast and you have access to a lot of
goodies.  So a script like

#!/usr/local/bin/cmucl -script
(format t "Hello, World!~%")

can execute in a few milliseconds.  On systems with mmap(2)
or equivalent, the core image is simply copy-on-write mmaped.
This is a win since only the required pages will be loaded
(and not all of 25MB) and COW allows local changes.

>From what I understand, to do something equivalent on plan9
would require creating a segment and copying the core file to
it.  Is this correct?  Presumably the reads are cached?  Even
so, there will the cost of copying to the segment.  Or can
one create multiple text and data segments in some way so
that stuff will be paged in as necessary?  Also, if a shared
segment is created won't the forked processes be able to
modify this segment?  Ideally one would like a private copy
for each child.  Is segattach + read the best (only?) way to
do this?

sbcl too uses a core file like cmucl.  They both compile code
so are generally faster than clisp, which is the third
alternative.  Note: my interest is purely hypothetical at the
moment.

Thanks!

Re: [9fans] lisp

[erik quanstrom]

i'm assuming by "core file" you don't mean executable.
plan 9 already keeps an executable cache.

> Presumably the reads are cached?

reads are not cached.  read on plan 9 is syncronous.
there is no block cache.

> Even so, there will the cost of copying to the segment.  Or can
> one create multiple text and data segments in some way so
> that stuff will be paged in as necessary?  Also, if a shared
> segment is created won't the forked processes be able to
> modify this segment?  Ideally one would like a private copy
> for each child.  Is segattach + read the best (only?) way to
> do this?

why wouldn't you use ramfs?

- erik

Re: [9fans] lisp

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1342 bytes --]

On Mon, Jul 7, 2008 at 11:45 AM, erik quanstrom <quanstro@coraid.com> wrote:

> i'm assuming by "core file" you don't mean executable.
> plan 9 already keeps an executable cache.
>

A Lisp core file can be equated to a specially formatted executable.  The
Lisp environment is sort of a loader for these.  When people distribute lisp
applications, they give you a core file, and it doesn't "run on it's own"
without the Lisp runtime loading it.  The same goes for some Scheme
implementations as well (with the notable exception of Gambit Scheme which
has a C library runtime, and can build standalone binaries as well as more C
code, and it's not even GNU specific C thank god)

Remember code is data and data is code in Lisp land.


>
> > Presumably the reads are cached?
>
> reads are not cached.  read on plan 9 is syncronous.
> there is no block cache.
>
> > Even so, there will the cost of copying to the segment.  Or can
> > one create multiple text and data segments in some way so
> > that stuff will be paged in as necessary?  Also, if a shared
> > segment is created won't the forked processes be able to
> > modify this segment?  Ideally one would like a private copy
> > for each child.  Is segattach + read the best (only?) way to
> > do this?
>
> why wouldn't you use ramfs?
>
> - erik
>
>
>

[-- Attachment #2: Type: text/html, Size: 1953 bytes --]

Re: [9fans] lisp

[Bakul Shah]

On Mon, 07 Jul 2008 14:45:53 EDT erik quanstrom <quanstro@coraid.com>  wrote:
> i'm assuming by "core file" you don't mean executable.
> plan 9 already keeps an executable cache.

It contains executable code but it is not an executable in
the sense you don't directly feed it to exec(2).  A lisp
process might add new functions and dump a new core image
file.  A later lisp process will have those functions
available to it.  One can even choose a diff. core file to
start with.

> > Presumably the reads are cached?
>
> reads are not cached.  read on plan 9 is syncronous.
> there is no block cache.
>
> > Even so, there will the cost of copying to the segment.  Or can
> > one create multiple text and data segments in some way so
> > that stuff will be paged in as necessary?  Also, if a shared
> > segment is created won't the forked processes be able to
> > modify this segment?  Ideally one would like a private copy
> > for each child.  Is segattach + read the best (only?) way to
> > do this?
>
> why wouldn't you use ramfs?

You mean to cache the core file in memory?  That can
work...

Thanks!

Re: [9fans] lisp

[Charles Forsyth]

> It contains executable code but it is not an executable in
> the sense you don't directly feed it to exec(2).  A lisp

in the script you gave earlier
	#!/usr/local/bin/cmucl -script
	(format t "Hello, World!~%")
cmucl is directly executable but that's presumably the original
lisp image; where is the new "core image" mentioned that makes
the script run quickly?

Re: [9fans] lisp

[Bakul Shah]

On Mon, 07 Jul 2008 20:55:36 BST Charles Forsyth <forsyth@terzarima.net>  wrote:
> > It contains executable code but it is not an executable in
> > the sense you don't directly feed it to exec(2).  A lisp
>
> in the script you gave earlier
> 	#!/usr/local/bin/cmucl -script
> 	(format t "Hello, World!~%")
> cmucl is directly executable but that's presumably the original
> lisp image; where is the new "core image" mentioned that makes
> the script run quickly?

cmucl is directly executable but it has only enough
intelligence to load a big lisp.core, which contains all the
smarts.

Re: [9fans] lisp

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

On Mon, Jul 7, 2008 at 1:15 PM, Bakul Shah
<bakul+plan9@bitblocks.com<bakul%2Bplan9@bitblocks.com>>
wrote:

> On Mon, 07 Jul 2008 20:55:36 BST Charles Forsyth <forsyth@terzarima.net>
>  wrote:
> > > It contains executable code but it is not an executable in
> > > the sense you don't directly feed it to exec(2).  A lisp
> >
> > in the script you gave earlier
> >       #!/usr/local/bin/cmucl -script
> >       (format t "Hello, World!~%")
> > cmucl is directly executable but that's presumably the original
> > lisp image; where is the new "core image" mentioned that makes
> > the script run quickly?
>
> cmucl is directly executable but it has only enough
> intelligence to load a big lisp.core, which contains all the
> smarts.
>
> Right, then

(format t "Hello, World!~%")

basically gets "read" then compiled then executed right?  (thinking REPL
here)

Right?  At least that's how SBCL works based on my understanding.

[-- Attachment #2: Type: text/html, Size: 1404 bytes --]

Re: [9fans] lisp

[Bakul Shah]

On Mon, 07 Jul 2008 13:21:33 PDT "David Leimbach" <leimy2k@gmail.com>  wrote:
> (format t "Hello, World!~%")
>
> basically gets "read" then compiled then executed right?  (thinking REPL
> here)
>
> Right?  At least that's how SBCL works based on my understanding.

Well, it is not a read-eval-print-loop -- it is not
interactive or a loop and the code has to explicit print
something.  But yes the expression does get read, compiled
and executed.  SBCL is forked from CMUCL so they share most
of their behavior.

[Aside: there seems to be a lag of about 15 minutes from my
sending an email to 9fans and getting it back]

Re: [9fans] lisp

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

On Mon, Jul 7, 2008 at 1:47 PM, Bakul Shah
<bakul+plan9@bitblocks.com<bakul%2Bplan9@bitblocks.com>>
wrote:

> On Mon, 07 Jul 2008 13:21:33 PDT "David Leimbach" <leimy2k@gmail.com>
>  wrote:
> > (format t "Hello, World!~%")
> >
> > basically gets "read" then compiled then executed right?  (thinking REPL
> > here)
> >
> > Right?  At least that's how SBCL works based on my understanding.
>
> Well, it is not a read-eval-print-loop -- it is not
> interactive or a loop and the code has to explicit print
> something.  But yes the expression does get read, compiled
> and executed.  SBCL is forked from CMUCL so they share most
> of their behavior.


right, I was just referring to REPL because obviously that S-expression is
scripted in.  It's not going through the standard REPL at all, but a sort of
a variant of it.  Perhaps just RE :-)

Dave


>
>
> [Aside: there seems to be a lag of about 15 minutes from my
> sending an email to 9fans and getting it back]
>
>

[-- Attachment #2: Type: text/html, Size: 1511 bytes --]

Re: [9fans] lisp

[Dave Eckhardt]

> cmucl is directly executable but it has only enough
> intelligence to load a big lisp.core, which contains
> all the smarts.

If these core files aren't generated all *that* often,
one could write a tool which would turn a core file
into a byte array, link a new lisp executable, and
exec() that.  I realize that's not an answer to the
question as posed, but...

Dave Eckhardt

Re: [9fans] lisp

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

On Tue, Jul 8, 2008 at 7:08 AM, Dave Eckhardt
<davide+p9@cs.cmu.edu<davide%2Bp9@cs.cmu.edu>>
wrote:

> > cmucl is directly executable but it has only enough
> > intelligence to load a big lisp.core, which contains
> > all the smarts.
>
> If these core files aren't generated all *that* often,
> one could write a tool which would turn a core file
> into a byte array, link a new lisp executable, and
> exec() that.  I realize that's not an answer to the
> question as posed, but...
>
> Dave Eckhardt
>
> I'm not sure if any action is being taken just yet on which Lisp to "go
for" but, if I were porting a lisp I'd be looking at either CMUCL or SBCL,
and probably lean more towards SBCL, as they've done a bit of work to make
bootstrapping a little nicer.

I like Dave Eckhardt's idea.  I don't think core files get changed that
often, however sometimes when one produces a "standalone" product from one
of the above SBCL or CMUCL, you do want to make your own core file and
distribute it.  That's still a pretty static process if my memory serves me.

Dave

[-- Attachment #2: Type: text/html, Size: 1428 bytes --]

Re: [9fans] lisp

[Bakul Shah]

On Tue, 08 Jul 2008 08:07:59 PDT "David Leimbach" <leimy2k@gmail.com>  wrote:
>
> On Tue, Jul 8, 2008 at 7:08 AM, Dave Eckhardt
> <davide+p9@cs.cmu.edu<davide%2Bp9@cs.cmu.edu>>
> wrote:
>
> > > cmucl is directly executable but it has only enough
> > > intelligence to load a big lisp.core, which contains
> > > all the smarts.
> >
> > If these core files aren't generated all *that* often,
> > one could write a tool which would turn a core file
> > into a byte array, link a new lisp executable, and
> > exec() that.  I realize that's not an answer to the
> > question as posed, but...
> >
> > Dave Eckhardt
> >
> > I'm not sure if any action is being taken just yet on which Lisp to "go
> for" but, if I were porting a lisp I'd be looking at either CMUCL or SBCL,
> and probably lean more towards SBCL, as they've done a bit of work to make
> bootstrapping a little nicer.

I had asked my friend who ported CMUCL to Windows about this.
He too first looked at SBCL but found it hard to port and the
bootstrapping process was a lot longer.  He was then able to
create the initial CMUCL port by porting just the runtime --
he added just enough of a compatibility layer to allow use of
a FreeBSD core image.  Compiled Lisp object would think it is
running on FreeBSD.

> I like Dave Eckhardt's idea.  I don't think core files get changed that
> often,

In the olden days Lisps used to create a custom executable by
storing the heap in a data segment. Something to look into.

Re: [9fans] lisp

[C H Forsyth]

if you make it executable, i think you should also find that the resulting
executable has its contents read into memory on demand (ie, a page fault causes
a read from the executable file), which might suit you.

Re: [9fans] lisp

[Bakul Shah]

On Tue, 08 Jul 2008 22:12:10 BST C H Forsyth <forsyth@vitanuova.com>  wrote:
> if you make it executable, i think you should also find that the resulting
> executable has its contents read into memory on demand (ie, a page fault causes
> a read from the executable file), which might suit you.

Right.  My questions at this point have more to do with
understanding the landscape but if I do this, my initial goal
would be to get it up and running in as little time as
possible even if the result is slow and hacky.  Cleanup &
optimizations can follow.

Thanks to all for their suggestions & comments.

Re: [9fans] lisp

[geoff]

It's not entirely true that reads are not cached.  The file servers of
course maintain large ram caches (or they should), and the clients can
request client-side caching with mount -C (the MCACHE flag), as is
normally done for the root:

; ns | grep '.-.*C'
mount -aC '#s/boot' /root