From mboxrd@z Thu Jan 1 00:00:00 1970 Date: Mon, 7 Dec 2009 13:06:52 +0100 From: Mechiel Lukkien To: Fans of the OS Plan 9 from Bell Labs <9fans@9fans.net> Message-ID: <20091207120652.GB16320@knaagkever.ueber.net> References: <20091205031747.GA8759@nipl.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="6sX45UoQRIJXqkqR" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.13 (2006-08-11) Content-Transfer-Encoding: 7bit Subject: Re: [9fans] ideas for helpful system io functions Topicbox-Message-UUID: a9a9132a-ead5-11e9-9d60-3106f5b1d025 --6sX45UoQRIJXqkqR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sat, Dec 05, 2009 at 08:24:45AM -1000, Tim Newsham wrote: > ps. if you wanted to hide this ugliness of passing a buffer and > fd to a child process instead of just passing an fd, you could > still solve it in userland without a syscall. Write a library > that does buffered IO. Include unget() if you like. Write the > library in a way that you can initialize it after a fork/exec > to pick up state from the parent (ie. by taking two fds, > reading the buffer from the first, and continuing on with the > 2nd when it is exhausted). > > Is there much benefit in doing this in the kernel instead? it's all library code, and it loses the "everything is a file (descriptor)" advantage. you cannot pass that library state to another program. you could if the state was a file descriptor. for inferno i wrote an http client library that turns a request into an fd to read the data from. that fd has http chunking,gzip,ssl peeled off. now i can pass the fd with the http response to other programs, do buffered i/o on it, etc. this is implemented in user-space btw, with inferno's sys->file2chan (as opposed to pipes, you can do error message propagation over file2chan's). since file descriptors are so essential, it may help to have "tools" to use them. yesterday evening i hacked up devbuf.c and devjoin.c after reading this thread. both offer a file "new". for devbuf.c you can write data to it, then later consume it (yes, you could just use a pipe instead). for devjoin.c, you can write fd numbers (of open files) to register an fd, then later reads will get data from the first registered file, when that returns 0 it continues on the next, and so on. so fd's can be chained for reading (not writing). i know this "join" functionality is different from what sam originally described. i've attached devbuf.c and devjoin.c, as example (for inferno). they have bugs (don't assign qid.path, probably *walk is broken too). testbufjoin.b is an example of how the dev's can be used. it creates a new fd that has a buffer at the front (e.g. leftovers from http header reading), then continues on stdin (where the leftover may have come from). then it reads the new fd and writes its data to stdout. these devices are not for performance. perhaps they make working with one of the most basic OS concepts (fd's) a bit easier. but perhaps this problem is not common enough, or can be handled (with fd's preferrably) in a better way. mjl --6sX45UoQRIJXqkqR Content-Type: text/x-csrc; charset=utf-8 Content-Disposition: attachment; filename="devbuf.c" Content-Transfer-Encoding: quoted-printable #include "dat.h" #include "fns.h" #include "error.h" typedef struct Buffile Buffile; struct Buffile { uchar *p; int s; int e; }; enum { Qdir, Qbuffile, }; Dirtab bufdir[] =3D { ".", {Qdir,0,QTDIR}, 0, DMDIR|0500, "new", {Qbuffile}, 0, 0660, }; static Buffile* buffilealloc(uchar *p, int n) { Buffile *b; b =3D malloc(sizeof b[0]+n); b->p =3D (uchar*)b+sizeof b[0]; memmove(b->p, p, n); b->s =3D 0; b->e =3D n; return b; } static Chan* bufattach(char *spec) { return devattach('=CE=B2', spec); } static Walkqid* bufwalk(Chan *c, Chan *nc, char **name, int nname) { return devwalk(c, nc, name, nname, bufdir, nelem(bufdir), devgen); } static int bufstat(Chan *c, uchar *db, int n) { return devstat(c, db, n, bufdir, nelem(bufdir), devgen); } static Chan* bufopen(Chan *c, int omode) { return devopen(c, omode, bufdir, nelem(bufdir), devgen); } static void bufclose(Chan *c) { free(c->aux); c->aux =3D nil; } static long bufread(Chan *c, void *va, long n, vlong off) { Buffile *b; int have; if(c->qid.type =3D=3D QTDIR) return devdirread(c, va, n, bufdir, nelem(bufdir), devgen); b =3D c->aux; if(b =3D=3D nil) return 0; =09 USED(off); have =3D b->e - b->s; if(have < n || n < 0) n =3D have; memmove(va, b->p, n); b->s +=3D n; return n; } static long bufwrite(Chan *c, void *va, long n, vlong off) { if(c->qid.type =3D=3D QTDIR) error(Eisdir); free(c->aux); c->aux =3D buffilealloc(va, n); return n; } Dev bufdevtab =3D { '=CE=B2', "buf", devinit, bufattach, bufwalk, bufstat, bufopen, devcreate, bufclose, bufread, devbread, bufwrite, devbwrite, devremove, devwstat, }; --6sX45UoQRIJXqkqR Content-Type: text/x-csrc; charset=utf-8 Content-Disposition: attachment; filename="devjoin.c" Content-Transfer-Encoding: quoted-printable #include "dat.h" #include "fns.h" #include "error.h" typedef struct Join Join; struct Join { Chan *c; Join *next; }; enum { Qdir, Qjoinfile, }; Dirtab joindir[] =3D { ".", {Qdir,0,QTDIR}, 0, DMDIR|0500, "new", {Qjoinfile}, 0, 0660, }; static void joinfree(Join *j) { if(j =3D=3D nil) return; joinfree(j->next); cclose(j->c); free(j); } static Chan* joinattach(char *spec) { return devattach('=CE=B4', spec); } static Walkqid* joinwalk(Chan *c, Chan *nc, char **name, int nname) { return devwalk(c, nc, name, nname, joindir, nelem(joindir), devgen); } static int joinstat(Chan *c, uchar *db, int n) { return devstat(c, db, n, joindir, nelem(joindir), devgen); } static Chan* joinopen(Chan *c, int omode) { return devopen(c, omode, joindir, nelem(joindir), devgen); } static void joinclose(Chan *c) { joinfree(c->aux); c->aux =3D nil; } static long joinread(Chan *c, void *va, long n, vlong off) { Join *j; long l; if(c->qid.type =3D=3D QTDIR) return devdirread(c, va, n, joindir, nelem(joindir), devgen); l =3D 0; while(c->aux !=3D nil) { j =3D c->aux; l =3D devtab[j->c->type]->read(j->c, va, n, off); if(l !=3D 0) break; c->aux =3D j->next; cclose(j->c); free(j); } return l; } static long joinwrite(Chan *c, void *va, long n, vlong off) { char buf[32]; int fd; Chan *jc; Join *j; Join *nj; if(c->qid.type =3D=3D QTDIR) error(Eisdir); if(n >=3D sizeof buf+1) error(Ebadarg); memmove(buf, va, n); buf[n] =3D 0; fd =3D atoi(buf); jc =3D fdtochan(up->env->fgrp, fd, -1, 0, 1); nj =3D malloc(sizeof nj[0]); nj->c =3D jc; nj->next =3D nil; if(c->aux =3D=3D nil) { c->aux =3D nj; } else { for(j =3D c->aux; j->next !=3D nil; j =3D j->next) {} j->next =3D nj; } return n; } Dev joindevtab =3D { '=CE=B4', "join", devinit, joinattach, joinwalk, joinstat, joinopen, devcreate, joinclose, joinread, devbread, joinwrite, devbwrite, devremove, devwstat, }; --6sX45UoQRIJXqkqR Content-Type: chemical/x-molconn-Z Content-Disposition: attachment; filename="testbufjoin.b" Content-Transfer-Encoding: quoted-printable implement Testbufjoin;=0A=0Ainclude "sys.m";=0A sys: Sys;=0A sprint: import= sys;=0Ainclude "draw.m";=0Ainclude "bufio.m";=0A bufio: Bufio;=0A Iobuf: i= mport bufio;=0A=0ATestbufjoin: module {=0A init: fn(nil: ref Draw->Context,= nil: list of string);=0A};=0A=0A=0Ainit(nil: ref Draw->Context, nil: list = of string)=0A{=0A sys =3D load Sys Sys->PATH;=0A bufio =3D load Bufio Bufio= ->PATH;=0A=0A bfd :=3D sys->open("/buf/new", Sys->ORDWR);=0A if(bfd =3D=3D = nil)=0A fail(sprint("%r"));=0A jfd :=3D sys->open("/join/new", Sys->ORDWR)= ;=0A if(jfd =3D=3D nil)=0A fail(sprint("%r"));=0A=0A if(sys->fprint(bfd, "= a write to bfd\n") < 0)=0A fail(sprint("write to bfd: %r"));=0A=0A if(sys-= >fprint(jfd, "%d", bfd.fd) < 0)=0A fail(sprint("write bfd.fd to jfd: %r"))= ;=0A if(sys->fprint(jfd, "0") < 0)=0A fail(sprint("write 0 to jfd: %r"));= =0A=0A cat(jfd);=0A}=0A=0Acat(fd: ref Sys->FD)=0A{=0A b :=3D bufio->fopen(f= d, Bufio->OREAD);=0A if(b =3D=3D nil)=0A fail(sprint("fopen: %r"));=0A buf= :=3D array[1024] of byte;=0A fd1 :=3D sys->fildes(1);=0A for(;;) {=0A n := =3D b.read(buf, len buf);=0A if(n < 0)=0A fail(sprint("read: %r"));=0A = if(n =3D=3D 0)=0A break;=0A warn(sprint("read %d bytes", n));=0A if(sys= ->write(fd1, buf, n) !=3D n)=0A fail(sprint("write: %r"));=0A }=0A}=0A=0A= warn(s: string)=0A{=0A sys->fprint(sys->fildes(2), "%s\n", s);=0A}=0A=0Afai= l(s: string)=0A{=0A warn(s);=0A raise "fail:"+s;=0A}=0A --6sX45UoQRIJXqkqR--