From mboxrd@z Thu Jan 1 00:00:00 1970 From: john at keeping.me.uk (John Keeping) Date: Thu, 9 Jan 2014 22:58:02 +0000 Subject: RFE: .so filters In-Reply-To: References: Message-ID: <20140109225802.GM7608@serenity.lan> On Thu, Jan 09, 2014 at 10:34:26PM +0100, Jason A. Donenfeld wrote: > I'm thinking about this filtering situation w.r.t. gravatar and > potentially running multiple filters on one page. Something I've been > considering is implementing a simple dlopen() mechanism for filters, > if the filter filename starts with "soname:" or "lib:" or similar, so > as to avoid the fork()ing and exec()ing we currently have, for high > frequency filters. The idea is that first use of the filter would be > dlopen()'d, but wouldn't be dlclose()'d until the end of the > processing. This way the same function could be used over and over > again without significant penalty. > > In my first thinking of this, the method of action would be the same > as the current system -- "int filter_run(int argc, char *argv[])" is > dlopen()'d, executed, and it reads and writes to the dup2()'d file > descriptor. Unfortunately, the piping in this introduces a cost that > I'd rather avoid. In the case of gravatar (or more generally, email > author filters), we'd be better off with a "char *filter_run(int argc, > char *argv[])", that can just return the string that the html > functions will then print. This, however, breaks the current filtering > paradigm, and might not be ideal for filters that enjoy a stream of > data (such as source code filters). This distinction more or less > points toward coming up with a library API of sorts, but I really > really really don't want to add a full fledged plugin system. So this > has me leaning toward the simpler first idea. > > But I'm undecided at the moment. Comments and suggestions are most > welcome. That interface doesn't really match the way the current filters work. Currently when we open a filter we replace cgit's stdout with a pipe into the filter process, so none of the existing CGit code will work with this interface. We could swap out write with a function pointer into the filter, but I don't think we guarantee that all of the data is written in one go which makes life harder for filter writers (although for simple cases like author info we probably could guarantee to write it all at once). If we allow filters to act incrementally, then we can just leave the filter running and swap it in or out when required. That would require a single dup2 to make it work the same way that the filters currently work. Interestingly, there is an "htmlfd" variable in html.c but it is never changed from STDOUT_FILENO; I wonder if that can be used or are there other places (possibly in libgit.a code) that just use stdout, in which case we should remove that variable. But there is the problem of terminating the response; Lukas' suggestion of using NUL for that may be the best, it's not that hard to printf '\0' in shell. OTOH, the particular case of author details the input is more clearly defined than items for which we currently provide filters, so maybe it could use a different interface. One final point (although I don't think you're suggesting this) is that we shouldn't require shared objects; I think scripts using stdin+stdout are a much simpler interface and provides a much lower barrier to entry, not least because the range of languages that can be used to implement the filters is so much greater.