Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Rob Browning <rlb@cs.utexas.edu>]

Simon Josefsson <simon@josefsson.org> writes:

> Where are we headed?

I wasn't saying I knew, but I figured that everyone would tell me if I
described what I was interested in, and if it stuff that wasn't
appropriate :>

> I think you'll find, as at least I have, that it's hard to solve these
> problems by hacking up a backend.  Backends simply doesn't have any
> influence on what Gnus does.

I looked at backends a while back to see how hard it would be to add
at least one more article state, and it looked to me like there was
way too much special casing/hard-coding for that to be very feasible.

> Absolutely.  It would be easy to copy nnml into nnmlng and implement
> nnmlng-request-set-mark.  This would make flags live in the backend.
> But we'd still have legacy stuff in .newsrc.eld that isn't really
> used.  In OGnus I hope we can make the entire .newsrc.eld file
> optional.

I'd like to go even farther and have gnus write out log/journal
snippets every time you take an action, or have it do atomic writes of
group state info (using link/unlink tricks) so that it would be very
hard to corrupt your state records.

> >   - it should allow every article to be marked with an arbitrary
> >     number of "tags".  Some of the tag names (perhaps all prefixed
> >     with "gnus:", like "gnus:unread") would be reserved for internal
> >     use, but others would be available for the user.
> 
> This isn't very difficult, I think.

It didn't look very easy to me last time I poked around, but I didn't
have time to get really familiar with the code.  A simple solution
would be to make the article state be a list (set) of symbols
(implemented as a hash if performance ever became an issue).  This
seems flexible enough to handle most things I can think of.

> >   - it should be easy to write code that will take actions based on
> >     the user tags.
> 
> What do you mean?

I mean that you should be able to do a lot of things based on tags via
the user interface (finds with and/or semantics for articles with
particular tags, etc.), and that there should be a clean API for
writing user code that took actions based on the article tags:

  (setq my-important-articles
    (map-ignoring-falses
      (lambda (article)
        (if (article-has-user-tag 'important)
            article
            #f))
      (group-get-articles backend "inbox")))

> >   - it should be easy to write code that changes the "system" state of
> >     an article: (add-system-article-mark article 'gnus:unread)
> 
> This is tricky.  What datatype is article?

In the long run?  An opaque one with all the necessary
getters/setters.

  (make-article)
  (article-set-body! article text)
  (article-get-body  article)
  (article-get-header 'to article)
  (article-set-header! 'to article)

etc.  This makes it easy to work with and means that the
implementation can change to reflect changes in emacs without
disturbing the rest of the code.

> This isn't very complicated today, but it's hidden from the user in a
> way that make it seem complex.  IMHO.

Right.  I'd like things to be cleaner, and with an API that's
documented so that you know which bits are less likely to change from
version to version.

> SQL seem to pop up once in a while as a solution to these problems,
> but the only performance issues I've seen are inside Gnus and they
> wouldn't be solved by changing the backend.

True, this isn't a big deal, though if you have more than 60000
articles that you're keeping around, from say, *all* of debian-user
over the past couple of years, the filesystem, unless you're going to
switch to something like reiser, can kill you with nnml unless you go
get that high end duron/athlon you've been eying.

> I've thought of this many times, but it's difficult to isolate
> "common code" and cast an API to it in stone.  Writing mail (message
> buffer) and reading mail (article buffer) are pretty isolated, and
> that's a good thing.  Server structures, article flag structures and
> group structures are touched everywhere by lots of code, and this is
> were much of the complexity stem from, I think.

I may be wrong, but my gut feeling is that for 99% of what you care
about doing: keeping track of groups, getting article lists from them,
checking and setting article states, adding new articles, etc.,
creating a good, well documented API just wouldn't be that hard;
perhaps I'm overlooking something obvious, but it doesn't *seem* like
rocket-science, and making the data-type internals opaque would
provide a lot of freedom for future internal changes, and would make
the codebase *much* more approachable.

As an added IMO *way-cool* feature, I wonder if it would be possible
to define an interface that was both elisp and scheme friendly, and
better yet, use a backend storage format that was too (i.e. have the
forms where you store the state stick to a common "form" subset that
both parsers can handle).  If you could do that, then you could
implement the gnus-engine API in both languages and then use emacs to
deal with your mail interactively, and guile (or perhaps even stalin)
for batch mode processing.

To go *way* out in fantasy-land, I wonder if you could implement the
library using guile-friendly code (or a compatibility lib/layer) that
the elisp interpreter was also happy with.  If so, then you'd also be
ready if/when emacs does switch to the guile interpreter.

OK, I'll go take my medication now :>

(Seriously, if any of this strikes a chord with someone out there, let
 me know.  I *would* be interested in spending some time working on
 some/all of these bits.)

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930