Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Simon Josefsson <simon@josefsson.org>]

Rob Browning <rlb@cs.utexas.edu> writes:

>  I've got some similar problems (though not quite the same), and I'm
>  trying to decide how hard it might be to finally fix gnus to address
>  my concerns.  In addition, I'm trying to decide whether or not my
>  fixes, if any, would be likely to be accepted upstream.  I don't want
>  to throw away the effort if I'm trying to go against the direction
>  gnus is headed :>)

Where are we headed?

> I've thought about hacking up a backend that fixes some of my major
> gripes with gnus, but I didn't know how hard it would be.  One of my
> gripes is that you can't easily just archive a group (by
> copying/moving directories), and have gnus DTRT.  This is both because
> gnus doesn't store the readness info with the files, and because gnus
> keeps its own idea about where your groups are - it doesn't just
> present you with whatever groups it finds in the nnml source
> directory.

I think you'll find, as at least I have, that it's hard to solve these
problems by hacking up a backend.  Backends simply doesn't have any
influence on what Gnus does.

>   - it should be easy to move your groups around, archiving and
>     unarchiving (tarring up, storing on tape, whatever), without
>     confusing gnus.  This means storing the mark info with the
>     backend.  If I delete an nnml directory from the filesystem, the
>     next time I start up gnus, I'd just like to see it gone.  If I
>     untar an old one, I'd like to see gnus notice it.

Yes.  In a similar note, nnml groups should by default live in
~/Mail/nnml/ to make this more doable.

>   - it should be *very* hard to have an emacs crash totally hose all
>     your state information (readness, group hierarchies, layout,
>     etc.).  This means storing the mark info with the backend *and*
>     writing it frequently - this may mean *not* writing out a
>     monolihic state file, not even one per backend group.

Absolutely.  It would be easy to copy nnml into nnmlng and implement
nnmlng-request-set-mark.  This would make flags live in the backend.
But we'd still have legacy stuff in .newsrc.eld that isn't really
used.  In OGnus I hope we can make the entire .newsrc.eld file
optional.

>   - it should allow every article to be marked with an arbitrary
>     number of "tags".  Some of the tag names (perhaps all prefixed
>     with "gnus:", like "gnus:unread") would be reserved for internal
>     use, but others would be available for the user.

This isn't very difficult, I think.

>   - it should be easy to write code that will take actions based on
>     the user tags.

What do you mean?

>   - it should be easy to write code that changes the "system" state of
>     an article: (add-system-article-mark article 'gnus:unread)

This is tricky.  What datatype is article?

Currently articles are atomic entities, numbered with elisp integers.
Some time ago, I had a slightly crazy idea to use URLs in Gnus.  I
still think it's a good idea, but it would require some work.  This
way we'd have an extensible framework in place, and adding support for
partial message fetching and similar things wouldn't be difficult.
With Bill Perry's URL package, we'd might even get asynchronous
support and other niceties.

>   - it should have one clear policy with respect to marking articles
>     read, and/or deleting them.
>
>   - it should be *easy* to understand this policy, and to override
>     it.  I'd like to be able to do things like this and expect them to
>     interact well with the rest of the system:
> 
>        (setq gnus-expire-check
>         (lambda (article)
>           (or (normal-system-expire-check article)
>               (article-user-mark? article 'garbage)
>               (article-user-mark? article 'spam))))

This isn't very complicated today, but it's hidden from the user in a
way that make it seem complex.  IMHO.

>   - it should be easy for the user to map over articles with code that
>     makes it obvious what's going on:
> 
>       (gnus-group-map-all-articles backend "inbox"
>         (lambda (article)
>           (if (search-article-for "MONEY")
>             (add-article-user-mark article 'spam))))

>   - it should handle very large groups efficiently (maybe even with an
>     sql backend option).

SQL seem to pop up once in a while as a solution to these problems,
but the only performance issues I've seen are inside Gnus and they
wouldn't be solved by changing the backend.  I seriously doubt that a
SQL backend would be faster than a specialized database such as nnml
or nnfolder (or IMAP for that matter).

> I've also wondered how hard it might be to turn much of the gnus code
> into a mail related emacs library.  Gnus has a tremendous quantity of
> excellent mail related code, and it's stuff that's hard to get right.
> It would be really nice if this code were avaialbe for re-use so that
> if someone's just not quite happy with gnus, they could still use the
> library, create something that more closely suits their needs, and
> continue to contribute to the development of the common code.

I've thought of this many times, but it's difficult to isolate "common
code" and cast an API to it in stone.  Writing mail (message buffer)
and reading mail (article buffer) are pretty isolated, and that's a
good thing.  Server structures, article flag structures and group
structures are touched everywhere by lots of code, and this is were
much of the complexity stem from, I think.