"Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

"Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Rob Browning]

Simon Josefsson <simon@josefsson.org> writes:

> prj@po.cwru.edu (Paul Jarc) writes:
> 
> > What do 'killed, 'dormant, 'save, 'download, and 'unsend mean?  Do
> > they persist between Gnus sessions?
> 
> Unsend is for nndraft:queue only (I think), killed isn't persistant (I
> think), the others are persistant but if the backend can't implement
> them (some IMAP can't store flags) Gnus keep them in .newsrc.eld.  So
> you shouldn't need to worry.

(This note is being precipitated by another one of my talented friends
 dumping gnus because it DTWT one too many times.  He loved gnus, but
 over time, there were some problems that just made it untenable for
 him to keep using it.

 I've got some similar problems (though not quite the same), and I'm
 trying to decide how hard it might be to finally fix gnus to address
 my concerns.  In addition, I'm trying to decide whether or not my
 fixes, if any, would be likely to be accepted upstream.  I don't want
 to throw away the effort if I'm trying to go against the direction
 gnus is headed :>)

I've thought about hacking up a backend that fixes some of my major
gripes with gnus, but I didn't know how hard it would be.  One of my
gripes is that you can't easily just archive a group (by
copying/moving directories), and have gnus DTRT.  This is both because
gnus doesn't store the readness info with the files, and because gnus
keeps its own idea about where your groups are - it doesn't just
present you with whatever groups it finds in the nnml source
directory.

For my purposes, I'd like a slightly simplified mail/news reader that
was suspciciously similar (i.e. nearly identical in most ways) to
gnus, but with the following additions (all mostly related to mail,
and not all probably fixable via the backend).

  (please excuse any schemisms in the pseudo-code below)

  - it should be easy to move your groups around, archiving and
    unarchiving (tarring up, storing on tape, whatever), without
    confusing gnus.  This means storing the mark info with the
    backend.  If I delete an nnml directory from the filesystem, the
    next time I start up gnus, I'd just like to see it gone.  If I
    untar an old one, I'd like to see gnus notice it.

  - it should be *very* hard to have an emacs crash totally hose all
    your state information (readness, group hierarchies, layout,
    etc.).  This means storing the mark info with the backend *and*
    writing it frequently - this may mean *not* writing out a
    monolihic state file, not even one per backend group.

  - it should allow every article to be marked with an arbitrary
    number of "tags".  Some of the tag names (perhaps all prefixed
    with "gnus:", like "gnus:unread") would be reserved for internal
    use, but others would be available for the user.

  - it should be easy to write code that will take actions based on
    the user tags.

  - it should be easy to write code that changes the "system" state of
    an article: (add-system-article-mark article 'gnus:unread)

  - it should have one clear policy with respect to marking articles
    read, and/or deleting them.

  - it should be *easy* to understand this policy, and to override
    it.  I'd like to be able to do things like this and expect them to
    interact well with the rest of the system:

       (setq gnus-expire-check
        (lambda (article)
          (or (normal-system-expire-check article)
              (article-user-mark? article 'garbage)
              (article-user-mark? article 'spam))))

  - it should be easy for the user to map over articles with code that
    makes it obvious what's going on:

      (gnus-group-map-all-articles backend "inbox"
        (lambda (article)
          (if (search-article-for "MONEY")
            (add-article-user-mark article 'spam))))

  - it should handle very large groups efficiently (maybe even with an
    sql backend option).

Perhaps gnus can already do this stuff and I just don't know it, but
if not, anyone have any thoughts about how hard this would be to add?

I've also wondered how hard it might be to turn much of the gnus code
into a mail related emacs library.  Gnus has a tremendous quantity of
excellent mail related code, and it's stuff that's hard to get right.
It would be really nice if this code were avaialbe for re-use so that
if someone's just not quite happy with gnus, they could still use the
library, create something that more closely suits their needs, and
continue to contribute to the development of the common code.

Thoughts?

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: "Fixing up" gnus - (how hard is this?).

[Paul Jarc]

Rob Browning <rlb@cs.utexas.edu> writes:
> One of my gripes is that you can't easily just archive a group (by
> copying/moving directories), and have gnus DTRT.  This is both
> because gnus doesn't store the readness info with the files,

Backends can do this.  If all backends were made to do this, Gnus
wouldn't have to duplicate the information in newsrc.eld, and
nn*-request-set-info could perhaps be discarded in favor of
-get-article-marks, -get-marked-articles, etc.

> and because gnus keeps its own idea about where your groups are - it
> doesn't just present you with whatever groups it finds in the nnml
> source directory.

I think backends can also tackle this problem.  I intend to do so for
nnmaildir.

>   - it should be *very* hard to have an emacs crash totally hose all
>     your state information (readness, group hierarchies, layout,
>     etc.).  This means storing the mark info with the backend *and*
>     writing it frequently - this may mean *not* writing out a
>     monolihic state file, not even one per backend group.

nnmaildir writes its state file frequently, but it is one monolithic
file per group.  I may move to some form of journaling, but I've got
other changes to get to first.

>   - it should allow every article to be marked with an arbitrary
>     number of "tags".

User-defined, you mean?  That would be useful indeed.  Obviously, it
requires cooperation from Gnus, though.

> I've also wondered how hard it might be to turn much of the gnus code
> into a mail related emacs library.

That would be good.  Gnus, and user applications in general, ought to
consist of largely application-independant, reusable code, along with
some UI-specific code and (user-redefinable, of course) bindings to
that UI code.


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Simon Josefsson]

Rob Browning <rlb@cs.utexas.edu> writes:

>  I've got some similar problems (though not quite the same), and I'm
>  trying to decide how hard it might be to finally fix gnus to address
>  my concerns.  In addition, I'm trying to decide whether or not my
>  fixes, if any, would be likely to be accepted upstream.  I don't want
>  to throw away the effort if I'm trying to go against the direction
>  gnus is headed :>)

Where are we headed?

> I've thought about hacking up a backend that fixes some of my major
> gripes with gnus, but I didn't know how hard it would be.  One of my
> gripes is that you can't easily just archive a group (by
> copying/moving directories), and have gnus DTRT.  This is both because
> gnus doesn't store the readness info with the files, and because gnus
> keeps its own idea about where your groups are - it doesn't just
> present you with whatever groups it finds in the nnml source
> directory.

I think you'll find, as at least I have, that it's hard to solve these
problems by hacking up a backend.  Backends simply doesn't have any
influence on what Gnus does.

>   - it should be easy to move your groups around, archiving and
>     unarchiving (tarring up, storing on tape, whatever), without
>     confusing gnus.  This means storing the mark info with the
>     backend.  If I delete an nnml directory from the filesystem, the
>     next time I start up gnus, I'd just like to see it gone.  If I
>     untar an old one, I'd like to see gnus notice it.

Yes.  In a similar note, nnml groups should by default live in
~/Mail/nnml/ to make this more doable.

>   - it should be *very* hard to have an emacs crash totally hose all
>     your state information (readness, group hierarchies, layout,
>     etc.).  This means storing the mark info with the backend *and*
>     writing it frequently - this may mean *not* writing out a
>     monolihic state file, not even one per backend group.

Absolutely.  It would be easy to copy nnml into nnmlng and implement
nnmlng-request-set-mark.  This would make flags live in the backend.
But we'd still have legacy stuff in .newsrc.eld that isn't really
used.  In OGnus I hope we can make the entire .newsrc.eld file
optional.

>   - it should allow every article to be marked with an arbitrary
>     number of "tags".  Some of the tag names (perhaps all prefixed
>     with "gnus:", like "gnus:unread") would be reserved for internal
>     use, but others would be available for the user.

This isn't very difficult, I think.

>   - it should be easy to write code that will take actions based on
>     the user tags.

What do you mean?

>   - it should be easy to write code that changes the "system" state of
>     an article: (add-system-article-mark article 'gnus:unread)

This is tricky.  What datatype is article?

Currently articles are atomic entities, numbered with elisp integers.
Some time ago, I had a slightly crazy idea to use URLs in Gnus.  I
still think it's a good idea, but it would require some work.  This
way we'd have an extensible framework in place, and adding support for
partial message fetching and similar things wouldn't be difficult.
With Bill Perry's URL package, we'd might even get asynchronous
support and other niceties.

>   - it should have one clear policy with respect to marking articles
>     read, and/or deleting them.
>
>   - it should be *easy* to understand this policy, and to override
>     it.  I'd like to be able to do things like this and expect them to
>     interact well with the rest of the system:
> 
>        (setq gnus-expire-check
>         (lambda (article)
>           (or (normal-system-expire-check article)
>               (article-user-mark? article 'garbage)
>               (article-user-mark? article 'spam))))

This isn't very complicated today, but it's hidden from the user in a
way that make it seem complex.  IMHO.

>   - it should be easy for the user to map over articles with code that
>     makes it obvious what's going on:
> 
>       (gnus-group-map-all-articles backend "inbox"
>         (lambda (article)
>           (if (search-article-for "MONEY")
>             (add-article-user-mark article 'spam))))

>   - it should handle very large groups efficiently (maybe even with an
>     sql backend option).

SQL seem to pop up once in a while as a solution to these problems,
but the only performance issues I've seen are inside Gnus and they
wouldn't be solved by changing the backend.  I seriously doubt that a
SQL backend would be faster than a specialized database such as nnml
or nnfolder (or IMAP for that matter).

> I've also wondered how hard it might be to turn much of the gnus code
> into a mail related emacs library.  Gnus has a tremendous quantity of
> excellent mail related code, and it's stuff that's hard to get right.
> It would be really nice if this code were avaialbe for re-use so that
> if someone's just not quite happy with gnus, they could still use the
> library, create something that more closely suits their needs, and
> continue to contribute to the development of the common code.

I've thought of this many times, but it's difficult to isolate "common
code" and cast an API to it in stone.  Writing mail (message buffer)
and reading mail (article buffer) are pretty isolated, and that's a
good thing.  Server structures, article flag structures and group
structures are touched everywhere by lots of code, and this is were
much of the complexity stem from, I think.

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Rob Browning]

Simon Josefsson <simon@josefsson.org> writes:

> Where are we headed?

I wasn't saying I knew, but I figured that everyone would tell me if I
described what I was interested in, and if it stuff that wasn't
appropriate :>

> I think you'll find, as at least I have, that it's hard to solve these
> problems by hacking up a backend.  Backends simply doesn't have any
> influence on what Gnus does.

I looked at backends a while back to see how hard it would be to add
at least one more article state, and it looked to me like there was
way too much special casing/hard-coding for that to be very feasible.

> Absolutely.  It would be easy to copy nnml into nnmlng and implement
> nnmlng-request-set-mark.  This would make flags live in the backend.
> But we'd still have legacy stuff in .newsrc.eld that isn't really
> used.  In OGnus I hope we can make the entire .newsrc.eld file
> optional.

I'd like to go even farther and have gnus write out log/journal
snippets every time you take an action, or have it do atomic writes of
group state info (using link/unlink tricks) so that it would be very
hard to corrupt your state records.

> >   - it should allow every article to be marked with an arbitrary
> >     number of "tags".  Some of the tag names (perhaps all prefixed
> >     with "gnus:", like "gnus:unread") would be reserved for internal
> >     use, but others would be available for the user.
> 
> This isn't very difficult, I think.

It didn't look very easy to me last time I poked around, but I didn't
have time to get really familiar with the code.  A simple solution
would be to make the article state be a list (set) of symbols
(implemented as a hash if performance ever became an issue).  This
seems flexible enough to handle most things I can think of.

> >   - it should be easy to write code that will take actions based on
> >     the user tags.
> 
> What do you mean?

I mean that you should be able to do a lot of things based on tags via
the user interface (finds with and/or semantics for articles with
particular tags, etc.), and that there should be a clean API for
writing user code that took actions based on the article tags:

  (setq my-important-articles
    (map-ignoring-falses
      (lambda (article)
        (if (article-has-user-tag 'important)
            article
            #f))
      (group-get-articles backend "inbox")))

> >   - it should be easy to write code that changes the "system" state of
> >     an article: (add-system-article-mark article 'gnus:unread)
> 
> This is tricky.  What datatype is article?

In the long run?  An opaque one with all the necessary
getters/setters.

  (make-article)
  (article-set-body! article text)
  (article-get-body  article)
  (article-get-header 'to article)
  (article-set-header! 'to article)

etc.  This makes it easy to work with and means that the
implementation can change to reflect changes in emacs without
disturbing the rest of the code.

> This isn't very complicated today, but it's hidden from the user in a
> way that make it seem complex.  IMHO.

Right.  I'd like things to be cleaner, and with an API that's
documented so that you know which bits are less likely to change from
version to version.

> SQL seem to pop up once in a while as a solution to these problems,
> but the only performance issues I've seen are inside Gnus and they
> wouldn't be solved by changing the backend.

True, this isn't a big deal, though if you have more than 60000
articles that you're keeping around, from say, *all* of debian-user
over the past couple of years, the filesystem, unless you're going to
switch to something like reiser, can kill you with nnml unless you go
get that high end duron/athlon you've been eying.

> I've thought of this many times, but it's difficult to isolate
> "common code" and cast an API to it in stone.  Writing mail (message
> buffer) and reading mail (article buffer) are pretty isolated, and
> that's a good thing.  Server structures, article flag structures and
> group structures are touched everywhere by lots of code, and this is
> were much of the complexity stem from, I think.

I may be wrong, but my gut feeling is that for 99% of what you care
about doing: keeping track of groups, getting article lists from them,
checking and setting article states, adding new articles, etc.,
creating a good, well documented API just wouldn't be that hard;
perhaps I'm overlooking something obvious, but it doesn't *seem* like
rocket-science, and making the data-type internals opaque would
provide a lot of freedom for future internal changes, and would make
the codebase *much* more approachable.

As an added IMO *way-cool* feature, I wonder if it would be possible
to define an interface that was both elisp and scheme friendly, and
better yet, use a backend storage format that was too (i.e. have the
forms where you store the state stick to a common "form" subset that
both parsers can handle).  If you could do that, then you could
implement the gnus-engine API in both languages and then use emacs to
deal with your mail interactively, and guile (or perhaps even stalin)
for batch mode processing.

To go *way* out in fantasy-land, I wonder if you could implement the
library using guile-friendly code (or a compatibility lib/layer) that
the elisp interpreter was also happy with.  If so, then you'd also be
ready if/when emacs does switch to the guile interpreter.

OK, I'll go take my medication now :>

(Seriously, if any of this strikes a chord with someone out there, let
 me know.  I *would* be interested in spending some time working on
 some/all of these bits.)

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: "Fixing up" gnus - (how hard is this?).

[Rob Browning]

prj@po.cwru.edu (Paul Jarc) writes:

> nnmaildir writes its state file frequently, but it is one monolithic
> file per group.  I may move to some form of journaling, but I've got
> other changes to get to first.

One simple thing to do is to always write the state file to a temp
file then do the link/unlink trick to make sure there's always a
coherent copy of the file in place.  This fixes corruption problems,
but not data-lossage.

> User-defined, you mean?  That would be useful indeed.  Obviously, it
> requires cooperation from Gnus, though.

Absolutely.  I tend to feel that gnus' current set of marks is sorta
ad-hoc when considered in the mail context.  While I like the idea of
using the same tool for mail and news, I think that mail does have
some special semantics that ought to be more explicitly accomodated,
and well documented.

Thanks for the responses.
-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Paul Jarc]

Rob Browning <rlb@cs.utexas.edu> writes:
> I'd like to go even farther and have gnus write out log/journal
> snippets every time you take an action, or have it do atomic writes of
> group state info (using link/unlink tricks) so that it would be very
> hard to corrupt your state records.

nnmaildir's been doing that since the 2000-09-03 release. :)  But use
rename(), not link()/unlink(), for appropriate atomicity.  It's not so
critical when there are no concurrent readers, but it still saves you
a system call.  I haven't checked the Emacs sources, but I'm assuming
(rename-file) uses rename() when ok-if-already-exists is non-nil.

> As an added IMO *way-cool* feature, I wonder if it would be possible
> to define an interface that was both elisp and scheme friendly, and
> better yet, use a backend storage format that was too (i.e. have the
> forms where you store the state stick to a common "form" subset that
> both parsers can handle).

Hmm... nnmaildir's state files are non-Lisp text, parsed manually.  I
suppose (read)ing Lisp expressions would probably be faster, though.
I hadn't thought of that.


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Rob Browning]

prj@po.cwru.edu (Paul Jarc) writes:

> nnmaildir's been doing that since the 2000-09-03 release. :)  But use
> rename(), not link()/unlink(), for appropriate atomicity.  It's not so
> critical when there are no concurrent readers, but it still saves you
> a system call.  I haven't checked the Emacs sources, but I'm assuming
> (rename-file) uses rename() when ok-if-already-exists is non-nil.

Well the tricks I know of were designed to make sure you DTRT (as much
as possible anyway), even in the case where the files are on NFS
mounted volumes, but I don't know if you want to worry about that, and
it's possible that rename handles that already...

> Hmm... nnmaildir's state files are non-Lisp text, parsed manually.
> I suppose (read)ing Lisp expressions would probably be faster,
> though.  I hadn't thought of that.

Might be a lot easier for you to handle code-wise too.

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Paul Jarc]

Rob Browning <rlb@cs.utexas.edu> writes:
> prj@po.cwru.edu (Paul Jarc) writes:
> > But use rename(), not link()/unlink(), for appropriate atomicity.
> 
> Well the tricks I know of were designed to make sure you DTRT (as much
> as possible anyway), even in the case where the files are on NFS
> mounted volumes, but I don't know if you want to worry about that, and
> it's possible that rename handles that already...

AFAIK, NFS handles rename correctly.  In the case of concurrent
readers, you definitely want rename, since it guarantees that when a
file exists with the new name, it will be replaced atomically, so
readers will always find one file or the other unde that name.

> > Hmm... nnmaildir's state files are non-Lisp text, parsed manually.
> > I suppose (read)ing Lisp expressions would probably be faster,
> > though.  I hadn't thought of that.
> 
> Might be a lot easier for you to handle code-wise too.

Yes, I imagine so.


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Rob Browning]

prj@po.cwru.edu (Paul Jarc) writes:

> > Might be a lot easier for you to handle code-wise too.
> 
> Yes, I imagine so.

So stepping back for a moment --- you've been working on an nnmaildir
backend, right?  As it stands now, what are the advantages/differences
of that from nnmail?

Because now I'm wondering if I couldn't get most of what I want from
working an appropriate backend, and if you're already pretty familiar
with what's involved, I'm wondering if we might be able to work
together to document the backend interface better, and perhaps work up
from there to add improvements to the API and the rest of gnus to
support some of the things I've mentioned.

In fact, if we can do most I'd want with nnmaildir, I'd be happy to
just help you with that.  Whether or not that would be possible, I
suppose depends on exactly where the performance problems are and what
the nnmaildir requirements are...

Of course this all presumes we'd be doing things that others would be
happy with and things that wouldn't unduly break other people's code.

Thanks for the interest.
-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Paul Jarc]

Rob Browning <rlb@cs.utexas.edu> writes:
> So stepping back for a moment --- you've been working on an nnmaildir
> backend, right?

Right.  <URL:http://multivac.cwru.edu/nnmaildir/nnmaildir.el>.  (This
is a new URL; the old one will still work for a while too.  I'll also
get some documentation up there eventually.)

> As it stands now, what are the advantages/differences of that from
> nnmail?

Well, mail is kept in the maildir it was delivered to, to begin with,
so you don't have to worry about mbox From_ quoting, etc.  nnmail
backends can pull in mail from a maildir, but then store it
differently (but not all of them quote From_, of course).  See
<URL:http://cr.yp.to/proto/maildir.html>.  I haven't measured, but I'd
expect nnmaildir to compare well speedwise.  Each message is in a
separate file, and it generates a NOV line when a new message is first
seen, so it should be comparable to nnml.  While the server is open, a
structure is kept in memory, that holds the NOV lines associated with
each article; they're read from disk only at startup.  Files are never
rewritten, unless you edit an article, so the file's last-modified
time gives you the delivery time.  (Modulo an unsolved bug in
-request-{move,accept}-article.)  The file that stores group
information, including NOV lines, is written frequently and reliably -
the file ".nnmaildir" always exists (after it's been written once) and
it's always consistent.  I'm not very familiar with the nnmail
backends, so I'm guessing where the differences are, somewhat.

No splitting.  (This is probably the big reason that nnmaildir doesn't
inherit from nnmail.)  Each maildir is a group.  Assuming you use
qmail with maildir delivery, it's easy to direct message to the
appropriate maildir in the delivery process via .qmail files.  Mailing
list subscriptions are especially easy: as user prj, I control all
prj-anything addresses along with prj itself, so I subscribe prj-ding
to this list, and that address gets its own delivery instructions,
directing messages into my ding maildir.

The next version will store marks, in such a way that you should be
able to use other maildir readers on the same maildir between Gnus
sessions, and the two will see (some of) each others' marks.
(Assuming the other reader uses the "2," info format described in the
link above - I don't know if there are any such readers.)

> I'm wondering if we might be able to work together to document the
> backend interface better,

I'd especially like to see documentation of what triggers each backend
function, and what assumptions, if any, they can make about their
environment - current buffer, etc.


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Rob Browning]

prj@po.cwru.edu (Paul Jarc) writes:

> I haven't measured, but I'd expect nnmaildir to compare well
> speedwise.  Each message is in a separate file, and it generates a
> NOV line when a new message is first seen, so it should be
> comparable to nnml.

So I'm not an expert on maildir, but as I recall it keeps articles in
three subdirectories of the main directory right, and one of those
subdirs is where all the new articles go, so for this backend,
figuring articles will be in different locations depending on their
state, or do you just treat the new article subdir as the spool dir
and immediately move the articles elsewhere when gnus checks for mail?

> While the server is open, a structure is kept in memory, that holds
> the NOV lines associated with each article; they're read from disk
> only at startup.

I'm not sure I know exactly what's in the NOV lines, but presuming
it's just article info, as long as you have some "safety-net" code in
there, this might even be safe in the context of multiple gnus
accessing the same group.  You'd just detect the changes the next time
you tried to access something that had changed...

> Files are never rewritten, unless you edit an article, so the file's
> last-modified time gives you the delivery time.

Is this an efficiency issue?  It seems a little risky.  If I tar up my
directory, and forget to use "p", or if the system clock is screwy, I
get the wrong answer, or is delivery time not something that's
normally in the article file?  An alternate approach would be to just
use caching of some flavor.  Store the date in a file in
groupdir/.nnmaildir (or whatever).

> (Modulo an unsolved bug in -request-{move,accept}-article.)  The
> file that stores group information, including NOV lines, is written
> frequently and reliably - the file ".nnmaildir" always exists (after
> it's been written once) and it's always consistent.

This sounds pretty good, but would it be helpful to make
groupdir/.nnmaildir a directory rather than a file?  We did that for
gnucash's user startup/config files, and it made keeping track of
things somewhat cleaner - separating state data from config data in
separate files, making parsing easier because the contents of each
data file can be more limited, etc.  FWIW.

> No splitting.  (This is probably the big reason that nnmaildir
> doesn't inherit from nnmail.)  Each maildir is a group.

So how/when does the user see new groups?  Do they have to manually
add each one?  It might sound crazy, but I had always kinda wanted a
backend that was more like dired.  I'd just see what's in the
(sub)tree and could manipulate things however I like, either from
gnus, or from the shell.  In that situation, you could either have a
manual update button to tell gnus to detect any changes (deleted/moved
groups), etc., or have gnus do the equivalent of a diff on "find
my-mail-tree -type d" every now and then to see if it needs to rescan.

> The next version will store marks, in such a way that you should be
> able to use other maildir readers on the same maildir between Gnus
> sessions, and the two will see (some of) each others' marks.
> (Assuming the other reader uses the "2," info format described in
> the link above - I don't know if there are any such readers.)

Looking at the spec, this sounds good, though it might be nice to have
two files in the long run, one that stores the "2," format as it
evolves and another that's a superset of that so if you want to store
more info, info that other mailers can't necessarily understand, you
have somewhere to keep it out of harms way (in other words, the "2,"
file would be derived from the data in the superset file which would
probably be elisp/scheme forms).  Though I suppose you can worry about
that once you find something you want to put in the superset...

In any case, it sounds like when you do this, the data in .newsrc.eld
will be irrelevant for your backend.  Will gnus automatically prefer
the data in the backend to the .newsrc.eld file?

> I'd especially like to see documentation of what triggers each backend
> function, and what assumptions, if any, they can make about their
> environment - current buffer, etc.

Yep.  It would be nice to have a well-documented interface (even if we
had to write a little bit of "adapter" code) that all of the backends
could be modified to conform to.  It would also be nice if things were
set up so that for similar backends, there was a smaller set of
functions that had to be written to create each one.  There might be a
set of default functions for things like article-state->form
form->article-state, etc.  You wouldn't have to use these, but you
could if/when you didn't need anything more tailored.

-- 
Rob Browning <rlb@cs.utexas.edu> PGP=E80E0D04F521A094 532B97F5D64E3930

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Paul Jarc]

Rob Browning <rlb@cs.utexas.edu> writes:
> So I'm not an expert on maildir, but as I recall it keeps articles in
> three subdirectories of the main directory right,

Yes: tmp/, new/, and cur/.  Messages are delivered by writing them to
new files in tmp/.  Once they're completely written, they get linked
into new/, and are then considered delivered.  MUAs look for new
messages in new/ and rename them into cur/.  nnmaildir does the
renaming in -request-scan, and doesn't mark an article as read or not
depending on whether it's in new/ or cur/.  It could get by just as
easily without moving articles at all.

> I'm not sure I know exactly what's in the NOV lines,

Article number, Subject:, From:, Date:, Message-ID:, References:,
number of characters in message, number of lines in body.

> as long as you have some "safety-net" code in there, this might even
> be safe in the context of multiple gnus accessing the same group.

Not at present, but I think that will be true to some degree in the
future, as an accidental result of changes I'm planning.  The
information about the messages is checked at startup - the existence
of the file is verified - but after that, if a message gets deleted
from outside Gnus, and then you try to read it, you get an error, and
nnmaildir doesn't condition-case it.

> > Files are never rewritten, unless you edit an article, so the file's
> > last-modified time gives you the delivery time.
> 
> Is this an efficiency issue?  It seems a little risky.

It's a matter of treating message files as sacred.  Delivery times
could also be stored somewhere else, but they're never used in Gnus.

> is delivery time not something that's normally in the article file?

It isn't, although the mbox format adds it in the From_ line.

> This sounds pretty good, but would it be helpful to make
> groupdir/.nnmaildir a directory rather than a file?

I don't think so.  .nnmaildir contains two pieces of information about
the group (name and largest article number), and then it's one line
per message, containing the filename and NOV data.  Fairly
homogeneous.

> > No splitting.  (This is probably the big reason that nnmaildir
> > doesn't inherit from nnmail.)  Each maildir is a group.
> 
> So how/when does the user see new groups?  Do they have to manually
> add each one?

At present, yes.  An nnmaildir select method looks like:
'(nnmaildir "server name" (nnmaildir-groups ("/path/to/maildir"
                                             "/path/to/another")))
and each maildir is a group.  But I think I'll change it to something
like this:
'(nnmaildir "server name" (nnmaildir-directory ("/path/to/dir")))
where /path/to/dir contains maildirs, or symlinks to them.  New groups
could be detected in -request-scan, and -request-create-group would
become possible.

> Looking at the spec, this sounds good, though it might be nice to have
> two files in the long run,

The "2," info is stored in the filenames themselves.  I plan to store
the 4 maildir marks there, and also all marks in .nnmaildir.
.nnmaildir will be updated when the server is opened to reflect
changes in marks made by a different mail reader, and files will be
renamed to reflect changes in marks made by Gnus, so those changes
will be visible to other mail readers.

> In any case, it sounds like when you do this, the data in .newsrc.eld
> will be irrelevant for your backend.  Will gnus automatically prefer
> the data in the backend to the .newsrc.eld file?

The backend can provide a -request-set-info function, which will be
called to correct Gnus's notion of marks and such.

> It would also be nice if things were set up so that for similar
> backends, there was a smaller set of functions that had to be
> written to create each one.

Moreso than nnoo allows, you mean?


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Kai Großjohann]

On 19 Oct 2000, Paul Jarc wrote:

> I'd especially like to see documentation of what triggers each
> backend function, and what assumptions, if any, they can make about
> their environment - current buffer, etc.

It took me a while to understand the documentation, but I think that
the Gnus info file does mention all the important things.  Or do you
find the node `Backend Interface' deficient in some way?

There was something about the `data buffer' which I didn't understand
right away.  The key issue is to understand that `there should be no
data returned' does NOT refer to the return value of the function, but
to the data buffer.

kai
-- 
I like BOTH kinds of music.

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> On 19 Oct 2000, Paul Jarc wrote:
> > I'd especially like to see documentation of what triggers each
> > backend function, and what assumptions, if any, they can make about
> > their environment - current buffer, etc.
> 
> It took me a while to understand the documentation, but I think that
> the Gnus info file does mention all the important things.  Or do you
> find the node `Backend Interface' deficient in some way?

I had to throw in some (message)s and such to see exactly when each
backend function is called.  Without that information, it's hard to
figure out which function should gather new mail in the absence of
-request-scan, for example.


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Kai Großjohann]

On 20 Oct 2000, Paul Jarc wrote:
> 
> I had to throw in some (message)s and such to see exactly when each
> backend function is called.  Without that information, it's hard to
> figure out which function should gather new mail in the absence of
> -request-scan, for example.

Hm.  Hmmm...  Ah, yes, your MTA delivers in new/, and you want to know
when to move messages from new/ to cur/, right?  Hm.  I guess you get
to decide when to do that.  For example, you could do it in
-request-group, or you could do it in ...  hm.  No, I think it should
be in -request-group.

I have never really looked what happens, though.

kai
-- 
I like BOTH kinds of music.

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> Hm.  Hmmm...  Ah, yes, your MTA delivers in new/, and you want to know
> when to move messages from new/ to cur/, right?

Yes, just like nnmail chooses a time to move mail from the spool to
~/Mail (or wherever).

> Hm.  I guess you get to decide when to do that.  For example, you
> could do it in -request-group, or you could do it in ...  hm.  No, I
> think it should be in -request-group.

Well, I have nnmaildir-request-scan now, but before that, I was doing
it in -request-list, which seemed to work well enough.  Before that, I
had it in -retrieve-headers, because I couldn't deduce from the
documentation that that was a bad place to do it.


paul

Re: "Fixing up" gnus - (how hard is this?). (was Re: (provide 'nnmaildir))

[Kai Großjohann]

It really seems the docs could do with some updating.  If you could
provide a little input, I could put it in the manual.

(I think if you were to write something substantial for the manual
yourself, you'd need to sign a copyright assignment first.
`Substantial' is defined as more than 10 lines of nontrivial code for
the code case -- not sure how to adapt it to documentation.)

kai
-- 
I like BOTH kinds of music.