(provide 'nnmaildir)

(provide 'nnmaildir)

[Paul Jarc]

I'm writing nnmaildir.el so I can take full advantage of
qmail/maildir, and of Gnus.  (See <URL:http://cr.yp.to/proto/maildir.html>.)
I've fumbled along to the point where I can read messages (woohoo!),
but it's still pretty primitive (e.g., no groups yet), as I'm not
overly familiar with the Gnus backend interface, and some of the stuff
I've done will probably have to be thrown out/redesigned later.  I've
been developing on emacs 20.3 and Gnus 5.6.45.  (Has the backend
interface changed since then?)  Here's what I'm doing:

- The maildir will be a full-fledged backend, not just a source to
  gather mail from to be stored somewhere else.  I've seen in the list
  archives that that has already been done, but I say it subverts
  much of the design of maildir and loses many benefits.  So I'm using
  nnoo and nnheader, but not nnmail.  (I might be able to use some
  stuff from nnmail, but probably nothing that needs to know where
  messages are stored.)

- Message files will never be rewritten.  The data in the files and
  the last-modified times are to be treated as precious.  I've thought
  about encoding NOV lines in the filenames for easy access, but that
  might get a bit unwieldy.  (I do add Lines: when writing the full
  headers to nntp-server-buffer, but I don't write it back to the
  message file.)  I could just write the NOV lines to a separate
  per-maildir file, but that's so boring.  It'd complicate message
  deletion, too.

- I've been using the message files' inode numbers as article
  numbers.  It seems this won't be viable, though, since messages are
  sorted by article number in the summary buffer.  In
  nnmaildir-retrieve-headers, I tried doing my own sorting of the
  requested articles and writing the headers to nntp-server-buffer in
  chronological delivery order, but they got resorted. :(  Also, Gnus
  might get the idea that I've read articles 1-5000 just because I
  happened to read article 5000, which was the lowest numbered article
  at the time.  If a lower-numbered article arrives later, I won't see
  it.  (Sadly, this (or something similar) actually happens with my
  NNTP server.)  So I may have to generate article numbers some other
  way. :(

So... what conditions are article numbers supposed to satisfy?  What
do NNTP servers do when they run out (I'm assuming they don't use
bignums), and how does Gnus react?

The server argument to the backend functions is the second element of
the select method, right?  A string, typically?  And as the author of
the backend, I decide what it means?  (Right now, I'm using it to
specify the path to the maildir, but I think I'll create a backend
variable for that purpose instead.)

I'm unclear about the usage of nnoo-change-server and
nnoo-push-server.  I've just been working from (un/sparsely commented)
examples in the various nn*.el to guess where to use each.  If I
define a backend variable like (defvoo nnmaildir-maildir "~/Maildir/")
and use a select method like
'(nnmaildir "blah" (nnmaildir-maildir "/path/to/maildir")) then my
understanding is that nnoo-change-server will, among other things, pick
up the value of nnmaildir-maildir from the defs argument, and that
value will be visible to nnmaildir-open-server (i.e., the caller of
nnoo-open-server).  Is this right?  What about nnoo-push-server?

Can I see a concrete example of what a real-life NOV line would look
like, as written to nntp-server-buffer?  Can I get away with using
article numbers for references there?  Why does gnus-nov-is-evil
exist?  Obeying it might be mildly troublesome/expensive.  Does the
`chars' field include the whole message, or just the body?  `lines' is
just the body, right?

How does Gnus know whether a backend supports article identification
by article number ranges or Message-IDs?  I guess backends must fail
sensibly if they don't support these, hm?

I'm thinking of implementing groups via something like
nnmail-split-fancy.  I've also thought of having a backend variable,
say, nnmaildir-hierarchy, that all groups would be organized under.
I.e., if nnamildir-hierarchy were set to "mail", and a message was
determined to belong to group "foo", it would actually be put in
"mail.foo".  (I suppose I could use the server name instead of a
backend variable.)  Is this group name prefixing a bad idea?

I think those are all the issues I've run into so far, but I haven't
made a go at any of the optional backend functions yet, so I'm sure
I'll be back with more questions later.


paul

Re: (provide 'nnmaildir)

[Steve Harris]

Paul,

Please be sure to browse two threads from April[1] and May[2]
regarding the horrendous problems I've had with nnimap and maildir. In
particular, check out Simon Josefsson's replies to the first[3] and
second[4] threads. I hope your efforts don't trip up on the same
problems with maildir's message IDs that nnimap does.

Footnotes: 
[1] http://www.deja.com/[ST_rn=ps]/viewthread.xp?AN=615031955
[2] http://www.deja.com/[ST_rn=ps]/viewthread.xp?AN=615619337
[3] http://www.deja.com/[ST_rn=ps]/getdoc.xp?AN=616445834
[4] http://www.deja.com/[ST_rn=ps]/getdoc.xp?AN=616438699

-- 
Steven E. Harris
Primus Knowledge Solutions, Inc.
http://www.primus.com

Re: (provide 'nnmaildir)

[Paul Jarc]

Steve Harris writes:
> Please be sure to browse two threads from April[1] and May[2]
> regarding the horrendous problems I've had with nnimap and maildir.

Hmm... it looks like the article numbers were being generated from the
Message-IDs.  It hadn't even occurred to me to do that.  I won't.  I'm
safe. :)


paul

Re: (provide 'nnmaildir)

[Steve Harris]

prj@po.cwru.edu (Paul Jarc) writes:

> Hmm... it looks like the article numbers were being generated from the
> Message-IDs.  It hadn't even occurred to me to do that.  I won't.  I'm
> safe. :)

Simon, is Paul onto something here that would help nnimap as well?

-- 
Steven E. Harris
Primus Knowledge Solutions, Inc.
http://www.primus.com

Re: (provide 'nnmaildir)

[Simon Josefsson]

Steve Harris <sharris@primus.com> writes:

> > Hmm... it looks like the article numbers were being generated from the
> > Message-IDs.  It hadn't even occurred to me to do that.  I won't.  I'm
> > safe. :)
> 
> Simon, is Paul onto something here that would help nnimap as well?

Nnimap uses UIDs from the server, which are garanteed to be
monotonously increasing, but not contigious or starting at a small
value (but that's often the case however) which Gnus prefers.  I can't
think of a better way to assign article numbers than using the UID.
Even more difficult would be to try to "guess" when UIDs have
characteristics that Gnus doesn't like, and convert them only in that
case.

Re: (provide 'nnmaildir)

[Steve Harris]

Can you envision what changes to the Gnus backend interface would be
necessary to allow nnimap/maildir's huge message IDs with huge gaps to
work okay? We ended one thread by saying that Oort Gnus might be the
time frame, but didn't specify _what_ would be changed. If nnimap is
the only backend that would benefit, would that be sufficient
justification to put the work into Gnus? Is there a history of changes
to Gnus to support a particular backend?

-- 
Steven E. Harris
Primus Knowledge Solutions, Inc.
http://www.primus.com

Re: (provide 'nnmaildir)

[Paul Jarc]

Steve Harris writes:
> Can you envision what changes to the Gnus backend interface would be
> necessary to allow nnimap/maildir's huge message IDs with huge gaps to
> work okay?

IIUC, no interface changes would be necessary.  The nnimap backend
would just use a different method of generating article numbers: do
its own sequence instead of using the numbers coming from the IMAP
server.  What might be nice is if Gnus would provide a function for
backends to use that increments and returns a per-server+newsgroup
counter, saving it in newsrc.eld between sessions, but the backends
can already get the same functionality with more work now.


paul

Re: (provide 'nnmaildir)

[Kai Großjohann]

On Mon, 17 Jul 2000, Paul Jarc wrote:

> [...] What might be nice is if Gnus would provide a function for
> backends to use that increments and returns a per-server+newsgroup
> counter, saving it in newsrc.eld between sessions, but the backends
> can already get the same functionality with more work now.

I think storing information on the client side is not a good idea for
nnimap.  Or am I misunderstanding something, and there won't be a
problem when using two instances of Gnus to talk to the same IMAP
server (and account)?

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> > Can you envision what changes to the Gnus backend interface would be
> > necessary to allow nnimap/maildir's huge message IDs with huge gaps to
> > work okay?
> 
> IIUC, no interface changes would be necessary.  The nnimap backend
> would just use a different method of generating article numbers: do
> its own sequence instead of using the numbers coming from the IMAP
> server.

But it need to use the server's sequence number to be able to fetch a
article, find out what flags the article got, search for articles and
stuff.  It's not up to nnimap to decide what article number a article
got, that's server assigned.

It might translate back and forth between Gnus-article# and
IMAP-article# but that would be really slow, and the whole point of
having state in the server is lost.

Re: (provide 'nnmaildir)

[Steve Harris]

Simon Josefsson <simon@josefsson.org> writes:

[...]

> It might translate back and forth between Gnus-article# and
> IMAP-article# but that would be really slow, and the whole point of
> having state in the server is lost.

Simon,

Having seen your reiteration above, do you have any answers to my
question of what needs to change in Gnus to support _proper_ operation
of nnimap with maildir-inspired IDs? Is it even possible?

-- 
Steven E. Harris
Primus Knowledge Solutions, Inc.
http://www.primus.com

Re: (provide 'nnmaildir)

[Simon Josefsson]

Steve Harris <sharris@primus.com> writes:

> > It might translate back and forth between Gnus-article# and
> > IMAP-article# but that would be really slow, and the whole point of
> > having state in the server is lost.
> 
> Simon,
> 
> Having seen your reiteration above, do you have any answers to my
> question of what needs to change in Gnus to support _proper_ operation
> of nnimap with maildir-inspired IDs? Is it even possible?

The maildir ID situation doesn't apply -- we don't get to assign
article numbers like nnmaildir can.  (If I'm wrong about this, please
describe how it would work, I can't see it.)

There are at least four solutions to the large-UIDs-from-IMAP-server
problem (I'm not saying they are good or realistic):

. Make elisp integers hold at least 32 bit integers and re-write
  gnus-range.el in C to make it fast.

. Implement the translation back and forth between gnus-article number
  and imap-article number, within nnimap.

        This is ``hairy''.  Remember -- the IMAP library uses elisp
        integers when dealing with articles so you'd basicly have to
        implement a (efficient) bignum-implementation and make it work
        within the IMAP parser.  When this is done you'd need, in
        nnimap, somehow convert the 1..2^32 range into the 1..2^28
        range, with the properties of starting at a low value and
        being monotonously increasing and contigious, that Gnus can
        handle.

. make Gnus accept non-integer article references.

        This might be a good thing anyway (think of using URLs to
        reference articles), but you'd still need to do most of the
        work in the previous point though.

. change/modify your server.

Since nnmaildir seem to be coming, it might be a more realistic
solution for you.

Re: (provide 'nnmaildir)

[Kai Großjohann]

I think that it would be cool to have bignums in Emacs Lisp.  I also
think that Gnus should be modified to use a data type for article
numbers and to fall back gracefully when some operations cannot be
done for article numbers, such as compressing a range, or computing
the number of messages in a range.

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Steve Harris]

Simon Josefsson <simon@josefsson.org> writes:

[...]

> There are at least four solutions to the large-UIDs-from-IMAP-server
> problem (I'm not saying they are good or realistic):
> 
> . Make elisp integers hold at least 32 bit integers and re-write
>   gnus-range.el in C to make it fast.
> 
> . Implement the translation back and forth between gnus-article number
>   and imap-article number, within nnimap.

[...]

> . make Gnus accept non-integer article references.
> 
>         This might be a good thing anyway (think of using URLs to
>         reference articles), but you'd still need to do most of the
>         work in the previous point though.

Yes, I was thinking of something like putting the huge UIDs in a hash
table as strings, or building a map of

  UID (as string) => small(er), contiguous number

That is, to stop thinking of the UID as a number. I can see how that's
similar to the "translation" bullet above.

Is the problem that nnimap eventually has to send ranges of these
UIDs, and thus must understand them *as numbers* at some point?

Also, does nnimap care - so far as talking to Gnus - whether one UID
is greater than another? That is, what is the most salient property of
the UID - its uniqueness or its comparative relation to other UIDs? If
it's the former, then maybe we can try kicking more holes in the
"treat-UID-as-string" idea.

Are there other protocols that Emacs has trouble with due to its
28-bit integers? I noticed that the IMAP4 spec. requires 32 bits for
both UIDs and the UIDVALIDITY number. How does nnimap handle
UIDVALIDITY? Is it just a string compare to see if it changed?

> . change/modify your server.
> 
> Since nnmaildir seem to be coming, it might be a more realistic
> solution for you.

This one's confusing. Will nnmaildir be able to talk IMAP, or do you
mean that nnmaildir would work for a local maildir spool? Since I'm
connecting to my mail host from afar, I need IMAP.

-- 
Steven E. Harris
Primus Knowledge Solutions, Inc.
http://www.primus.com

Re: (provide 'nnmaildir)

[Paul Jarc]

Steve Harris <sharris@primus.com> writes:
> Is the problem that nnimap eventually has to send ranges of these
> UIDs, and thus must understand them *as numbers* at some point?

Well, nnimap has to use *something*, treat it as a number (since
that's what Gnus expects), and preferably be able to handle ranges
sent back to it by Gnus.  Courier's UIDs aren't an ideal choice of
this something because of their low density.  But there must at least
be a mapping between nnimap's article numbers and IMAP's UIDs.

> Also, does nnimap care - so far as talking to Gnus - whether one UID
> is greater than another?

I don't know whether nnimap cares, exactly, but Gnus cares: article
numbers (which currently are UIDs) must be increasing - the article
must be numbered in the order they arrive.

> > Since nnmaildir seem to be coming, it might be a more realistic
> > solution for you.
> 
> This one's confusing. Will nnmaildir be able to talk IMAP, or do you
> mean that nnmaildir would work for a local maildir spool? Since I'm
> connecting to my mail host from afar, I need IMAP.

nnmaildir will work only for local maildirs - for some value of
`local'.  If you can NFS-mount your maildir on your local system,
nnmaildir should work for you.  I've fixed the new-messages-showing-
up-as-old bug.  I'm using it now for my mail reading, and it seems to
work.  Still incomplete, though: no groups or expiration yet.
<URL:http://multivac.cwru.edu/prj/nnmaildir.el>


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

I wrote:
> Still incomplete, though: no groups or expiration yet.
> <URL:http://multivac.cwru.edu/prj/nnmaildir.el>

Work on groups is progressing, along with fetching articles by
Message-ID.  Expiration will probably be next.

- Any advice on how to maintain (not necessarily persistent)
  per-server/per-group state between calls to backend functions?
  Maybe declare some server parameters, but advertise them as reserved
  for internal use?

- Any advice on what kind of information should be provided through
  server parameters, and what kind through package variables?

- Does Gnus give backends any way to save state between sessions in
  .newsrc.eld or the like?

- When Gnus calls my -open-server, it passes the server parameters
  from the select method set in .emacs, or the like (which I relay to
  nnoo-change-server).  When it calls my other functions, does Gnus
  ensure that the server to be operated on is current, by first
  calling -open-server?  Or am I to call -open-server myself, if
  necessary?  If I call it, I don't have the server parameters - will
  nnoo-change-server remember them from the first call with a given
  server name?  If not, how do I reestablish the proper values of
  server parameters?

- For moving articles between groups in nnmaildir, it makes sense to
  simply rename a message file and/or tweak some per-server persistent
  state data - in particular, *not* to copy a message from one file to
  another.  Can I work this into the -request-move-article and
  -request-accept-article interface, or is this too different of a
  framework?

- What's the likelihood of nnmaildir being officially incorporated
  into Gnus?

- I read on the list archive a while back that there might be some
  reworking of nnmail to be more flexible in dealing with mail
  backends (which might mean that nnmaildir could be a nnmail
  backend).  Is this still in the works?  Or maybe did it happen, and
  I missed it?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai =?iso-8859-1?q?Gro=DFjohann?=) writes:
> Don't all the nnchoke-* functions accept a server parameter which
> contains the necessary information?

The server parameter is just the name - the second element of the
select method.  The optional defs parameter of the -open-server
function is the cddr of the select method, which contains all the
server parameters.  Gnus will first call the backend's -open-server
function, and it will pass the server parameters in defs.  The
-open-server function (typically) then relays the defs to
nnoo-change-server, which remembers them for later.  If you call
nnoo-change-server again with the same server but with defs being nil,
it will remember the old server parameters and do the appropriate setq
assignments.  So all is well for me.

> On 24 Jul 2000, Paul Jarc wrote:
> > - For moving articles between groups in nnmaildir, it makes sense to
> >   simply rename a message file and/or tweak some per-server
> >   persistent state data - in particular, *not* to copy a message
> >   from one file to another.

I've redesigned my multiple-groups approach, so this is no longer an
issue.  A new version is here, and I think it's (almost) ready for
general use.  Important things still to be done: expiration, moving
articles between groups.
<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
bugfix: new mail is now incorporated by nnmaildir-request-list
        instead of nnmaildir-retrieve-headers
feature: headers and articles can be fetched by Message-ID
feature: nnmaildir-retrieve-headers handles numeric fetch-old
feature: multiple groups per server; one group per maildir
feature: nnmaildir-directory is gone; nnmaildir-groups maps
         group names to maildirs

Having only one group per maildir simplifies things greatly in
nnmaildir; .qmail files can be used to filter messages into the
appropriate places.  A select method now looks like this:
'(nnmaildir "whatever"
            (nnmaildir-groups (("groupname1" "/path/to/maildir1")
                               ("groupname2" "/path/to/maildir2"))))
If there is only one group, it doesn't need to be put in a list.  The
select method is destructively modified behind your back by
nnmaildir-open-server - I don't think this will cause problems, will
it?  I think nntp.el does it too.

If you were using the old version, migration will be mildly painful.
(But it shouldn't be in the future.)  First, kill (not just
unsubscribe) all your nnmaildir groups.  Now, in the cur/ directory in
each of the maildirs you're using, execute this command:
bash -c 'for i in *; do mv "$i" ../new/"${i%%:*}"; done'
That will make all messages appear as new again.  Now set up your new
select method, fire up Gnus, and resubscribe to your groups.

nnmaildir-retrieve-headers doesn't look outside the specified group,
whereas nnmaildir-request-article does.  Will this cause problems?

The summary buffer will show some articles as having zero lines.  I
just now figured out how to fix this.


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

I wrote:
> Important things still to be done: expiration, moving articles
> between groups.  <URL:http://multivac.cwru.edu/prj/nnmaildir.el>

Halfway there, and then some.
bugfix: narrow search for existing Lines: to just this article
bugfix: request-list was reporting the lowest article number of
        new, empty maildirs as 1, and the largest as 0
internal: request-type - returns `mail'
feature: request-expire-articles, controlled by
         nnmail-expiry-wait{,-function}
feature: request-rename-group - requires user to update select
         method manually

Since group names are defined in the select method, and not stored in
the maildir, request-rename-group can only do so much.  It'll update
the select method and other data structures in memory, but the
definition in the user's .emacs must be updated manually.  Maybe I
should rethink this approach to group naming?

To migrate from yesterday's version to today's, just remove the
cur/.nnmaildir-min-* files from your maildirs.  They're no longer
used.  Don't touch the .nnmaildir-max-* ones, though.

I'm going to speed up range processing by reusing article numbers.
I'll use request-update-info to reassign article numbers, keeping
them small and consecutive.  Do any existing backends do this?  Even
if the reassignment is too expensive to be worthwhile, I expect there
would be some benefit in forgetting the readness of articles less than
the smallest existing article.  Even nntp.el could do that much.

The expirable mark seems not to persist between sessions for articles
that haven't expired yet.  Is the backend supposed to maintain these
marks?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

bugfix: nnheader-fold-continuation-lines disappeared; it's back
bugfix: open-server would trigger an error when reporting one
feature: request-delete-group - also requires user assistance


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
bugfix: -request-expire-articles couldn't default to current group
internal: -i-curgrp is now defvoo'd instead of a server-indexed
          alist, and refers to a -groups entry instead of a string

BTW, old versions, if anyone cares, are available as
nnmaildir.el.2000-07-20, -07-31, ....  If you've been waiting for
maturity so you won't have to mess around with my data in your
maildirs to upgrade to newer versions - wait one more day. :)


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
internal: .nnmaildir-max is now a symlink, no longer in cur/
bugfix: -open-server no longer stomps on your select method

To migrate from earlier versions, execute this command in each maildir
you're using with nnmaildir:
bash -c 'for max in cur/.nnmaildir-max-*;do ln -s "${max##*/.*-}" .nnmaildir-max;rm "$max";done'
I expect the on-disk storage will not undergo any more changes; future
upgrades should not require more than restarting emacs.

Still on the todo list: moving articles between groups and renumbering
articles with -request-update-info.  There are no known outstanding
bugs.

I haven't tried using nnmaildir with XEmacs, so I'm especially
interested in hearing about problems there.  Are there any online
guides to writing code portable between Emacs & XEmacs?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

prj@po.cwru.edu (Paul Jarc) writes:
> Still on the todo list: moving articles between groups and renumbering
> articles with -request-update-info.

I've got -request-update-info working, but if I just start Gnus and
immediately quit, Gnus doesn't save my .newsrc.eld, even though I
changed the group info.  How do I tell Gnus, from within
-request-update-info, that the .newsrc.eld need to be saved at the end
of the session?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
internal: -request-list runs faster, updating max only once
feature: -request-move-article
bugfix: -request-article could have returned an article from
        another group when requesting a nonexistent article by
        article number

So now you can move articles out of nnmaildir groups, but not yet into
them.

Does nnmbox actually use the information in the `From ' line?  If so,
then nnmbox-request-accept-article, when extracting the envelope
sender, ought to check for Return-Path: as the first field if it
doesn't find X-From-Line:.  qmail delivers a message to a maildir with
the envelope sender (in <angle brackets>) in Return-Path: as the first
field, and nnmaildir preserves this.  The delivery time could be taken
from the third field (Received:), or perhaps from the filename, which
nnmaildir-request-move-article makes available as nnmaildir-filename.

nnmbox ought to remove the `From ' line when moving articles rather
than changing it to a X-From-Line: field.  Otherwise, moving an
article from an nnmaildir group to an nnmbox group and back results in
an X-From-Line: field being added.  It'd be much cleaner if the
article ended up containing the same data that it started
with.  -request-move-article is supposed to tidy up, removing anything
that isn't actually part of the article, and the `From ' line isn't
part of the article; it's part of the mbox storage format.  `>From '
quoting is also part of the mbox format, and should be cleaned up,
probably in nnmbox-request-article.
(replace-regexp "^>\\(>*From \\)" "\1")

> I expect the on-disk storage will not undergo any more changes; future
> upgrades should not require more than restarting emacs.

Uh, scratch that.  There will be another change when I get into NOV.


paul

Re: (provide 'nnmaildir)

[Kai Großjohann]

On 11 Aug 2000, Paul Jarc wrote:

> nnmbox ought to remove the `From ' line when moving articles rather
> than changing it to a X-From-Line: field.

Since the From_ line contains useful information, I'd rather not have
Gnus throw it away.

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Paul Jarc]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> On 11 Aug 2000, Paul Jarc wrote:
> > nnmbox ought to remove the `From ' line when moving articles rather
> > than changing it to a X-From-Line: field.
> 
> Since the From_ line contains useful information, I'd rather not have
> Gnus throw it away.

Sure, that information should be kept - but should Gnus dictate that
it be kept in the message header, when it isn't necessarily part of
the message?  This mandates lossage, which is unfortunate.  Maybe
-request-move-article functions could set up variables containing the
envelope sender and delivery time (that's what's in the From_ line,
right?) and then the receiving -request-accept-article function could
put those values somewhere appropriate for that backend's storage
format.  (It would be cleaner to pass the values as arguments, but
that would require changes to Gnus itself, in the accept-form
business, instead of just the backends.)  For maildir, the delivery
time is the file's last-modified time.  In general, the envelope
sender isn't necessarily stored anywhere, although when qmail delivers
a message to a maildir, it adds a Return-Path: field for the envelope
sender.


paul

Re: (provide 'nnmaildir)

[Kai Großjohann]

A buffer-local variable (or two of them) to store the information from
the envelope sounds like a good idea.

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
internal: major rewrite for performance; .nnmaildir-max replaced
          with .nnmaildir, storing NOV lines & other info
internal: old tmp/ files are deleted in -open-server instead of
          -close-server (which has evaporated); tmp/ files with
          multiple hard links are also deleted
feature: -request-replace-article
feature: -groups is now just a list of maildirs; group names are
         stored in the maildirs
feature: -add-lines controls whether Lines: is added when missing
bugfix: -retrieve-headers never actually handled numeric fetch-old
bugfix: -request-{rename,delete}-group no longer require user help
bugfix: -request-expire-articles returns a correct article list

This version uses more memory and is somewhat slower in starting up
and assimilating new messages, but it's much faster in generating
Summary buffers and requesting articles.

An nnmaildir select method looks like this:
'(nnmaildir "whatever" (nnmaildir-groups ("/path/to/maildir"
                                          "/path/to/another")))
Or, for just a single group:
'(nnmaildir "whatever" (nnmaildir-groups "/path/to/maildir"))
If you don't want Lines: added to requested articles when it's missing:
'(nnmaildir "whatever" (nnmaildir-add-lines nil) (nnmaildir-groups ...))

Another somewhat painful migration from previous versions:
- start Gnus; kill all nnmaildir groups; exit Gnus; exit Emacs
- in each maildir, run:
  $ rm .nnmaildir-max
  $ cd cur
  $ bash -c 'for i in *;do mv "$i" ../new/"${i%%:*}";done'

(This same process can be used anytime to reassign article numbers
starting from 1, if they start to get too sparse for you.)  If you
don't want to lose your marks, you'll have to create the .nnmaildir
files by hand. :/ It'd probably be easier to save a copy of each
Summary buffer and re-mark the articles after starting up the new
version.

To implement -request-accept-article, I need to write the current time
into a string as a plain time_t (e.g., "968039378").  Will this
necessarily involve tedious string operations to do the arithmetic to
combine the two elements of (current-time), or has this already been
done somewhere?

I'd like to make nnmaildir interoperable with other maildir user
agents (not that *I* ever user any others, but...), so that other
maildir agents could be used on the same maildir as nnmaildir (between
Gnus sessions; probably not simultaneously).  This requires nnmaildir
storing some marks: read, expirable, replied, and any other single
mark.  I'd have to tell Gnus about marks it doesn't know about when
starting up (which is possible with -request-update-info), and I'd
also have to grab notification whenever any of those marks are
set/unset (only partially doable, with -request-update-mark).  The
backend interface would have to change somewhat to accomodate this
goal - but is it worthwhile anyway?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

prj@po.cwru.edu (Paul Jarc) writes:
> Another somewhat painful migration from previous versions:
> - start Gnus; kill all nnmaildir groups; exit Gnus; exit Emacs
> - in each maildir, run:
>   $ rm .nnmaildir-max
>   $ cd cur
>   $ bash -c 'for i in *;do mv "$i" ../new/"${i%%:*}";done'

Whoops - you'll also need to update your select method before using
the new version.  That's rather important.


paul

Re: (provide 'nnmaildir)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> To implement -request-accept-article, I need to write the current time
> into a string as a plain time_t (e.g., "968039378").  Will this
> necessarily involve tedious string operations to do the arithmetic to
> combine the two elements of (current-time), or has this already been
> done somewhere?

(format-time-string "%s")

> This requires nnmaildir storing some marks: read, expirable,
> replied, and any other single mark.  I'd have to tell Gnus about
> marks it doesn't know about when starting up (which is possible with
> -request-update-info), and I'd also have to grab notification
> whenever any of those marks are set/unset (only partially doable,
> with -request-update-mark).  The backend interface would have to
> change somewhat to accomodate this goal - but is it worthwhile
> anyway?

Nnimap does something similar, and uses the -request-set-mark
function.  Perhaps it can be of use to nnmaildir too?

Re: (provide 'nnmaildir)

[Paul Jarc]

Simon Josefsson <simon@josefsson.org> writes:
> (format-time-string "%s")

Perfect.  Thanks.

> > I'd have to tell Gnus about marks it doesn't know about when
> > starting up (which is possible with -request-update-info), and I'd
> > also have to grab notification whenever any of those marks are
> > set/unset (only partially doable, with -request-update-mark).
> 
> Nnimap does something similar, and uses the -request-set-mark
> function.  Perhaps it can be of use to nnmaildir too?

-request-set-mark isn't documented, AFAICT, at least not in the 5.7
info.  How does it work?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
feature: -request-accept-article
bugfix: -request-move-article now removes the in-memory record of
        a message along with the message file
bugfix: more error reporting
bugfix: generated Xrefs are now well-formed

Easy upgrade this time: just restart Emacs.  (Unless you want to fix
your old Xrefs, in which case you'll have to use the painful procedure
or edit the .nnmaildir files manually.)

There's a nonfatal bug in -request-{move,accept}-article.  -accept- is
supposed to use hard links whenever possible instead of copying the
buffer into a new file.  So it checks whether nnmaildir-filename is
bound, and if so, it uses add-name-to-file instead of write-region,
falling back to write-region if the maildirs are on different
filesystems.  -move- sets nnmaildir-filename to the name of the file
being moved, but -accept- isn't seeing it for some reason, so it
always copies instead of linking.  -move- binds nnmaildir-filename
inside a let, and it isn't buffer-local.  Can someone take a look and
see what I'm doing wrong?

Also, is there any way to get the effect of O_CREAT|O_EXCL in
write-region?  How about fsync?

Until now, I was able to use nnmaildir without declaring it with
gnus-declare-backend.  But moving articles between groups requires
that the destination group belong to a declared backend, it seems.  So
I'm using:
  (gnus-declare-backend "nnmaildir" 'mail 'respool)
Where does this belong?  I can't find any instance of this function,
other than the definition, in the Gnus source.  I have it in my .emacs
for now - should it be in nnmaildir.el instead?  Having it in .emacs
results in the creation of a *Group* buffer.  Evaluating the same form
after startup doesn't.  Hm.


paul

Re: (provide 'nnmaildir)

[Kai Großjohann]

On 21 Sep 2000, Paul Jarc wrote:

> Until now, I was able to use nnmaildir without declaring it with
> gnus-declare-backend.  But moving articles between groups requires
> that the destination group belong to a declared backend, it seems.
> So I'm using: (gnus-declare-backend "nnmaildir" 'mail 'respool)
> Where does this belong?

In nnir.el, the code starts with a few require statements and then:

/----
| (nnoo-declare nnir)
| (nnoo-define-basics nnir)
| (gnus-declare-backend "nnir" 'mail)
\----

That seems to work.  I don't understand what it does, but as long as
it works...

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai =?iso-8859-1?q?Gro=DFjohann?=) writes:
> In nnir.el, the code starts with a few require statements and then:
> 
> /----
> | (nnoo-declare nnir)
> | (nnoo-define-basics nnir)
> | (gnus-declare-backend "nnir" 'mail)
> \----

Ok, thanks.  I'll do that in the next release.


paul

Re: (provide 'nnmaildir)

[Simon Josefsson]

Paul Jarc <prj@po.cwru.edu> writes:

> I've been developing on emacs 20.3 and Gnus 5.6.45.  (Has the
> backend interface changed since then?)

No.

> So... what conditions are article numbers supposed to satisfy?

They must be a positive elisp integer, and article numbers should be
persistant between gnus sessions.

If you can make them small and contiguous, do so.  Gnus keeps
(un)compressing ranges and sorting them and stuff, and that is _slow_
even when the highest article number is a few hundred thousand.

> What do NNTP servers do when they run out (I'm assuming they don't
> use bignums), and how does Gnus react?

Gnus just barf at article numbers > 2^28.

> The server argument to the backend functions is the second element of
> the select method, right?  A string, typically?  And as the author of
> the backend, I decide what it means?  (Right now, I'm using it to
> specify the path to the maildir, but I think I'll create a backend
> variable for that purpose instead.)

Yes.  A good idea for the variable name would be `nnmaildir-directory'
to match nnml etc.

> I'm unclear about the usage of nnoo-change-server and
> nnoo-push-server.  I've just been working from (un/sparsely commented)
> examples in the various nn*.el to guess where to use each.  If I
> define a backend variable like (defvoo nnmaildir-maildir "~/Maildir/")
> and use a select method like
> '(nnmaildir "blah" (nnmaildir-maildir "/path/to/maildir")) then my
> understanding is that nnoo-change-server will, among other things, pick
> up the value of nnmaildir-maildir from the defs argument, and that
> value will be visible to nnmaildir-open-server (i.e., the caller of
> nnoo-open-server).  Is this right?

I think so.

>  What about nnoo-push-server?

I don't think you should use it.

> Can I see a concrete example of what a real-life NOV line would look
> like, as written to nntp-server-buffer?

747	(provide 'nnmaildir)	Paul Jarc <prj@po.cwru.edu>	14 Jul 2000 19:10:18 -0400	<m3ya3448bp.fsf@multivac.student.cwru.edu>		5909	85	Xref: barbar.josefsson.org ding:747	

> Can I get away with using article numbers for references there?

I don't think so, they should be taken from the References: header.

> Why does gnus-nov-is-evil exist?

Some NNTP servers are broken wrt NOV.

> How does Gnus know whether a backend supports article identification
> by article number ranges or Message-IDs?  I guess backends must fail
> sensibly if they don't support these, hm?

Gnus doesn't know, so yes, you need to take care of it.  Call
(nnheader-report 'nnmaildir "I don't support message-id fetching") or
something.

Re: (provide 'nnmaildir)

[Paul Jarc]

Simon Josefsson writes:
> Paul Jarc <prj@po.cwru.edu> writes:
> > So... what conditions are article numbers supposed to satisfy?
> 
> They must be a positive elisp integer, and article numbers should be
> persistant between gnus sessions.

So an article number for a particluar message must be constant for the
life of that message... after a message expires, can its article
number be reused?  My experiments indicate that nothing breaks
violently, but Gnus remembers the readness of the article even after
it expires, so a new article using the same number won't show up as
new. :(  It'd be nice if Gnus didn't mark numbers as read when they
have no article and are less than the number of a truly read article.

> > What do NNTP servers do when they run out (I'm assuming they don't
> > use bignums), and how does Gnus react?
> 
> Gnus just barf at article numbers > 2^28.

Ok... but do actual NNTP servers count into infinity?  They never
shift article numbers back to make room; do they ever wrap around to 0
(or 1)?  I would have thought that there must be a server out there
that's been in operation long enough to hit 2^32... or at least that
the authors of NNTP servers would take it into consideration.

> > (Right now, I'm using it to specify the path to the maildir, but I
> > think I'll create a backend variable for that purpose instead.)
> 
> Yes.  A good idea for the variable name would be `nnmaildir-directory'
> to match nnml etc.

Check, wilco.

> >  What about nnoo-push-server?
> 
> I don't think you should use it.

What I saw in other backends was that nnoo-change-server was used in
*-open-server, and nnoo-push-server was used everywhere else.  Should
I use nnoo-change-server in all cases?  What about backend functions
that don't take a defs argument?  How will they get the proper values
of the backend variables for the current server?  Or is *-open-server
always called first, if necessary, to make a server be current before
calling another backend function?

> > Can I get away with using article numbers for references there?
> 
> I don't think so, they should be taken from the References: header.

Yes, but what does Gnus do with them?  Construct threads, yes?  Is
this done is such a way that article numbers would work as well?  Or:
could Gnus be *made* to allow this? ;)  If I can keep article numbers
small, and if I can use them for References, then encoding NOV lines
in filenames might be feasible.

> > Why does gnus-nov-is-evil exist?
> 
> Some NNTP servers are broken wrt NOV.

So if my non-NNTP backend isn't broken, must I still obey it?

Since my last message, I've found more threads about maildir in the
archive (and haven't checked gnu.emacs.gnus yet), so I may yet
discover that this is a solved problem... particularly, I get the
impression there's some support in Pterodactyl Gnus.  I don't know yet
if it's just treating a maildir as an inbox, or if it's actually
*using* a maildir.


paul

Re: (provide 'nnmaildir)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> > > So... what conditions are article numbers supposed to satisfy?
> > 
> > They must be a positive elisp integer, and article numbers should be
> > persistant between gnus sessions.
> 
> So an article number for a particluar message must be constant for the
> life of that message... after a message expires, can its article
> number be reused?  My experiments indicate that nothing breaks
> violently, but Gnus remembers the readness of the article even after
> it expires, so a new article using the same number won't show up as
> new. :(

Yup.  Don't forget about the cache and the agent, they won't like two
articles having the same number.  So we have that Gnus article numbers
should be monotonously increasing, contigious and preferably start at
1.

> > >  What about nnoo-push-server?
> > 
> > I don't think you should use it.
> 
> What I saw in other backends was that nnoo-change-server was used in
> *-open-server, and nnoo-push-server was used everywhere else.  Should
> I use nnoo-change-server in all cases?

Gnus 5.8 doesn't seem to use nnoo-push-server at all, perhaps this has
changed.

> What about backend functions that don't take a defs argument?  How
> will they get the proper values of the backend variables for the
> current server?  Or is *-open-server always called first, if
> necessary, to make a server be current before calling another
> backend function?

I think so, yes.

> > > Why does gnus-nov-is-evil exist?
> > 
> > Some NNTP servers are broken wrt NOV.
> 
> So if my non-NNTP backend isn't broken, must I still obey it?

nnimap doesn't, but I'm not sure if anything gets hurt by this.

Re: (provide 'nnmaildir)

[Russ Allbery]

Paul Jarc <prj@po.cwru.edu> writes:

> So an article number for a particluar message must be constant for the
> life of that message... after a message expires, can its article
> number be reused?

No.  From draft-ietf-nntpext-base-09.txt:

    The server MUST ensure that article numbers are issued in order of
    arrival timestamp; that is, articles arriving later MUST have higher
    numbers than those that arrive earlier. The server SHOULD allocate the
    next sequential unused number to each new article.

There is no provision in the NNTP protocol for ever reusing an article
number.

>> Gnus just barf at article numbers > 2^28.

> Ok... but do actual NNTP servers count into infinity?

No.  The article number is required to fit in ten decimal digits, and in
practice is limited to 2^32 - 1.

> They never shift article numbers back to make room; do they ever wrap
> around to 0 (or 1)?  I would have thought that there must be a server
> out there that's been in operation long enough to hit 2^32... or at
> least that the authors of NNTP servers would take it into consideration.

It wasn't taken into consideration in the standard, and there are enough
protocol requirements that the NNTP server authors pretty much have their
hands tied.

So far, there has never been a Usenet newsgroup that's received sufficient
traffic to even come close to wrapping over the entire life of Usenet.
control.cancel is the leading contender, since for a while we were seeing
upwards of ten cancels a second, which would have wrapped the article
number in thirteen years of sustained activity at that level.  Thankfully,
since then spam has dropped off and so have the spam cancels.

At some point, a server might have to have a flag day and reset the
article number somehow, thereby confusing a lot of readers.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: (provide 'nnmaildir)

[Kai Großjohann]

On Fri, 14 Jul 2000, Paul Jarc wrote:

> So an article number for a particluar message must be constant for
> the life of that message... after a message expires, can its article
> number be reused?

No, don't reuse article numbers.

Hm.  But maybe you can implement this `ask backend for marks' stuff to
just reuse article numbers, where you just look at the list of
messages currently in the group and then number them from 1 to N.

I don't know how that `ask backend for marks' stuff works, though.
Simon should be the expert, here.

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Hm.  But maybe you can implement this `ask backend for marks' stuff to
> just reuse article numbers, where you just look at the list of
> messages currently in the group and then number them from 1 to N.

This function is only run sometimes, and in between, Gnus is using
it's locally stored .newsrc.eld flags.  So it wouldn't work very well.
It would be nice if Gnus didn't store flags locally but rather let the
backends handle them, but this would take a redesign of lots of stuff,
I think.

Re: (provide 'nnmaildir)

[Kai Großjohann]

On 14 Jul 2000, Paul Jarc wrote:

> - Message files will never be rewritten.  The data in the files and
>   the last-modified times are to be treated as precious.  I've
>   thought about encoding NOV lines in the filenames for easy access,
>   but that might get a bit unwieldy.  (I do add Lines: when writing
>   the full headers to nntp-server-buffer, but I don't write it back
>   to the message file.)  I could just write the NOV lines to a
>   separate per-maildir file, but that's so boring.  It'd complicate
>   message deletion, too.

I think doing the NOV thing is important to gain speed.  The Gnus
backend interface heavily relies on the data contained in NOV lines,
and there are quite highly optimized functions for dealing with NOV
lines.

nnimap has code for retrieving header data from the IMAP server and
then caching this data in NOV format on the local disk.  Maybe that
kind of approach would work for you, too?  The first time you enter a
group, you gather NOV lines for all messages, and on subsequent visits
you just read that cache file.

> - I've been using the message files' inode numbers as article
>   numbers.  It seems this won't be viable, though, since messages
>   are sorted by article number in the summary buffer.

Yes, article numbers must be monotonically increasing.  I you use a
NOV cache as described above, you could put the article numbers into
that file.  You could also store the maildir file name in the NOV
cache, and then you could get a list of files in the directory and a
list of file names from the NOV cache, and compare.  This would help
you find NOV lines for the new messages, and it would help you delete
stale lines from the NOV cache.

> So... what conditions are article numbers supposed to satisfy?  What
> do NNTP servers do when they run out (I'm assuming they don't use
> bignums), and how does Gnus react?

Gnus gets real mad when the article numbers are reset.  This is a weak
spot.  Gnus has not been designed for this.

You can use M-x gnus-group-clear-data RET and/or
gnus-group-clear-data-on-native-groups to make Gnus forget all article
numbers.  This means that all messages will be considered new, unread.

> The server argument to the backend functions is the second element
> of the select method, right?

I don't think so.  It just identifies a server.  There are two ways to
identify a server.  One way is to use a string, which is treated like
a name and then looked up in the list of all servers.  The other
possibility is to have a complete server definition, along the lines
of (nnml "frumple" (nnml-get-new-mail nil)), ie with server parameters
and all.

Luckily, it appears that you don't have to think about all this stuff
-- you can just use nnoo-change-server and then access local
variables, or stuff like this.

Hm.

Hmmm...  I have written a backend which seems to work, and I still
don't know what that server argument really means.  It appears that
nnoo is your friend, because it means you don't have to know all the
gory details.

> A string, typically?  And as the author of the backend, I decide
> what it means?  (Right now, I'm using it to specify the path to the
> maildir, but I think I'll create a backend variable for that purpose
> instead.)

Well, you could assign to it some meaning.  But I think it would be
better to just treat it as a name.  This way, users can decide what
their group names look like.

You see, if you specify (nnmaildir "frumple" ...) as a server
definition, then you will have group names like
nnmaildir+frumple:foo.bar.  If you specity (nnmaildir "" ...), ie an
empty server name, then the groups are named nnmaildir:foo.bar.

By `name' I mean that what you enter after a `B m' prompt, for
example.

> I'm unclear about the usage of nnoo-change-server and
> nnoo-push-server.  I've just been working from (un/sparsely
> commented) examples in the various nn*.el to guess where to use
> each.

I wish I knew...

> Can I see a concrete example of what a real-life NOV line would look
> like, as written to nntp-server-buffer?

Lessee:

1	Username/password for submission of ECDL paper	Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai =?iso-8859-1?q?Gro=DFjohann?=)	28 Apr 2000 16:17:50 +0200	<vafzoqewbqp.fsf@lucy.cs.uni-dortmund.de>		160	7	Xref: lucy.cs.uni-dortmund.de conf.y2000.ecdl:1	To: ecdl2000@bn.pt	

These are tab-separated fields.  The first field is the article
number, the second field is the Subject header, then comes the From
header, then the date, then the message id, and I'm not sure about the two
numbers 160 and 7.  Then comes the Xref header (I'm not sure what that
means), and the remaining fields can be any headers from the message.
(See gnus-extra-headers and nnmail-extra-headers for the remaining
fields.)

But this format is surely described somewhere in the Gnus info file?

> Can I get away with using article numbers for references there?

Gnus identifies messages by their numbers.

> Why does gnus-nov-is-evil exist?  Obeying it might be mildly
> troublesome/expensive.

Yes.  But if the user wishes to do that...

> Does the `chars' field include the whole message, or just the body?
> `lines' is just the body, right?

Err...

> How does Gnus know whether a backend supports article identification
> by article number ranges or Message-IDs?  I guess backends must fail
> sensibly if they don't support these, hm?

I think Gnus will barf if the backend doesn't support article numbers.
Maybe it's sufficient to just signal an error for message ids.

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Paul Jarc]

There's a crude but working version of nnmaildir at
<URL:http://multivac.cwru.edu/prj/nnmaildir.el>.  Feel free to have a
look and take it for a spin, but there will be incompatible changes
soon, so don't migrate just yet.

- The server name is the group that mail shows up in.  I know this is
  wrong now, so it'll change very soon - the server name will be
  treated as opaque data, and groups will be handled by something like
  nnmail-split-fancy.

- Article numbers are sequential, starting from 1.  When assigning
  numbers to new messages, nnmaildir-i-cur-max peeks at (among other
  things) gnus-newsrc-alist to get the highest article number known to
  Gnus.  Can someone look at this and tell me if I did this right?

- Problem: my .emacs contains (require 'nnmaildir), and after that,
  (setq nnmaildir-directory "~/var/mail").  nnmaildir.el contains
  (defvoo nnmaildir-directory "~/Maildir").  Before I start Gnus the
  first time, nnmaildir-directory has the value "~/var/mail", just
  like I want.  But when I start Gnus, it looks in "~/Maildir"
  instead.  Subsequently setting it back to my value doesn't help; it
  always gets changed back when I start Gnus.  Any ideas?

- This happened twice just now.  I had 4 new messages (a different set
  of 4 messages in each of the two trials).  I erased Gnus's memory of
  read articles for my mail group in my newsrc.eld.  When I started
  Gnus, it said there were 4 new messages.  When I selected the group,
  I saw all 4 messages in the summary buffer, but the first three were
  marked as old (`O').  When I quit the group without reading any
  messages, the group buffer said there was only 1 new message in that
  group.  Hmm.

Kai Großjohann writes:
> I think doing the NOV thing is important to gain speed.  The Gnus
> backend interface heavily relies on the data contained in NOV lines,
> and there are quite highly optimized functions for dealing with NOV
> lines.

Yes... but if nnmaildir-retrieve-headers returns 'headers instead of
'nov, won't Gnus just scan through that data and construct NOV lines
before any other processing?  IOW, it's no benefit to return NOV lines
unless I can do a better job of caching them than Gnus does of
extracting them, right?

> nnimap has code for retrieving header data from the IMAP server and
> then caching this data in NOV format on the local disk.  Maybe that
> kind of approach would work for you, too?  The first time you enter a
> group, you gather NOV lines for all messages, and on subsequent visits
> you just read that cache file.

Yeah, that'd work, but keeping that information in a separate file
means that expiring a message will be more work than just deleting a
single file.  So far, I've been able to get away with not keeping any
centralized state, in memory or on disk.  I'd like to avoid biting
that bullet, if possible.

> > The server argument to the backend functions is the second element
> > of the select method, right?
> 
> I don't think so.  It just identifies a server.  There are two ways to
> identify a server.  One way is to use a string, which is treated like
> a name and then looked up in the list of all servers.  The other
> possibility is to have a complete server definition, along the lines
> of (nnml "frumple" (nnml-get-new-mail nil)), ie with server parameters
> and all.

According to my experiments, the second element of a select method
*is* the `server' argument passed to the backed functions.  The third
element is the optional `defs' argument passed to *-open-server.  This
is typically relayed to nnoo-change-server, which picks up all those
name-value pairs and makes them real.  AFAICT.

> Hmmm...  I have written a backend which seems to work, and I still
> don't know what that server argument really means.  It appears that
> nnoo is your friend, because it means you don't have to know all the
> gory details.

Sure, but who's your friend when you *want* to know those details? :)

> > A string, typically?  And as the author of the backend, I decide
> > what it means?  (Right now, I'm using it to specify the path to the
> > maildir, but I think I'll create a backend variable for that purpose
> > instead.)
> 
> Well, you could assign to it some meaning.  But I think it would be
> better to just treat it as a name.  This way, users can decide what
> their group names look like.

Ok.  I'll do that.

> > Can I see a concrete example of what a real-life NOV line would look
> > like, as written to nntp-server-buffer?
> 
> Lessee:
> 
> 1	Username/password for submission of ECDL paper	Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann)	28 Apr 2000 16:17:50 +0200	<vafzoqewbqp.fsf@lucy.cs.uni-dortmund.de>		160	7	Xref: lucy.cs.uni-dortmund.de conf.y2000.ecdl:1	To: ecdl2000@bn.pt	
> 
> These are tab-separated fields.

So what happens when there's a tab in the subject?

> I'm not sure about the two numbers 160 and 7.

Chars and lines, preceded by references, empty in this case.

> Then comes the Xref header (I'm not sure what that means),

In NNTP, it contains the server name (without the domain, according to
RFC 1036, contrary to your example and others I've found) and
`newsgroup:article number' tuples.

> But this format is surely described somewhere in the Gnus info file?

Sort of.  The info doesn't say whether, e.g., I should write the
subject line verbatim, or if I should make it read-able as a Lisp
string.

> > Can I get away with using article numbers for references there?
> 
> Gnus identifies messages by their numbers.

But can it construct threads via article numbers directly, or does it
assume that references will be Message-IDs, as is the case with NNTP?

> > How does Gnus know whether a backend supports article identification
> > by article number ranges or Message-IDs?  I guess backends must fail
> > sensibly if they don't support these, hm?
> 
> I think Gnus will barf if the backend doesn't support article numbers.

Yes, but *ranges* (((1 . 4)) instead of (1 2 3 4)) are not universally
supported by backends, I think.  I'm using gnus-member-of-range now,
though, so I don't have to worry about that one.


paul

Re: (provide 'nnmaildir)

[Kai Großjohann]

On Sun, 16 Jul 2000, Paul Jarc wrote:

> - Problem: my .emacs contains (require 'nnmaildir), and after that,
>   (setq nnmaildir-directory "~/var/mail").  nnmaildir.el contains
>   (defvoo nnmaildir-directory "~/Maildir").  Before I start Gnus the
>   first time, nnmaildir-directory has the value "~/var/mail", just
>   like I want.  But when I start Gnus, it looks in "~/Maildir"
>   instead.  Subsequently setting it back to my value doesn't help;
>   it always gets changed back when I start Gnus.  Any ideas?

I think defvoo makes server parameters, like this:

(add-to-list 'gnus-secondary-select-methods
             '(nnmaildir "foo"
               (nnmaildir-directory "~/var/mail")))

Does that do it?

kai
-- 
I like BOTH kinds of music.

Re: (provide 'nnmaildir)

[Paul Jarc]

Kai Großjohann writes:
> I think defvoo makes server parameters, like this:
> 
> (add-to-list 'gnus-secondary-select-methods
>              '(nnmaildir "foo"
>                (nnmaildir-directory "~/var/mail")))
> 
> Does that do it?

Oops.  Right.  I knew that.


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
feature: -wipe-on-delete
bugfix: -request-accept-article correctly finds end of a header
bugfix: Xrefs track changes in group names
bugfix: gnus-declare-backend is called

If you want messages and .nnmaildir files to be wiped on disk before being
deleted (this applies only to files with a single hard link), use a
select method like:
'(nnmaildir "whatever" (nnmaildir-wipe-on-delete t) ...)

There is a change in the .nnmaildir file format - Xrefs are generated
on the fly, and not stored in .nnmaildir.  But it seems Gnus tolerates
the extra garbage at the end of a NOV line, so you can just wait for
all your old .nnmaildir NOV lines to disappear with expiration.  You
can also fix your old .nnmaildir files with:
M-x replace-regexp RET C-q TAB Xref:[^ C-q TAB C-q C-j ]* RET RET
Or, if you don't care about losing marks, the usual: kill your
nnmaildir groups, exit Gnus, delete your .nnmaildir files, start Gnus
again and resubscribe.

Does anyone have a better way to wipe a file than this?
    (insert (format (concat "%" (number-to-string nchars) "s") ""))
    (write-region (point-min) (point-max) file)
After implementing this, I realized it might not do any good,
depending on how write-region works.  If it opens the file with
O_TRUNC, then I might be writing to different disk blocks than were
being used for this file's data before write-region opened it.  If
instead the file is opened (without O_TRUNC), written, and then
ftruncate'd, all is well.  Does anyone know which it is?

I still haven't figured out why -request-accept-article isn't seeing
that nnmaildir-filename is bound.  I've tried making it buffer-local
instead of bound by let; no dice.  It seems that -request-move-article
isn't even being called: I put in some message calls for debugging,
and nothing showed up in *Messages*.

What's the difference between gnus-summary-copy-article and
-crosspost-article?  Is the backend required to update a message's
Xref in either case?


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

I wrote:
> <URL:http://multivac.cwru.edu/prj/nnmaildir.el>

Whoops, that was still pointing to the last version.  Fixed now.


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/prj/nnmaildir.el>
feature: -request-scan
feature: -retrieve-groups
bugfix: -request-list lists groups as writable
bugfix: no need for 'utc in (format-time-string "%s")
bugfix: -request-list was setting msgid globally, not in (let)
bugfix: tabs are replaced with spaces in NOV lines

Just restart Emacs when upgrading.  New NOV lines will be correct, but
if you've got any old corrupt lines, you'll have to fix them by hand:
edit .nnmaildir and replace the offending tabs with spaces.


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

I wrote:
> <URL:http://multivac.cwru.edu/prj/nnmaildir.el>

Oh, one more thing: unless and until the backend interface changes so
that backends can store marks, future releases will be bugfixes only -
I think I've implemented all the features that make sense for
nnmaildir.  (I could do -request-create-group, maybe, but that would
require some major architectural changes - not to say that would be a
bad idea, but I'm not planning to do it anytime soon.)  Also, the
on-disk storage format should be stable, so upgrades should require no
more than restarting Emacs.


paul

Re: (provide 'nnmaildir)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Oh, one more thing: unless and until the backend interface changes so
> that backends can store marks

Wasn't `nnfoo-request-set-mark' enough?  nnimap store marks on the
imap server.

Re: (provide 'nnmaildir)

[Paul Jarc]

Simon Josefsson <simon@josefsson.org> writes:
> Wasn't `nnfoo-request-set-mark' enough?  nnimap store marks on the
> imap server.

I never got an answer to my question:
-request-set-mark isn't documented, AFAICT, at least not in the 5.7
info.  How does it work?


paul

Re: (provide 'nnmaildir)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> > Wasn't `nnfoo-request-set-mark' enough?  nnimap store marks on the
> > imap server.
> 
> I never got an answer to my question:
> -request-set-mark isn't documented, AFAICT, at least not in the 5.7
> info.  How does it work?

>From current documentation:

`(nnchoke-request-set-mark GROUP ACTION &optional SERVER)'
     Set/remove/add marks on articles.  Normally Gnus handles the
     article marks (such as read, ticked, expired etc) internally, and
     store them in `~/.newsrc.eld'.  Some backends (such as IMAP)
     however carry all information about the articles on the server, so
     Gnus need to propagate the mark information to the server.

     ACTION is a list of mark setting requests, having this format:

          (RANGE ACTION MARK)

     Range is a range of articles you wish to update marks on.  Action
     is `set', `add' or `del', respectively used for removing all
     existing marks and setting them as specified, adding (preserving
     the marks not mentioned) mark and removing (preserving the marks
     not mentioned) marks.  Mark is a list of marks; where each mark is
     a symbol.  Currently used marks are `read', `tick', `reply',
     `expire', `killed', `dormant', `save', `download' and `unsend',
     but your backend should, if possible, not limit itself to these.

     Given contradictory actions, the last action in the list should be
     the effective one.  That is, if your action contains a request to
     add the `tick' mark on article 1 and, later in the list, a request
     to remove the mark on the same article, the mark should in fact be
     removed.

     An example action list:

          (((5 12 30) 'del '(tick))
           ((10 . 90) 'add '(read expire))
           ((92 94) 'del '(read)))

     The function should return a range of articles it wasn't able to
     set the mark on (currently not used for anything).

     There should be no result data from this function.

Re: (provide 'nnmaildir)

[Paul Jarc]

Simon Josefsson <simon@josefsson.org> writes:
> `(nnchoke-request-set-mark GROUP ACTION &optional SERVER)'

Ok, this should work.

>      Currently used marks are `read', `tick', `reply',
>      `expire', `killed', `dormant', `save', `download' and `unsend',

According to <URL:http://cr.yp.to/proto/maildir.html>, maildir MUAs
can record some of these marks in message filenames: seen ('read),
replied ('reply), trash ('expire), and a user-defined flag, which I
suppose I'll map to 'tick.  Is there a better choice?  What do
'killed, 'dormant, 'save, 'download, and 'unsend mean?  Do they
persist between Gnus sessions?


paul

Re: (provide 'nnmaildir)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> What do 'killed, 'dormant, 'save, 'download, and 'unsend mean?  Do
> they persist between Gnus sessions?

Unsend is for nndraft:queue only (I think), killed isn't persistant (I
think), the others are persistant but if the backend can't implement
them (some IMAP can't store flags) Gnus keep them in .newsrc.eld.  So
you shouldn't need to worry.

Re: (provide 'nnmaildir)

[Paul Jarc]

Woohoo!  The new version is here!
<URL:http://multivac.cwru.edu/nnmaildir/>

Instructions for upgraing are on the web.  More documentation is on
the way.

Qualitative observations: startup and scanning for new mail are a bit
slower; *Summary* buffer generation is faster.  It uses more memory,
but also uses it better.  After starting Gnus on a fresh Emacs:
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
prj       6064 29.0  7.7 12892 9944 tty1     S    11:55   0:29 emacs
This nnmaildir server has 61 groups and 3571 articles.

CHANGES:
  bugfixes:
    - tabs are replaced with spaces in Message-IDs
    - XEmacs doesn't have (forward-point); thanks Rick Wash
    - XEmacs capitalizes error messages differently; thanks Rick Wash
    - messages are moved between groups by linking instead of copying
      (sometimes, anyway.  still working on this.)
    - others; it's hard to keep track during a rewrite
  features:
    - marks are stored in the backend; article numbers are dynamically
      reassigned to keep them small and consecutive, thanks to
      -request-update-info and -request-set-mark
    - server parameter "directory" names a directory containing maildirs; new
      maildirs are scanned for by -request-scan
    - server parameter "directory-files" specifies the directory-files function
      to use to scan "directory"
    - server parameter "-wipe-on-delete" is gone; better to decrypt encrypted
      articles on the fly in -request-article (but we don't, yet)
    - all group parameters can be set to forms which will be eval'ed
    - group parameter "directory-files" specifies the directory-files function
      to use to scan maildir directories
    - group parameter "read-only" allows reading unwritable maildirs
    - group parameter "distrust-Lines:" forces counting of lines for NOV data
    - group parameter "always-marks" gives a list of marks that all articles
      should appear to have set
    - group parameter "never-marks" gives a list of marks that no articles
      should appear to have set; overrides always-marks
    - group parameter "expire-age" gives the minimum age of an article before
      expiry, in seconds
    - group parameter "expire-group" gives the name of a group to move articles
      into when expiring
    - variable "nnmaildir-article-file-name" is set by -request-article
    - -request-post uses message-send-mail function instead of
      mail-send-and-exit
  internals:
    - major rewrite for flexibility, reliability, and performance
    - .nnmaildir replaced with .nnmaildir/*
    - no more nnoo, no more nnmail
    - no need for gnus-atomic-progn; memory data structures are designed for
      safe updates


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/nnmaildir/>

This version keeps some more directory modtimes in memory, so it
doesn't need to rescan directories that haven't changed; scanning for
new mail is consequently much faster.

CHANGES:
  bugfixes:
    - -server-opened checks whether the server was opened *successfully*
    - -open-server sets the error message for the correct server
  internals:
    - symlink names are the only way to name groups
    - -request-scan doesn't need groups' devices and inodes anymore
    - -request-scan scans the server directory only after it changes
    - -update-info scans mark directories only after they change
    - modtimes are now stored as (high low), not (high . low)
    - use delete-file instead of --unlink for files found in directory-files
    - don't store maildir paths in groups; calculate them from the server
      directory and group name
    - insert numbers with princ


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/nnmaildir/>

A fix for the last known current bug.

CHANGES:
  bugfixes:
    - new mail is found with only one "g" instead of two
  internals:
    - code cleanup for -request-group


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

<URL:http://multivac.cwru.edu/nnmaildir/>
CHANGES:
  bugfixes:
    - gnus-extra-headers is handled correctly; thanks Jost Krieger


paul

Re: (provide 'nnmaildir)

[Paul Jarc]

I wrote:
> <URL:http://multivac.cwru.edu/nnmaildir/>
> CHANGES:
>   bugfixes:
>     - gnus-extra-headers is handled correctly; thanks Jost Krieger

Oops.  The new version was there, but the HTML link wasn't pointing to
it.  Fixed now.


paul