maildir - whats the skinny?

maildir - whats the skinny?

[Harry Putnam]

There is precious little in gnus info about using maildir.  Under
mail-sources section it is described a bit how to collect from one.

Looks like one needs to name the subdirectories under maildir.  Is
this necessary?

If I have procmail writing to maildir and creating directories as
needed, then writing the numbered messages under that directory.  Do I
need to tell gnus about teh subdirectory?

We might see ~/maildir/group1

Then procmail might create maildir/group4.  Later maildir/group6 etc
Do I need to tell gnus in advance about these names? Or does gnus know
to look in maildir and recurse, the subdirectories.  Naming nnml
groups after them?

The examples in gnus info:
  Two example maildir mail sources:

          (maildir :path "/home/user-name/Maildir/"
                   :subdirs ("cur" "new"))

          (maildir :path "/user@remotehost.org:~/Maildir/"
                   :subdirs ("new"))

I won't always know the groups being created by procmail in advance.
I have procmail  setup to name groups after information found in the
Newsgroups: header.  This is an incoming nntpfeed being written to maildir.

Re: maildir - whats the skinny?

[Matt Armstrong]

Harry Putnam <reader@newsguy.com> writes:

> If I have procmail writing to maildir and creating directories as
> needed, then writing the numbered messages under that directory.  Do I
> need to tell gnus about teh subdirectory?

I believe the answer is yes -- I think it'd be a new mail source of
type "maildir".


> We might see ~/maildir/group1
>
> Then procmail might create maildir/group4.  Later maildir/group6 etc
> Do I need to tell gnus in advance about these names? Or does gnus know
> to look in maildir and recurse, the subdirectories.  Naming nnml
> groups after them?

The only way I know of for Gnus to automatically discover new groups
based on procmail output is with the "directory" style mail-source.  I
have this in my .gnus:

(setq mail-sources '((directory :path "~/.incoming")))

But this expects unix mbox files with a .spool suffix to be in that
dir, not maildirs.

I'm not sure if Gnus supports a similar thing with maildirs.  Last
time I investigated, I concluded that it didn't.


> The examples in gnus info:
>   Two example maildir mail sources:
>
>           (maildir :path "/home/user-name/Maildir/"
>                    :subdirs ("cur" "new"))
>
>           (maildir :path "/user@remotehost.org:~/Maildir/"
>                    :subdirs ("new"))

I believe the maildir mail source names a single mail folder.

-- 
matt

Re: maildir - whats the skinny?

[Harry Putnam]

"Matt Armstrong" <matt+dated+1007770937.e3ec50@lickey.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> If I have procmail writing to maildir and creating directories as
>> needed, then writing the numbered messages under that directory.  Do I
>> need to tell gnus about teh subdirectory?
>
> I believe the answer is yes -- I think it'd be a new mail source of
> type "maildir".

Looks like it.  So this may not be a good technique for what I'm
doing.  Although it probably wouldn't be that hard to scan the
directory on the fly with a function hooked into mail collection some
how, and use that info to generate a dynamic list to pass to gnus
collection tools.

May not be worth all that.  And I don't know how to do it anyway.

[...]

> The only way I know of for Gnus to automatically discover new groups
> based on procmail output is with the "directory" style mail-source.  I
> have this in my .gnus:

[...]

Yeah, that is the cannonical way.  And I've used it for some time so
am familiar with it.

> But this expects unix mbox files with a .spool suffix to be in that
> dir, not maildirs.

The mbox technique was my first thought.  And I have that working now.

I had sought to avoid some other pitfalls I ran into by staying clear
of mbox format.  But looks like it may not be possible.  So figuring a
way to fix the pitfalls is next.

Here is the problem briefly.   I'm funneling an nntp feed directly
into formail/procmail and writing mbox files based on the newsgroup
name plus a local prefix and suffix, for other purposes.

It allows a direct conversion of news to mail, and gnus promptly
writes them to nnml groups thus exposing news to all the functions and
cool techniques that can be applied to mail.

One pitfall I've run into is that NNTP output is not all that
clean. It is supposed to be in a certain format:

That is, messages begin with `Path: ' header and end with a dot on a
line by itself, then a blank line (I think), That is mostly what comes
down the wire, but some are different and have the Path header after
several other headers or the like.  Some have no blank line.

`formail' is able to discern the `Path: ' header but gets confused
when it isn't the lead header.  Sometimes writing several messges from
one nntp message.  A case in point is the faq posted for the newsreader
nn. It doesn't start with Path: but it does have lots of short
sections.  Each beginning with what `formail' thinks are more message
headers.

consequently it writes several dozen 5 and 6 line messages from that
one nntp message.

So far I've just coralled those misfits into there own NO_PATH group.
Probably just ignore it.  But I think there may be some other stuff that
will turn up like messages from mail to news converters where some
mailling list is converted to news and piped onto usenet.

I may have to abandon the `formail' part of my filter and write my own
awk scripting that can deal with messages regardless of how they
begin.  Maybe by dealing only with the dot on a line by itself to
invoke the action needed to write the message to my spool directory.

That isn't a very hard task I don't think, and would still allow
procmail to do the heavy work, locking and so forth. so that may allow
me to use the normal procmail proceedure with gnus.

Currently I have:
nntpserver_output =>|formail procmail => spool/mboxfiles \
=>gnus =>nnml

So I may try it like:
nntpserver_output =>|scripting|procmail => spool/mboxfiles \
=>gnus =>nnml


And then get some kind of smooth working technique for posting from
those nnml groups.  At that point I will say goodbye to gnus-agent.

Nifty as the agent is it has never been fully integrated into
gnus. And has always had some semi-major irritations.  I think I will
enjoy having full functionallity on news this way.

Re: maildir - whats the skinny?

[Matt Armstrong]

Harry Putnam <reader@newsguy.com> writes:

> I had sought to avoid some other pitfalls I ran into by staying
> clear of mbox format.  But looks like it may not be possible.  So
> figuring a way to fix the pitfalls is next.

I'd like to have a 'directory' mail source that supported Maildir
spool folders, if only to avoid the mbox escaping of "^From " in the
message body.


> Each beginning with what `formail' thinks are more message headers.

Slightly off topic, but formail has an -m argument.  By default
formail will think it sees a new message when it sees 2 headers in a
row.  Use -m 5 and it'll require 5 headers, which might fix the
problem.


> And then get some kind of smooth working technique for posting from
> those nnml groups.  At that point I will say goodbye to gnus-agent.

I run "leafnode" (http://www.leafnode.org) to have a local nntp cache
of the groups I read.  Accessing the messages is still slower than an
nntp group, but fast enough for me.  Leafnode allows you to post to
it, and then it goes and posts to the main server when you tell it to.
So even if you don't read news via leafnode, you can post to leafnode
as a kind of nntp send queue.  (there may be a better way though)


> Nifty as the agent is it has never been fully integrated into
> gnus. And has always had some semi-major irritations.  I think I
> will enjoy having full functionallity on news this way.

Prompted by some recent discussion here about Wanderlust, I checked it
out and was impressed.  It appeared generally faster than Gnus and to
have a better (or, at least, less organically grown) back end system
that supported a richer set of disconnected modes.  E.g. you can have
mail stored in a plain Maildir, and Wanderlust will cache metadata in
a separate dir -- so you end up with the functional equivalent to nnml
but with a plain Maildir folder (which procmail can deliver to
directly).  But, alas, as far as I could tell it didn't have a way to
read mail "newsreader style" where only new messages show by default,
so with Gnus I stay.  I also have considerable inertia invested in
Gnus over the years.  ;-)


-- 
matt

Re: maildir - whats the skinny?

[Paul Jarc]

"Matt Armstrong" <matt+dated+1007787514.dc3e19@lickey.com> wrote:
> I'd like to have a 'directory' mail source that supported Maildir
> spool folders,

One might consider that to be essentially equivalent to nnmaildir.


paul

Re: maildir - whats the skinny?

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> "Matt Armstrong" <matt+dated+1007787514.dc3e19@lickey.com> wrote:
>> I'd like to have a 'directory' mail source that supported Maildir
>> spool folders,
>
> One might consider that to be essentially equivalent to nnmaildir.

So I can point nnmaildir at a directory, and nnmaildir will slurp in
all the maildir sub directories matching some regexp and create the
needed Gnus groups?

-- 
matt

Re: maildir - whats the skinny?

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> If I have procmail writing to maildir and creating directories as
> needed, then writing the numbered messages under that directory.  Do I
> need to tell gnus about teh subdirectory?

You could also use nnmaildir to just read the maildirs directly.

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: maildir - whats the skinny?

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> If I have procmail writing to maildir and creating directories as
>> needed, then writing the numbered messages under that directory.  Do I
>> need to tell gnus about teh subdirectory?
>
> You could also use nnmaildir to just read the maildirs directly.

Paul pointed that out on a different forum.  And I read thru the docu
stuff on nnmaildir.  Some things sound promising, like the point you
raise but I couldn't really make heads or tails of it overall.  

It wasn't clear if gnus regular mail functions and goodies like the
complex expiry and scoring etc applies to nnmaildir or not.  The
discription including always-marks, never-marks and the unusual expire
dating things threw me for a loop.

My whole point in trying to funnel an nntp feed into procmail and
hence convert it to mail that gnus can turn into nnml groups is to
access the nice stuff gnus can do with nnml groups.  Wasn't at all
clear if any of that also applies to nnmaildir.

Re: maildir - whats the skinny?

[Harry Putnam]

"Matt Armstrong" <matt@lickey.com> writes:

[...]

>> Each beginning with what `formail' thinks are more message headers.
>
> Slightly off topic, but formail has an -m argument.  By default
> formail will think it sees a new message when it sees 2 headers in a
> row.  Use -m 5 and it'll require 5 headers, which might fix the
> problem.

Haa... yeah that is exaclty what I am seeing.  Messages created on the
strength of 2 header like lines.  Seems like formail would pay more
attention to the dot on a line by itself and only print when that has
shown up.  Probably introduces other problems or sloth.

[...]

> I run "leafnode" (http://www.leafnode.org) to have a local nntp cache
> of the groups I read.  Accessing the messages is still slower than an
> nntp group, but fast enough for me.  Leafnode allows you to post to
> it, and then it goes and posts to the main server when you tell it to.
> So even if you don't read news via leafnode, you can post to leafnode
> as a kind of nntp send queue.  (there may be a better way though)

I ran that for a little while once but it doesn't really change any
thing inside gnus.  The news is still in the restricted category of
nntp. And leafnode trys to be a little too smart for its own good with
the heuristics.  I thought that the agent categories and activate
levels etc gave more functionality.  So went back to regular nntp
server. 

I tried slrnpull for a while too.  I think it is better than leafnode
in that it doesn't outsmart the user so much.  But again there is no
gain inside gnus.  But converting news to mail on the other hand,
opens up quite a lot more fucntionality.

Re: maildir - whats the skinny?

[Paul Jarc]

"Matt Armstrong" <matt+dated+1007795501.b00efc@lickey.com> wrote:
> So I can point nnmaildir at a directory, and nnmaildir will slurp in
> all the maildir sub directories matching some regexp and create the
> needed Gnus groups?

Yes, except the regexp is not configurable.  It's always "\\`[^.]".


paul

Re: maildir - whats the skinny?

[Paul Jarc]

Harry Putnam <reader@newsguy.com> wrote:
> It wasn't clear if gnus regular mail functions and goodies like the
> complex expiry and scoring etc applies to nnmaildir or not.  The
> discription including always-marks, never-marks and the unusual expire
> dating things threw me for a loop.

Scoring is pretty backend-independent, I think - at least, no part of
the backend interface seems to have anything to do with scoring.
Expiry is indeed a bit different with nnmaildir, but no less
expressive than with other backends.  So you can do everything you're
doing now; you just have to express it differently.  You can ignore
the always-marks and never-marks stuff; they're proof-of-concept
features that ought to be added to Gnus proper or abandoned entirely.


paul

Re: maildir - whats the skinny?

[Paul Jarc]

I wrote:
> "Matt Armstrong" <matt+dated+1007795501.b00efc@lickey.com> wrote:
>> So I can point nnmaildir at a directory, and nnmaildir will slurp in
>> all the maildir sub directories matching some regexp and create the
>> needed Gnus groups?
>
> Yes, except the regexp is not configurable.  It's always "\\`[^.]".

Actually, you could make it different by supplying an appropriately
clever directory-file server parameter.  Neat.


paul

Re: maildir - whats the skinny?

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> I wrote:
>> "Matt Armstrong" <matt+dated+1007795501.b00efc@lickey.com> wrote:
>>> So I can point nnmaildir at a directory, and nnmaildir will slurp in
>>> all the maildir sub directories matching some regexp and create the
>>> needed Gnus groups?
>>
>> Yes, except the regexp is not configurable.  It's always "\\`[^.]".
>
> Actually, you could make it different by supplying an appropriately
> clever directory-file server parameter.  Neat.

Cool, so why isn't nnmaildir in CVS?  ;-)


-- 
matt

Re: maildir - whats the skinny?

[Paul Jarc]

"Matt Armstrong" <matt+dated+1007833240.e38fe2@lickey.com> wrote:
> Cool, so why isn't nnmaildir in CVS?  ;-)

I've done the FSF paperwork, but Lars hasn't yet responded to my
request for CVS write access.  No one who already has write access has
added it yet either, which may be just as well - before it's added,
I'd like to move the proof-of-concept features into Gnus or get rid of
them if they're not worthwhile.

(A note on the following syntax: for extra dynamism, nnmaildir's group
parameter values are Lisp forms which get evaluated when they are
used.  But since a group parameter value can't be a list, we wrap them
in a vector.)  nnmaildir looks at these two group parameters (taken
from my documentation on the web):

always-marks: a list of mark symbols, such as ['(read expire)].
Whenever Gnus asks nnmaildir for message marks, nnmaildir will say
that all messages have these marks, regardless of whether the marks
stored in the filesystem say so.

never-marks: a list of mark symbols, such as ['(tick expire)].
Whenever Gnus asks nnmaildir for message marks, nnmaildir will say
that no messages have these marks, regardless of whether the marks
stored in the filesystem say so.  never-marks overrides always-marks.

Are these worthwhile?  I've never needed them; I just liked the idea.
If they are desirable, they ought to be added to Gnus proper in some
way, so all backends(' users) can use them.  If not, I'll remove them
from nnmaildir.  I can imagine that always-marks as ['(read)] might be
useful for archive-only groups.


paul

Re: maildir - whats the skinny?

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> "Matt Armstrong" <matt+dated+1007833240.e38fe2@lickey.com> wrote:
>> Cool, so why isn't nnmaildir in CVS?  ;-)
>
> I've done the FSF paperwork, but Lars hasn't yet responded to my
> request for CVS write access.

Bummer.

> I can imagine that always-marks as ['(read)] might be useful for
> archive-only groups.

Yes, it annoys me that I periodically must "catchup" my sent-mail
folder (to which I Gcc: all mail I send).


-- 
matt

Re: maildir - whats the skinny?

[Henrik Enberg]

"Matt Armstrong" <matt+dated+1007836323.a3fd44@lickey.com> writes:

> prj@po.cwru.edu (Paul Jarc) writes:

[...]

>> I can imagine that always-marks as ['(read)] might be useful for
>> archive-only groups.
>
> Yes, it annoys me that I periodically must "catchup" my sent-mail
> folder (to which I Gcc: all mail I send).

>From My customize block:

 '(gnus-inews-mark-gcc-as-read t)

But I agree that it would be nice,  I expire some groups into archive
groups where they are marked as unread, so I have to do catch-up on
them.  Or is this possible change this right now?

Henrik
-- 
Well, I think if you say you're going to do something and don't do it,
that's trustworthiness.
		-- George W Bush

Re: maildir - whats the skinny?

[Kai Großjohann]

prj@po.cwru.edu (Paul Jarc) writes:

> I've done the FSF paperwork, but Lars hasn't yet responded to my
> request for CVS write access.  No one who already has write access has
> added it yet either, which may be just as well - before it's added,
> I'd like to move the proof-of-concept features into Gnus or get rid of
> them if they're not worthwhile.

Ick.  I could commit it if you like.

It would be way cool if you had CVS (write) access, though.  Hm.

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)