nnmaildir oddities!

nnmaildir oddities!

[Josh Huber]

I'm setting up a new Gnus running on a new server, and have decided to
use the nnmaildir backend (at least for now).

The behavior I'm seeing is this:
1. B q shows that a message will be stored in a particular group, but
when respooling said message, it goes to the "sent-mail" folder!  This
seems very odd to me.

2. Messages saved to the sent-mail folder (which are actually sent
mail) are not marked as read, even though I have gnus-gcc-mark-as-read
set.

Any ideas?

Here is the (I think) relevant parts from my .gnus.el:

(setq gnus-select-method
      '(nnmaildir ""
		  (directory "~/mail-groups")
		  (create-directory "../maildirs")
		  (get-new-mail t)))

(setq mail-sources
      '((maildir :path "~/Maildir")))

(setq gnus-message-archive-group
      '((if (message-news-p)
	        "sent-news"
	    "sent-mail")))

(setq gnus-gcc-mark-as-read t)

;; use the cache for saving messages...
(setq gnus-use-cache t)

(add-hook 'gnus-group-mode-hook 'gnus-topic-mode)
(setq gnus-subscribe-options-newsgroup-method 'gnus-subscribe-topics)
(setq gnus-subscribe-newsgroup-method 'gnus-subscribe-topics)

(setq gnus-goto-next-group-when-activating nil)

(setq gnus-posting-styles
      '((".*"
	 ;; default
	 (address "huber@alum.wpi.edu")
	 (name "Josh Huber")
	 (signature-file "~/.signature"))))

;;*======================
;;* split methods.

;; first, turn on message id caching
(setq nnmail-treat-duplicates 'delete)

;; next, cache messages that are moved from folder to folder
;(setq nnmail-cache-accepted-message-ids t)

;; ignore messages ids saved to my outgoing folders
;(setq nnmail-cache-ignore-groups "\\`\\(sent-mail\\|sent-news\\|drafts\\)$")

;; increase the message id cache size
;(setq nnmail-message-id-cache-length 2500)

;; don't even fancy-split-with-parent into these groups
(setq nnmail-split-fancy-with-parent-ignore-groups
      '("\\`sent-mail$"
	"\\`sent-news$"
	"\\`drafts$"))

(setq nnmail-split-methods 'nnmail-split-fancy)
(setq nnmail-split-fancy
      '(|
	;; debian lists
	("Resent-Sender"
	  "debian-\\(\\w+\\)\\(-request\\)?@lists\\.debian\\.org"
	   "mail.lists.debian.\\1")
	(from
	  "debian-security-announce@lists\\.debian\\.org"
	   "mail.lists.debian.security")
	(to
	  "debian-devel-announce@lists\\.debian\\.org"
	   "mail.inbox.debian")

	;; nihongo
	("Delivered-To" "mailing list nihongo" "mail.lists.nihongo")

	;; vr6 mailing list
	("Delivered-To" "mailing list vr6@yahoogroups.com"
	  "mail.lists.vr6")

	;; TMDA users list
	("List-Id" "tmda-users"
	  "mail.lists.tmda")

	;; mclx alums
	("Sender" "mclx_alums-admin@spineless\\.org"
	  "mail.lists.mclx_alums")

	;; mitoc
	(to "\\(mitoc\\(-winter\\)?\\|climbers\\)@MIT.EDU"
	        "mail.lists.mitoc")

	;; personal debian mail
	(any "huber@debian\\.org" "mail.inbox.debian")

	"mail.inbox.main"))


Thanks for taking a look...

-- 
Josh Huber

Re: nnmaildir oddities!

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> 1. B q shows that a message will be stored in a particular group, but
> when respooling said message, it goes to the "sent-mail" folder!  This
> seems very odd to me.

Does (setq nnmaildir-get-new-mail t) help?

> 2. Messages saved to the sent-mail folder (which are actually sent
> mail) are not marked as read, even though I have gnus-gcc-mark-as-read
> set.

gnus-inews-do-gcc is supposed to take care of this:
	      (when (and group-art
			 (or gnus-gcc-mark-as-read
			     gnus-inews-mark-gcc-as-read))
		(gnus-group-mark-article-read group (cdr group-art)))

If you debug-on-entry gnus-group-mark-article-read, is it called?  If
not, you could debug nnmaildir-request-accept-article and make sure it
returns a non-nil value (which is stored in group-art above, I think).

> ;; use the cache for saving messages...
> (setq gnus-use-cache t)

You should probably set gnus-uncacheable-groups to include
"\\`nnmaildir[:+]" for now; nnmaildir article numbers are not quite
persistent yet.


paul

Re: nnmaildir oddities!

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Does (setq nnmaildir-get-new-mail t) help?

I think it gets a little better...

But, it still gets spooled to the sent-mail folder.  I see this in the
message buffer:

Saving to: nnmaildir: (31)... 

>> 2. Messages saved to the sent-mail folder (which are actually sent
>> mail) are not marked as read, even though I have gnus-gcc-mark-as-read
>> set.
>
> gnus-inews-do-gcc is supposed to take care of this:
> 	      (when (and group-art
> 			 (or gnus-gcc-mark-as-read
> 			     gnus-inews-mark-gcc-as-read))
> 		(gnus-group-mark-article-read group (cdr group-art)))
>
> If you debug-on-entry gnus-group-mark-article-read, is it called?  If
> not, you could debug nnmaildir-request-accept-article and make sure it
> returns a non-nil value (which is stored in group-art above, I think).

Ah -- I think I know the cause of my problems, perhaps.  I'm using
nnmaildir as my primary select method, is this not supported?

gnus-group-mark-article-read is trying to mark an article in
nnmaildir:sent-news, and there is no such group.  It's just
"sent-news"

> You should probably set gnus-uncacheable-groups to include
> "\\`nnmaildir[:+]" for now; nnmaildir article numbers are not quite
> persistent yet.

Okay, turning of the cache got rid of another problem I just noticed
-- the summary buffer lines not matching up with what was displayed in
the article buffer.

I'd really like to use this as my primary select method, since 99% of
my groups are mail.

Any thoughts?

-- 
Josh Huber

Re: nnmaildir oddities!

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> I'm using nnmaildir as my primary select method, is this not
> supported?

It should work AFAIK, but I haven't used it that way for a while.  I
also don't use splitting.

> gnus-group-mark-article-read is trying to mark an article in
> nnmaildir:sent-news, and there is no such group.  It's just
> "sent-news"

I would have expected the backend+address:group.name format to work
regardless of primary/secondary/foreign status.  If it doesn't, I
think we should fix that.

>> You should probably set gnus-uncacheable-groups to include
>> "\\`nnmaildir[:+]" for now; nnmaildir article numbers are not quite
>> persistent yet.

I guess this won't do anything, since even if "nnmaildir:foo" could be
used as *a* name for the group, it would be the group's canonical
name.

> I'd really like to use this as my primary select method, since 99% of
> my groups are mail.

What benefit do you get from making nnmaildir primary?  Maybe there's
another way to get that.


paul

Re: nnmaildir oddities!

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> It should work AFAIK, but I haven't used it that way for a while.  I
> also don't use splitting.

I was thinking about using an external program (TMDA) actually to do
the splitting before the messages hit Gnus.  If a new group is created
by the outside filtering program, does nnmaildir automatically notice
this?  (or, can it be made to?)

> I would have expected the backend+address:group.name format to work
> regardless of primary/secondary/foreign status.  If it doesn't, I
> think we should fix that.

I agree.  Personally the difference between the primary method and the
secondary methods is something we should get rid of in general.
Everything should be required to have the backend type, but we should
make it easier to hide the backend type from the Group list.

I think some logic which looked at what groups you had subscribed and
only showed the backend/servername if there would be a conflict (same
group name subscribed via two different servers, etc)

> I guess this won't do anything, since even if "nnmaildir:foo" could
> be used as *a* name for the group, it would be the group's canonical
> name.

> What benefit do you get from making nnmaildir primary?  Maybe there's
> another way to get that.

Well, none really.  I just thought it made sense since it was my
"primary" backend, meaning I use it all the time.

I've changed the primary backend to nnnil and moved nnmaildir to the
secondary select methods list and things are working much better now.

Messages get marked as read in the sent-(mail|news) groups, but
splitting is still messed up.  Actually, the initial splitting is
working fine, it's the respooling which is busted.  I think I'll break
out edebug and find out what's going on.

If you have any other hints/tips I'd be grateful :)

Have a nice day,

-- 
Josh Huber

Re: nnmaildir oddities!

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> If a new group is created by the outside filtering program, does
> nnmaildir automatically notice this?  (or, can it be made to?)

Yes.  When you type g, nnmaildir scans the "directory" server
parameter for new groups.

> Messages get marked as read in the sent-(mail|news) groups, but
> splitting is still messed up.  Actually, the initial splitting is
> working fine, it's the respooling which is busted.  I think I'll break
> out edebug and find out what's going on.

Sounds like a plan.  Note that nnmaildir-get-new-mail is never
defvar'ed (I don't feel safe setting it to t globally); it's only
bound to t in a let form when nnmaildir calls nnmail-get-new-mail.  I
thought that was the only place where nnmaildir-get-new-mail would
matter, but I could be wrong about that.


paul

Re: nnmaildir oddities!

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Sounds like a plan.  Note that nnmaildir-get-new-mail is never
> defvar'ed (I don't feel safe setting it to t globally); it's only
> bound to t in a let form when nnmaildir calls nnmail-get-new-mail.
> I thought that was the only place where nnmaildir-get-new-mail would
> matter, but I could be wrong about that.

I think my head is about to explode trying to trace my way through the
nnmaildir code :) (no offense, I'm just not used to reading it...)

I *think* I may have found the problem, but I might be completely
missing it.

nnml-request-accept article uses nnmail-article-group when the group
is not passed in:

(if (stringp group)
    (...)

   (and
  (nnmail-activate 'nnml)
  (if (and (not (setq result (nnmail-article-group 'nnml-active-number)))
	   (yes-or-no-p "Moved to `junk' group; delete article? "))
      (setq result 'junk)
    (setq result (car (nnml-save-mail result))))
...

nnmail-article-group goes through the split methods to see what group
to save the article in.

In contrast, nnmaildir uses nnmaildir--prepare to generate a group if
one is not passed in, which just returns the current group:

(defun nnmaildir-request-accept-article (gname &optional server last)
  (let ((group (nnmaildir--prepare server gname))
	(coding-system-for-write nnheader-file-coding-system)
	(buffer-file-coding-system nil)
	(file-coding-system-alist nil)
	srv-dir dir file tmpfile curfile 24h num article)
    (catch 'return
      (if group nil
	(setf (nnmaildir--srv-error nnmaildir--cur-server)
	      (concat "No such group: " gname))
	(throw 'return nil))
      (setq gname (nnmaildir--grp-name group))
 [...]


Am I off base here?  If I'm reading this right (a big *if*), then
respooling with split rules doesn't work with nnmaildir -- it seems to
just re-save the article to the curent group!

This probably isn't the desired behavior...

Hopefully someone can explain this?

-- 
Josh Huber

Re: nnmaildir oddities!

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> In contrast, nnmaildir uses nnmaildir--prepare to generate a group if
> one is not passed in, which just returns the current group:

Right.  But the group argument is not &optional, and the backend
interface does not define how nil should be treated, so I'd say that
any caller that passes a nil value there is buggy.  Or else the manual
is.


paul

Re: nnmaildir oddities!

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Right.  But the group argument is not &optional, and the backend
> interface does not define how nil should be treated, so I'd say that
> any caller that passes a nil value there is buggy.  Or else the
> manual is.

Well, okay. :)

What would you suggest?  What do you use for mail filtering?  I've
used procmail before, but I'd like to try something new...perhaps
maildrop?

Thanks for the help,

-- 
Josh Huber

Re: nnmaildir oddities!

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> What would you suggest?  What do you use for mail filtering?

Most of my filtering is separating list subscriptions.  I do that by
using a unique address for each subscription; each of those addresses
has its own .qmail file to deliver messages to the corresponding
maildir (although a .qmail-default would also be doable).  I have a
Python script that sets up a new maildir and .qmail file and sends a
subscription request for when I want to subscribe to a new list.

The rest of my filtering consists of mostly simple .qmail lines.  For
more complicated rules, this patch is useful:
<URL:http://multivac.cwru.edu./qmail/>

I like delivery-time filtering because it's asynchronous with the UI;
no matter how slow it is, it feels faster.


paul

Re: nnmaildir oddities!

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Most of my filtering is separating list subscriptions.  I do that by
> using a unique address for each subscription; each of those
> addresses has its own .qmail file to deliver messages to the
> corresponding maildir (although a .qmail-default would also be
> doable).  I have a Python script that sets up a new maildir and
> .qmail file and sends a subscription request for when I want to
> subscribe to a new list.

I see.  I might start doing this, since I'm trying out TMDA.

> The rest of my filtering consists of mostly simple .qmail lines.
> For more complicated rules, this patch is useful:
> <URL:http://multivac.cwru.edu./qmail/>

Well, I'm not using qmail, but this is still good to know.

> I like delivery-time filtering because it's asynchronous with the
> UI; no matter how slow it is, it feels faster.

Agreed, but I'm not sure that I want to give up
fancy-split-with-parent and splitting based on bbdb fields (this I
think I'll really miss).

Perhaps I'll just ignore the respool issue, since everything else
seems to be working :)

Thanks again,

-- 
Josh Huber

Re: nnmaildir oddities!

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I like delivery-time filtering because it's asynchronous with the
>> UI; no matter how slow it is, it feels faster.
>
> Agreed, but I'm not sure that I want to give up
> fancy-split-with-parent and splitting based on bbdb fields (this I
> think I'll really miss).

Well, that could be done at delivery time too.  But maybe Emacs
wouldn't be the best way to do that.


paul

Re: nnmaildir oddities!

[Kai Großjohann]

Josh Huber <huber@alum.wpi.edu> writes:

> Agreed, but I'm not sure that I want to give up
> fancy-split-with-parent and splitting based on bbdb fields (this I
> think I'll really miss).

I used to use the with-parent splitting, too, and now I use Sieve for
splitting.  I emulate the with-parent splitting by putting a cookie
in the message id which Sieve can look at.  Maybe that's workable for
you, too.

Of course, this requires the other end to set up a proper References
header.  But this is the same for with-parent splitting.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: nnmaildir oddities!

[Josh Huber]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> I used to use the with-parent splitting, too, and now I use Sieve
> for splitting.  I emulate the with-parent splitting by putting a
> cookie in the message id which Sieve can look at.  Maybe that's
> workable for you, too.

I see what you do.  I'll think about it, but for now I'm just ignoring
the fact that respooling doesn't do what I expect it to.

-- 
Josh Huber

Re: nnmaildir oddities!

[Simon Josefsson]

Josh Huber <huber@alum.wpi.edu> writes:

> I see what you do.  I'll think about it, but for now I'm just ignoring
> the fact that respooling doesn't do what I expect it to.

I didn't follow this thread closely, but respooling in Gnus used
either nnmail internal stuff or some undocumented backend interface,
which explains why it doesn't work for nnimap.  Maybe the same applied
to nnmaildir.  Documenting the backend interface for respooling should
be possible though, as well as fixing nnimap.el.  Anyone? :)

Re: nnmaildir oddities!

[Josh Huber]

Simon Josefsson <jas@extundo.com> writes:

> I didn't follow this thread closely, but respooling in Gnus used
> either nnmail internal stuff or some undocumented backend interface,
> which explains why it doesn't work for nnimap.  Maybe the same
> applied to nnmaildir.  Documenting the backend interface for
> respooling should be possible though, as well as fixing nnimap.el.
> Anyone? :)

Well, the problem in nnmaildir was that when you call the respool
function (and eventually call nnmaildir-request-move-article, I
think), it's called with a nil destination group value.  In nnml, this
triggers a lookup with your split rules, with nnmaildir this triggers
the backend to save in the current group. (i.e. where it came from)

IMHO, the determination of the destination group should happen much
earlier in the game -- doing in buried in each backend is just not the
right thing to do.

Now, the code was a little hard to follow, so I could be completely
off base here. (just to cover my ass! :)

later,

-- 
Josh Huber

Re: nnmaildir oddities!

[Simon Josefsson]

Josh Huber <huber+dated+1028145806.d34009@alum.wpi.edu> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> I didn't follow this thread closely, but respooling in Gnus used
>> either nnmail internal stuff or some undocumented backend interface,
>> which explains why it doesn't work for nnimap.  Maybe the same
>> applied to nnmaildir.  Documenting the backend interface for
>> respooling should be possible though, as well as fixing nnimap.el.
>> Anyone? :)
>
> Well, the problem in nnmaildir was that when you call the respool
> function (and eventually call nnmaildir-request-move-article, I
> think), it's called with a nil destination group value.  In nnml, this
> triggers a lookup with your split rules, with nnmaildir this triggers
> the backend to save in the current group. (i.e. where it came from)

Ah, yes, this was the undocumented backend interface I was talking
about.  Perhaps it should be mentioned in the manual that a nil group
means the backends' split stuff should place the article.  What about
the following?  Maybe nnimap will support it now that it is documented. :-)

--- gnus.texi.~6.294.~	2002-07-26 20:04:47.000000000 +0200
+++ gnus.texi	2002-07-26 22:31:48.000000000 +0200
@@ -23632,6 +23632,9 @@
 The function should return a cons where the @code{car} is the group name and
 the @code{cdr} is the article number that the article was entered as.
 
+If @var{group} is nil, it means the article is respooled and it should
+be placed in a group determined by the backends' split rules (if any).
+
 There should be no data returned.
 
> IMHO, the determination of the destination group should happen much
> earlier in the game -- doing in buried in each backend is just not the
> right thing to do.

Split rules are backend specific in Gnus.  But I agree, it would be
nicer if there was a `gnus-split-rules' that could crosspost across
backends too.

Re: nnmaildir oddities!

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> Josh Huber <huber+dated+1028145806.d34009@alum.wpi.edu> writes:
>> IMHO, the determination of the destination group should happen much
>> earlier in the game -- doing in buried in each backend is just not the
>> right thing to do.
>
> Split rules are backend specific in Gnus.  But I agree, it would be
> nicer if there was a `gnus-split-rules' that could crosspost across
> backends too.

Well, if [B t] can figure out where the article ought to go, why not
simply pass that group instead of nil?


paul

Re: nnmaildir oddities!

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> Well, if [B t] can figure out where the article ought to go, why not
> simply pass that group instead of nil?

Exactly, this just seems to much cleaner to me...

-- 
Josh Huber

Re: nnmaildir oddities!

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> wrote:
>> Josh Huber <huber+dated+1028145806.d34009@alum.wpi.edu> writes:
>>> IMHO, the determination of the destination group should happen much
>>> earlier in the game -- doing in buried in each backend is just not the
>>> right thing to do.
>>
>> Split rules are backend specific in Gnus.  But I agree, it would be
>> nicer if there was a `gnus-split-rules' that could crosspost across
>> backends too.
>
> Well, if [B t] can figure out where the article ought to go, why not
> simply pass that group instead of nil?

B t is a kludge, it accesses the backend specific variables to do its
work.  B r uses undocumented backend features right now.  Since Gnus
is designed around having splitting in the backends, I think it is
better to try to make that more common.

Re: nnmaildir oddities!

[Simon Josefsson]

Josh Huber <huber+dated+1028149268.fd896c@alum.wpi.edu> writes:

> prj@po.cwru.edu (Paul Jarc) writes:
>
>> Well, if [B t] can figure out where the article ought to go, why not
>> simply pass that group instead of nil?
>
> Exactly, this just seems to much cleaner to me...

But then respooling would be a completely nnmail only thingie, whereas
now the backend is able to implement B r properly and B t is nnmail
only.  If the backend interface B r uses is documented, that is.