More on the new nnimap

More on the new nnimap

[James Cloos]

This is as of commit 43de9c6fe344a3075129ef8fee230f67a4cd17a1.

First the good:

In addition to the general speed improvement, it is wonderful that the
bug which occasionally caused an article to appear as a copy of another
recently viewed article has been quashed.  This is a *significant*
progression.  Thanks!

Then the issues:

NNimap does an lsub at startup and still fails to subscribe new groups
even though I set (gnus-subscribe-newsgroup-method) for all "^nn.+:"
groups.  If it is going to list the groups, then it knows about all of
them and should honour gnus-subscribe-newsgroup-method and subscribe
all new groups via it.

C in the server buffer fails to close an nnimap server.  That needs to
work.  There are valid reasons to be able to close any server on demand.

New since my last upgrade, I now get two imapd's.  And they use a pipe
rather than a pty as they used to.  Not using a pts is a bug; separating
stdout from stderr is imperative.  (It is possible the last is an Emacs
bug rather than a gnus bug; my emacs is bzr trunk rev 101727.)

I already wrote about the new h bug.  (Hmm, h as in hellish. :)

gnus-group-make-group now tries to make the group on the imap server
rather than just make a group to access the existing group on the imap
server.  If the former is useful, then it should not be an error if the
group already exists on the server.

Rather than use F, which is *slow*, I have a script which accesses the
backend directly and writes out an .el file to add the new groups.  (The
backend knows which are new; IMAP not so much.)  That script writes
a series of sexps running (gnus-group-make-group) with suitable args.

It works very well.

Until today.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More on the new nnimap

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> NNimap does an lsub

Well, LIST, but it's kinda the same thing...

> at startup and still fails to subscribe new groups even though I set
> (gnus-subscribe-newsgroup-method) for all "^nn.+:" groups.  If it is
> going to list the groups, then it knows about all of them and should
> honour gnus-subscribe-newsgroup-method and subscribe all new groups
> via it.

What is your gnus-subscribe-newsgroup-method exactly? 

> C in the server buffer fails to close an nnimap server.  That needs to
> work.  There are valid reasons to be able to close any server on demand.

Ok; implemented.

> New since my last upgrade, I now get two imapd's.

If you get two connections, then Gnus thinks that there are two
different servers.  This usually means that you have different
variations in the method definitions -- one has one more slot set or
something. 

> And they use a pipe rather than a pty as they used to.  Not using a
> pts is a bug; separating stdout from stderr is imperative.  (It is
> possible the last is an Emacs bug rather than a gnus bug; my emacs is
> bzr trunk rev 101727.)

This is with the 'shell connection?  It just does a process-start...

> gnus-group-make-group now tries to make the group on the imap server
> rather than just make a group to access the existing group on the imap
> server.

That's what it's always done, I think?

> If the former is useful, then it should not be an error if the
> group already exists on the server.

If you ask it to create a group, and it already exists on the server,
that sounds like a bug to me.

> Rather than use F, which is *slow*, I have a script which accesses the
> backend directly and writes out an .el file to add the new groups.  (The
> backend knows which are new; IMAP not so much.)  That script writes
> a series of sexps running (gnus-group-make-group) with suitable args.

Why is `F' slow?  Which backends take long to respond?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More on the new nnimap

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> James Cloos <cloos@jhcloos.com> writes:
>> NNimap does an lsub

LMI> Well, LIST, but it's kinda the same thing...

OK.  I guessed based on the reply from imapd.

>> at startup and still fails to subscribe new groups even though I set
>> (gnus-subscribe-newsgroup-method) for all "^nn.+:" groups.  If it is
>> going to list the groups, then it knows about all of them and should
>> honour gnus-subscribe-newsgroup-method and subscribe all new groups
>> via it.

LMI> What is your gnus-subscribe-newsgroup-method exactly? 

(add-hook 'gnus-startup-hook
	  '(lambda ()
	     (setq gnus-subscribe-newsgroup-method
		   '(lambda (newsgroup)
		      (if (string-match "^nn.+:" newsgroup)
			  (gnus-subscribe-alphabetically newsgroup))))))

The format must have come from a suggestion either here, in the manual
or on one of the other emacs lists.  It goes back too many years to remember.

>> C in the server buffer fails to close an nnimap server.  That needs to
>> work.  There are valid reasons to be able to close any server on demand.

LMI> Ok; implemented.

Thanks.

>> New since my last upgrade, I now get two imapd's.

LMI> If you get two connections, then Gnus thinks that there are two
LMI> different servers.  This usually means that you have different
LMI> variations in the method definitions -- one has one more slot set or
LMI> something. 

>> And they use a pipe rather than a pty as they used to.  Not using a
>> pts is a bug; separating stdout from stderr is imperative.  (It is
>> possible the last is an Emacs bug rather than a gnus bug; my emacs is
>> bzr trunk rev 101727.)

LMI> This is with the 'shell connection?  It just does a process-start...

That used to use a pty.  

Many imapds have at least the potential to write to stderr.  That screws
up the imap stream when stderr is muxed with stdout.

Oh.  Figured it out.  You have:

(defun nnimap-open-shell-stream (name buffer host port)
  (let ((process-connection-type nil))
...

process-connection-type has to be t or pty to use a pty.

>> gnus-group-make-group now tries to make the group on the imap server
>> rather than just make a group to access the existing group on the imap
>> server.

LMI> That's what it's always done, I think?

No.  That is new.

>> If the former is useful, then it should not be an error if the
>> group already exists on the server.

LMI> If you ask it to create a group, and it already exists on the server,
LMI> that sounds like a bug to me.

Creating the group, though, is to allow gnus access.

>> Rather than use F, which is *slow*, I have a script which accesses the
>> backend directly and writes out an .el file to add the new groups.  (The
>> backend knows which are new; IMAP not so much.)  That script writes
>> a series of sexps running (gnus-group-make-group) with suitable args.

LMI> Why is `F' slow?  Which backends take long to respond?

Getting the list of groups over imap takes minutes.  Even tens of
minutes.  Doing a single sql query to get the new groups taks less
than 10 ms, according to explain analyze.

(Ideally, of course, I'd skip the imap middleware and have an nn backend
which speaks sql.  But that is a /large/ project and the current setup
is only slow at startup and for things like finding new groups.

Re: More on the new nnimap

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> LMI> What is your gnus-subscribe-newsgroup-method exactly? 
>
> (add-hook 'gnus-startup-hook
> 	  '(lambda ()
> 	     (setq gnus-subscribe-newsgroup-method
> 		   '(lambda (newsgroup)
> 		      (if (string-match "^nn.+:" newsgroup)
> 			  (gnus-subscribe-alphabetically newsgroup))))))
>
> The format must have come from a suggestion either here, in the manual
> or on one of the other emacs lists.  It goes back too many years to remember.

Right.  That should work.  Does `F' subscribe you to new nnimap groups,
or do you never see new nnimap groups?

> Oh.  Figured it out.  You have:
>
> (defun nnimap-open-shell-stream (name buffer host port)
>   (let ((process-connection-type nil))
> ...
>
> process-connection-type has to be t or pty to use a pty.

The ChangeLog message for this is:

	(nnimap-open-shell-stream): Bind `process-connection-type' to nil, so
	that CRLF doesn't get translated to \n.

So binding it to nil fixes a real bug.  Is there no way to get rid of
stderr without using a pty?        

> Creating the group, though, is to allow gnus access.

The create group command is really meant to create the group on the
backend.  You can subscribe groups in Gnus, but if this has worked
before, it was probably an accident.  All the other backends create
groups in response to that command.

So you'll have to adjust your code to have Gnus subscribe the group
instead of calling out `*-request-create-group' to the backend.

> LMI> Why is `F' slow?  Which backends take long to respond?
>
> Getting the list of groups over imap takes minutes.  Even tens of
> minutes.  Doing a single sql query to get the new groups taks less
> than 10 ms, according to explain analyze.

You use IMAP as the interface to a SQL-based mail storage?

Anyway, for nnimap `F' just issues a LIST command.  Does a LIST in your
setup take minutes?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More on the new nnimap

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> James Cloos <cloos@jhcloos.com> writes:
LMI> What is your gnus-subscribe-newsgroup-method exactly? 

LMI> Right.  That should work.  Does `F' subscribe you to new nnimap groups,
LMI> or do you never see new nnimap groups?

Not now (still on commit 43de9c6fe344).  It only (claimed to) subscribe(d)
one group which wasn't actually new.

The 29 new groups were ignored.

LMI> The ChangeLog message for this is:

LMI> 	(nnimap-open-shell-stream): Bind `process-connection-type' to nil, so
LMI> 	that CRLF doesn't get translated to \n.

LMI> So binding it to nil fixes a real bug.  Is there no way to get rid of
LMI> stderr without using a pty?        

The equic of stty raw on the pty also will fix that.

>> Creating the group, though, is to allow gnus access.

LMI> The create group command is really meant to create the group on the
LMI> backend.

Seems odd.

It never did that when I used nnmh, nor have I seen it with nnmaildir.

I obviously cannot do that for nntp.

And there is nothing in the doc string to suggest that Gm would make any
changes on any server, just that it tells gnus to create a group to read 
ADDRESS.

LMI> You can subscribe groups in Gnus, but if this has worked
LMI> before, it was probably an accident.  All the other backends create
LMI> groups in response to that command.

NIME.

LMI> So you'll have to adjust your code to have Gnus subscribe the group
LMI> instead of calling out `*-request-create-group' to the backend.



LMI> Why is `F' slow?  Which backends take long to respond?

Imap.  LIST only gets about 20 groups/second.  A direct sql query
reports the whole list of groups in 7+ ms.

Checking for \noinferiors and \hasnochildren is probably what slows it
down.

LMI> You use IMAP as the interface to a SQL-based mail storage?

When I gave up on nnmh (due to many of the bugs you recently fixed, but
also the cost of accessing so many inodes) dbmail-2.2 was the best option
available.  I had to write a patch to use dense, per-group UIDs instead
of sparse, per-server UIDs, but it was still better than the alternatives.

And it mostly still is.

LMI> Anyway, for nnimap `F' just issues a LIST command.  Does a LIST in your
LMI> setup take minutes?

About 13.  And gnus ignored the new groups.

I just tried running LIST outside of gnus; the new groups are included.
One example is:

* LIST (\noinferiors \hasnochildren) "/" "gimp-announce@lists.xcf.berkeley.edu"

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More on the new nnimap

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> Not now (still on commit 43de9c6fe344).  It only (claimed to) subscribe(d)
> one group which wasn't actually new.
>
> The 29 new groups were ignored.

nnimap-request-newgroups returned nil, so it was ignored.  This might
work now...

> The equic of stty raw on the pty also will fix that.

Do you know how to do that?

> It never did that when I used nnmh, nor have I seen it with nnmaildir.

Reading the code, it looks like it does:

(deffoo nnmh-request-create-group (group &optional server args)

[...]

      (nnmh-possibly-create-directory group)

> And there is nothing in the doc string to suggest that Gm would make any
> changes on any server, just that it tells gnus to create a group to read 
> ADDRESS.

I've now updated the doc.

> Imap.  LIST only gets about 20 groups/second.  A direct sql query
> reports the whole list of groups in 7+ ms.

[...]

> LMI> Anyway, for nnimap `F' just issues a LIST command.  Does a LIST in your
> LMI> setup take minutes?
>
> About 13.  And gnus ignored the new groups.

So you have (* 20 60 13) => 15600 IMAP groups?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More on the new nnimap

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Reading the code, it looks like it does:
>
> (deffoo nnmh-request-create-group (group &optional server args)
>
> [...]
>
>       (nnmh-possibly-create-directory group)

Could that "possibly" be the issue here?  The original post said "it
should not be an error if the group already exists on the server."  I
agree that that shouldn't be an error.

>> LMI> Anyway, for nnimap `F' just issues a LIST command.  Does a LIST in your
>> LMI> setup take minutes?
>>
>> About 13.  And gnus ignored the new groups.
>
> So you have (* 20 60 13) => 15600 IMAP groups?

Maybe he meant 13 seconds??

Dan

Re: More on the new nnimap

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Could that "possibly" be the issue here?  The original post said "it
> should not be an error if the group already exists on the server."  I
> agree that that shouldn't be an error.

I don't know...  if you ask Gnus to create a group on the server, it
should give an error if that isn't possible.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More on the new nnimap

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> Dan Christensen <jdc@uwo.ca> writes:
>> Could that "possibly" be the issue here?  The original post said "it
>> should not be an error if the group already exists on the server."  I
>> agree that that shouldn't be an error.

LMI> I don't know...  if you ask Gnus to create a group on the server, it
LMI> should give an error if that isn't possible.

The other backends seem to all just accept that it already exists and
subscribe to it.  Since the function looks like:

(deffoo nnimap-request-create-group (group &optional server args)
  (when (nnimap-possibly-change-group nil server)
    (with-current-buffer (nnimap-buffer)
      (car (nnimap-command "CREATE %S" (utf7-encode group t))))))

How about passing group rather than nil to (nnimap-possibly-change-group),
returning if it exists and only creating if it does not?

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More on the new nnimap

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> James Cloos <cloos@jhcloos.com> writes:
>> Not now (still on commit 43de9c6fe344).  It only (claimed to) subscribe(d)
>> one group which wasn't actually new.
>> 
>> The 29 new groups were ignored.

LMI> nnimap-request-newgroups returned nil, so it was ignored.  This might
LMI> work now...

Yes, as of ec5f4e48dc25 it works (at least at startup).

>> The equic of stty raw on the pty also will fix that.

LMI> Do you know how to do that?

The src tells me that emacs almost does it already.

That ptys work fine with imap.el suggests that not adding CRs when doing
shell would also work.

How do you feel about making LF vs CRLF configurable or dependant on the
type of stream?

>> It never did that when I used nnmh, nor have I seen it with nnmaildir.

LMI> Reading the code, it looks like it does:

Yeah.  I commented based on experience rather than on the code....

The difference is that it is not an error if the group already exists,
except only for the new nnimap.  So how about an nnimap-request-
possibly-create-group?

LMI> So you have (* 20 60 13) => 15600 IMAP groups?

I split debian-bugs, gentoo-bugs, gnome-bugs, gnome-commits into
per-package (and for bz, per-package/reason) groups.  that accounts for
a lot of groups.  (In fact, 14670 and counting, including the empty
groups -- akin to rec vs rec.humor.)  Then there are the rss2email
groups, mailing lists, per-vendor groups.

But mostly the first bunch I mentioned above.

And thanks to the pg backend, everything is super speedy, except LIST.
(And sorting the LIST results; emacs itself now takes about 2½ Tops —
most probably spent waiting for ram — to deal with the LIST results,
before it starts EXAMINing.)

(The overall startup time is now 90 min, and that is with 21 Gbps
RAM throughput.)

Oh, and the bug where an already-read article's contents show up in the
next group is still there.  I had hoped that it were fixed by the patch
you described as something like 'fixing backends which lack NOV', but
I just got it again today.

It only occurs for me the first time I open a given group, and only if
I did not close are re-start the imapd before doing so.  I had gotten
info the habit, when setting the level for new groups, of, for each new
group, closing the server, opening the group, and reading at least one
article.  Just to avoid mismatch.

When the mismatch occurs, one must read a number of other articles and
then go back to the failed one in order to see what should be there.

(I do not use agent for nnimap.)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More on the new nnimap

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> How do you feel about making LF vs CRLF configurable or dependant on the
> type of stream?

There's already code in there to do that, and the 'shell branch set the
variable to flip that before.  However, there were other problems
related to using a pty-based shell stuff.  But I can't recall why, and
the commit message only says that I switched to non-pty-based, but not
why.  *sigh*

Anybody remembers?

> The difference is that it is not an error if the group already exists,
> except only for the new nnimap.  So how about an nnimap-request-
> possibly-create-group?

But what you want to do is to subscribe Gnus to a group from a server,
right?  The function to do that is `gnus-subscribe-group'.

> (The overall startup time is now 90 min, and that is with 21 Gbps
> RAM throughput.)

Wow.

Anyway, there is no way in the IMAP protocol to ask "what are new
mailboxes since <DATE>", so the only way Gnus has to find new groups is
to issue a LIST, right?  (Well, or LSUB, which is kinda the same thing.)
So if you have this setup, then there's no way that Gnus can get
acceptable performance if you want auto-subscription of new IMAP groups
working.

Or am I missing something?

Can't you just alter your IMAP->pg thing to make LIST fast?  :-)  Gnus
just needs the mailbox names, and doesn't care about anything else.

> Oh, and the bug where an already-read article's contents show up in the
> next group is still there.  I had hoped that it were fixed by the patch
> you described as something like 'fixing backends which lack NOV', but
> I just got it again today.
>
> It only occurs for me the first time I open a given group, and only if
> I did not close are re-start the imapd before doing so.  I had gotten
> info the habit, when setting the level for new groups, of, for each new
> group, closing the server, opening the group, and reading at least one
> article.  Just to avoid mismatch.
>
> When the mismatch occurs, one must read a number of other articles and
> then go back to the failed one in order to see what should be there.

This is most bizarre.  Gnus requests articles from backends by issuing
first a SELECT, and then a UID FETCH.  I can't see any way for that to
fetch an article from the wrong group, unless there's something
fundamentally wrong in the sequence Gnus does these things, and I
haven't heard anything odd in the area before.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: More on the new nnimap

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> The difference is that it is not an error if the group already exists,
>> except only for the new nnimap.  So how about an nnimap-request-
>> possibly-create-group?

LMI> But what you want to do is to subscribe Gnus to a group from a server,
LMI> right?  The function to do that is `gnus-subscribe-group'.

OK.  I re-did the view to create lisp which uses (gnus-subscribe-group).
They don't automatically show up in the *Group* buffer as before, so I
have to learn to do C-uAl after rather than before, but it does work.

LMI> Anyway, there is no way in the IMAP protocol to ask "what are new
LMI> mailboxes since <DATE>", so the only way Gnus has to find new groups is
LMI> to issue a LIST, right?  (Well, or LSUB, which is kinda the same thing.)
LMI> So if you have this setup, then there's no way that Gnus can get
LMI> acceptable performance if you want auto-subscription of new IMAP groups
LMI> working.

The problem with dbmail is that it first does a SELECT to grab the group
IDs requested by a LIST/LSUB, and then iterates through those to grab
the info needed to create the reply line.

Were it instead to do a single query for all of the data it needs it
would be *fast*.

As I previously wrote, I had to patch dbmail to use dense UIDs; I'll
have to do another patch to improve LIST, once I figure out the required
SQL.  (haschildren vs hasnochildren is the hard part.)

I suspect dbmail works the way it does because it was first written for
mysql; support for sqlite and postgresql was added later.

LMI> Can't you just alter your IMAP->pg thing to make LIST fast?  :-)  Gnus
LMI> just needs the mailbox names, and doesn't care about anything else.

See above.  If I can reduce it to a single select it should blaze.

>> When the mismatch occurs, one must read a number of other articles and
>> then go back to the failed one in order to see what should be there.

LMI> This is most bizarre.  Gnus requests articles from backends by issuing
LMI> first a SELECT, and then a UID FETCH.  I can't see any way for that to
LMI> fetch an article from the wrong group, unless there's something
LMI> fundamentally wrong in the sequence Gnus does these things, and I
LMI> haven't heard anything odd in the area before.

I cannot rule out a bug in dbmail or in my patch to dbmail.  I don't see
such a bug, but it might be there.

I did go back to gnus-single-article-buffer t; I don't know whether that
is relevant.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: More on the new nnimap

[Lars Magne Ingebrigtsen]

James Cloos <cloos@jhcloos.com> writes:

> As I previously wrote, I had to patch dbmail to use dense UIDs; I'll
> have to do another patch to improve LIST, once I figure out the required
> SQL.  (haschildren vs hasnochildren is the hard part.)

Gnus doesn't really care about haschildren...

> I did go back to gnus-single-article-buffer t; I don't know whether that
> is relevant.

No, that can't have any effect here.  But it's theoretically possible
that it's something wrong somewhere in the backlog code.  Try setting
`gnus-keep-backlog' to nil and see what happens.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen