nnimap slowness

nnimap slowness

[David Abrahams]

I have a fast connection here (10Mb/s cable modem) and my IMAP server
seems perfectly snappy and responsive when I use other newsreaders,
but GNUs is just painfully slow, especially when downloading a message
with a large attachment.  A 1.6MB attachment can take minutes!

Is there anything magical I can do to make this go faster?  I'd even
settle for the mildly paranormal as long as it speeds things up!

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> I have a fast connection here (10Mb/s cable modem) and my IMAP server
> seems perfectly snappy and responsive when I use other newsreaders,
> but GNUs is just painfully slow, especially when downloading a message
> with a large attachment.  A 1.6MB attachment can take minutes!
>
> Is there anything magical I can do to make this go faster?  I'd even
> settle for the mildly paranormal as long as it speeds things up!

Is it downloading or article rendering that takes time?  See the
troubleshooting section in the manual for how to tell.  If downloading
is slower than in other clients, something is buggy, but viewing large
messages is known to be slow in Gnus since the article is copied
around a few times.

Hm. Zero-copy would be nice, but would probably require some new emacs
primitive.  Like a text overlay to inline one buffer into another.

Re: nnimap slowness

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> I have a fast connection here (10Mb/s cable modem) and my IMAP server
>> seems perfectly snappy and responsive when I use other newsreaders,
>> but GNUs is just painfully slow, especially when downloading a message
>> with a large attachment.  A 1.6MB attachment can take minutes!
>>
>> Is there anything magical I can do to make this go faster?  I'd even
>> settle for the mildly paranormal as long as it speeds things up!
>
> Is it downloading or article rendering that takes time?  See the
> troubleshooting section in the manual for how to tell.  

What specifically are you referring to?  I don't see the words
"downloading", "rendering", or "imap" in there.

> If downloading is slower than in other clients, something is buggy,

It sure seems like downloading is the problem; the attachment isn't
displayed except as a MIME button.

> but viewing large messages is known to be slow in Gnus since the
> article is copied around a few times.
>
> Hm. Zero-copy would be nice, but would probably require some new emacs
> primitive.  Like a text overlay to inline one buffer into another.

You're way out of my league here.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> David Abrahams <dave@boost-consulting.com> writes:
>>
>>> I have a fast connection here (10Mb/s cable modem) and my IMAP server
>>> seems perfectly snappy and responsive when I use other newsreaders,
>>> but GNUs is just painfully slow, especially when downloading a message
>>> with a large attachment.  A 1.6MB attachment can take minutes!
>>>
>>> Is there anything magical I can do to make this go faster?  I'd even
>>> settle for the mildly paranormal as long as it speeds things up!
>>
>> Is it downloading or article rendering that takes time?  See the
>> troubleshooting section in the manual for how to tell.
>
> What specifically are you referring to?  I don't see the words
> "downloading", "rendering", or "imap" in there.

I was referring to profiling Gnus to find out what part of it is
taking time (but it is most likely MIME decoding and displaying that
takes time, see below).

>> If downloading is slower than in other clients, something is buggy,
>
> It sure seems like downloading is the problem; the attachment isn't
> displayed except as a MIME button.

Ah, but the message is still copied around.  Collapsing it into a
button is only done at the final moment of display.  Yes, this is
inefficient.  Nobody has worked on fixing this yet though...

Re: nnimap slowness

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>> What specifically are you referring to?  I don't see the words
>> "downloading", "rendering", or "imap" in there.
>
> I was referring to profiling Gnus to find out what part of it is
> taking time

OK, I'm happy to try that, but see below.

> (but it is most likely MIME decoding and displaying that
> takes time, see below).
> 
>>> If downloading is slower than in other clients, something is buggy,
>>
>> It sure seems like downloading is the problem; the attachment isn't
>> displayed except as a MIME button.
>
> Ah, but the message is still copied around.  Collapsing it into a
> button is only done at the final moment of display.  Yes, this is
> inefficient.  Nobody has worked on fixing this yet though...

Seriously?  It only takes a fraction of a second to copy 1.6M on this
machine.  When I look at the status line and see

          imap read: 774K

with the numbers spinning upwards at a rate of only about 15K per
second I grow highly doubtful that it's doing anything other than
reading data from my IMAP server.

If you still want me to profile something, just say the word.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> (but it is most likely MIME decoding and displaying that
>> takes time, see below).
>> 
>>>> If downloading is slower than in other clients, something is buggy,
>>>
>>> It sure seems like downloading is the problem; the attachment isn't
>>> displayed except as a MIME button.
>>
>> Ah, but the message is still copied around.  Collapsing it into a
>> button is only done at the final moment of display.  Yes, this is
>> inefficient.  Nobody has worked on fixing this yet though...
>
> Seriously?  It only takes a fraction of a second to copy 1.6M on this
> machine.

Between buffers inside emacs?

> When I look at the status line and see
>
>           imap read: 774K
>
> with the numbers spinning upwards at a rate of only about 15K per
> second I grow highly doubtful that it's doing anything other than
> reading data from my IMAP server.

Hm, yes, this sounds as if it is the downloading that takes time.  Hm,
maybe it is due to subprocesses in emacs under Windows.  Please try
profiling the imap and nnimap packages to see where it is spending the
time.  Maybe the gnus package too, so that the total time is available
too (display probably still takes some non-negligible time).

Re: nnimap slowness

[David Abrahams]

[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> (but it is most likely MIME decoding and displaying that
>>> takes time, see below).
>>> 
>>>>> If downloading is slower than in other clients, something is buggy,
>>>>
>>>> It sure seems like downloading is the problem; the attachment isn't
>>>> displayed except as a MIME button.
>>>
>>> Ah, but the message is still copied around.  Collapsing it into a
>>> button is only done at the final moment of display.  Yes, this is
>>> inefficient.  Nobody has worked on fixing this yet though...
>>
>> Seriously?  It only takes a fraction of a second to copy 1.6M on this
>> machine.
>
> Between buffers inside emacs?

I don't know about that.

>> When I look at the status line and see
>>
>>           imap read: 774K
>>
>> with the numbers spinning upwards at a rate of only about 15K per
>> second I grow highly doubtful that it's doing anything other than
>> reading data from my IMAP server.
>
> Hm, yes, this sounds as if it is the downloading that takes time.  Hm,
> maybe it is due to subprocesses in emacs under Windows.  Please try
> profiling the imap and nnimap packages to see where it is spending the
> time.  Maybe the gnus package too, so that the total time is available
> too (display probably still takes some non-negligible time).

Here's the result for a `M-G' which is not retrieving any messages at
all, since none were sent (but still does the slow "imap read: ..."
dance up to about 24K).  If you want more data, I'll send myself
something big and profile that.


[-- Attachment #2: profile.zip --]
[-- Type: application/zip, Size: 3559 bytes --]

[-- Attachment #3: Type: text/plain, Size: 62 bytes --]



-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>>> When I look at the status line and see
>>>
>>>           imap read: 774K
>>>
>>> with the numbers spinning upwards at a rate of only about 15K per
>>> second I grow highly doubtful that it's doing anything other than
>>> reading data from my IMAP server.
>>
>> Hm, yes, this sounds as if it is the downloading that takes time.  Hm,
>> maybe it is due to subprocesses in emacs under Windows.  Please try
>> profiling the imap and nnimap packages to see where it is spending the
>> time.  Maybe the gnus package too, so that the total time is available
>> too (display probably still takes some non-negligible time).
>
> Here's the result for a `M-G' which is not retrieving any messages at
> all, since none were sent (but still does the slow "imap read: ..."
> dance up to about 24K).  If you want more data, I'll send myself
> something big and profile that.

For small data, the cause is much more likely to be server slowness or
network latency, so please send try sending a 5MB attachment or
something.

To find out what causes slowness for small data, you must tcpdump or
ethereal the stream to see which side is delaying the transfer, or if
it is the network that has high latency.  ELP data isn't that useful
in this case.

Re: nnimap slowness

[David Abrahams]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>>> When I look at the status line and see
>>>>
>>>>           imap read: 774K
>>>>
>>>> with the numbers spinning upwards at a rate of only about 15K per
>>>> second I grow highly doubtful that it's doing anything other than
>>>> reading data from my IMAP server.
>>>
>>> Hm, yes, this sounds as if it is the downloading that takes time.  Hm,
>>> maybe it is due to subprocesses in emacs under Windows.  Please try
>>> profiling the imap and nnimap packages to see where it is spending the
>>> time.  Maybe the gnus package too, so that the total time is available
>>> too (display probably still takes some non-negligible time).
>>
>> Here's the result for a `M-G' which is not retrieving any messages at
>> all, since none were sent (but still does the slow "imap read: ..."
>> dance up to about 24K).  If you want more data, I'll send myself
>> something big and profile that.
>
> For small data, the cause is much more likely to be server slowness or
> network latency, so please send try sending a 5MB attachment or
> something.

OK, here:


[-- Attachment #2: profile.zip --]
[-- Type: application/zip, Size: 4951 bytes --]

[-- Attachment #3: Type: text/plain, Size: 518 bytes --]


But I still think the fact that I get the "imap read: ..." dance
counting up to 27K *twice* when I do `M-G' and receive no messages is
highly suspicious.

> To find out what causes slowness for small data, you must tcpdump or
> ethereal the stream to see which side is delaying the transfer, or if
> it is the network that has high latency.  ELP data isn't that useful
> in this case.

Maybe I'll try that stuff later today if I can figure it out, thanks.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> For small data, the cause is much more likely to be server slowness or
>> network latency, so please send try sending a 5MB attachment or
>> something.
>
> OK, here:

Function Name                       Call Count  Elapsed Time  Average Time
==================================  ==========  ============  ============
gnus-summary-reselect-current-group      1           8.132         8.132
gnus-summary-rescan-group                1           8.132         8.132
nnimap-request-group                     2           6.43          3.215
imap-send-command-wait                  14          6.339         0.4527857142
imap-wait-for-tag                       14          6.339         0.4527857142
nnimap-request-update-info-internal      2           6.2989999999  3.1494999999
imap-search                             11          5.927         0.5388181818
gnus-group-get-new-news-this-group       1           4.055         4.055
gnus-summary-read-group                  1           3.586         3.586
gnus-group-read-group                    1           3.586         3.586
gnus-summary-read-group-1                1           3.586         3.586
gnus-select-newsgroup                    1           3.526         3.526
gnus-activate-group                      1           3.465         3.465
gnus-request-group                       1           2.965         2.965
gnus-retrieve-headers                    2           0.58          0.29
...

It seems the IMAP SEARCH command is causing the delay (most of the
nnimap related delay is in imap-search), suggesting that the server
opens large messages when the SEARCH command is invoked.  This is
unnecessary, since Gnus only searches for flags (unseen, seen, ticked,
etc).  Perhaps you could forward this observation (after verifying it
in the source) to the IMAP server administrator/developers to have
them improve searches for flags.

> But I still think the fact that I get the "imap read: ..." dance
> counting up to 27K *twice* when I do `M-G' and receive no messages is
> highly suspicious.

I don't think so -- that is displayed when imap.el waits for feedback
from the server.  M-g causes many commands to be sent to the server,
several of which are SEARCH commands (see *imap-log*), so if more than
one command takes time, that message is displayed.

>> To find out what causes slowness for small data, you must tcpdump or
>> ethereal the stream to see which side is delaying the transfer, or if
>> it is the network that has high latency.  ELP data isn't that useful
>> in this case.
>
> I got myself a copy of Ethereal and captured packets from a `M-G', but
> I have no clue yet as to how to get any useful information from what
> I'm seeing...
>
> ...I hate to keep harping on this, but it really has the look/feel of
> some inefficiency in what GNUs is doing.
>
> Looking at the packets one more time it sure looks very much as though
> GNUs is receiving the same sequence of packets (about 27K worth) twice
> from the IMAP server, though it's pretty hard for me to tell
> anything, especially peering through SSL encoding.

Right, you need to get a protocol dump with timestamps of the IMAP
connection.

Gnus do send several SEARCH commands, and this is quite inefficient,
but I believe fixing this would require changing the Gnus backend
interface, and that means work.  Several servers implement flag
searching without opening all articles though, so many people don't
have this problem.

Re: nnimap slowness

[David Abrahams]

Thanks for following up, Simon...

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> For small data, the cause is much more likely to be server slowness or
>>> network latency, so please send try sending a 5MB attachment or
>>> something.
>>
>> OK, here:
>
> Function Name                       Call Count  Elapsed Time  Average Time
> ==================================  ==========  ============  ============
> gnus-summary-reselect-current-group      1           8.132         8.132
> gnus-summary-rescan-group                1           8.132         8.132
> nnimap-request-group                     2           6.43          3.215
> imap-send-command-wait                  14          6.339         0.4527857142
> imap-wait-for-tag                       14          6.339         0.4527857142
> nnimap-request-update-info-internal      2           6.2989999999  3.1494999999
> imap-search                             11          5.927         0.5388181818
> gnus-group-get-new-news-this-group       1           4.055         4.055
> gnus-summary-read-group                  1           3.586         3.586
> gnus-group-read-group                    1           3.586         3.586
> gnus-summary-read-group-1                1           3.586         3.586
> gnus-select-newsgroup                    1           3.526         3.526
> gnus-activate-group                      1           3.465         3.465
> gnus-request-group                       1           2.965         2.965
> gnus-retrieve-headers                    2           0.58          0.29
> ...
>
> It seems the IMAP SEARCH command is causing the delay (most of the
> nnimap related delay is in imap-search), suggesting that the server
> opens large messages when the SEARCH command is invoked.  This is
> unnecessary, since Gnus only searches for flags (unseen, seen, ticked,
> etc).  Perhaps you could forward this observation (after verifying it
> in the source) 

The source of what?

> to the IMAP server administrator/developers to have them improve
> searches for flags.

We're using Communigate Pro, for what that's worth to you.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> It seems the IMAP SEARCH command is causing the delay (most of the
>> nnimap related delay is in imap-search), suggesting that the server
>> opens large messages when the SEARCH command is invoked.  This is
>> unnecessary, since Gnus only searches for flags (unseen, seen, ticked,
>> etc).  Perhaps you could forward this observation (after verifying it
>> in the source) 
>
> The source of what?

Of the server.  Assuming you have it, of course.

>> to the IMAP server administrator/developers to have them improve
>> searches for flags.
>
> We're using Communigate Pro, for what that's worth to you.

I guess it doesn't come with source.  I recall the same problem was
identified quite some time ago, but perhaps it was never reported
upstream.  Asking the vendor to optimize the SEARCH commands for flags
is probably the only practical solution then.

Re: nnimap slowness

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> It seems the IMAP SEARCH command is causing the delay (most of the
>>> nnimap related delay is in imap-search), suggesting that the server
>>> opens large messages when the SEARCH command is invoked.  This is
>>> unnecessary, since Gnus only searches for flags (unseen, seen, ticked,
>>> etc).  Perhaps you could forward this observation (after verifying it
>>> in the source) 
>>
>> The source of what?
>
> Of the server.  Assuming you have it, of course.
>
>>> to the IMAP server administrator/developers to have them improve
>>> searches for flags.
>>
>> We're using Communigate Pro, for what that's worth to you.
>
> I guess it doesn't come with source.  I recall the same problem was
> identified quite some time ago, but perhaps it was never reported
> upstream.  Asking the vendor to optimize the SEARCH commands for flags
> is probably the only practical solution then.

My sysadmin says:

    I'll ask about this in CommuniGate mailing list, but I think that
    implementation of IMAP SEARCH in CGPro is correct. This is a quote
    from press release:

    The POP and IMAP components of the CommuniGate Pro server not only
    support the latest Internet standards and all not-yet-standardized
    protocol extensions, but they also allow for simultaneous
    symmetric access to mailboxes. The high performance of IMAP
    built-in search engine results in huge benefits for individuals
    and organizations dealing with multi-thousand-message mailboxes.

    I think that problem is in our server - it's 650Mhz AMD with IDE
    hard drives.

But I'm suspicious that it's not the server since other mail clients
don't seem to have any particular problems with speed.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> I guess it doesn't come with source.  I recall the same problem was
>> identified quite some time ago, but perhaps it was never reported
>> upstream.  Asking the vendor to optimize the SEARCH commands for flags
>> is probably the only practical solution then.
>
> My sysadmin says:
>
>     I'll ask about this in CommuniGate mailing list, but I think that
>     implementation of IMAP SEARCH in CGPro is correct. This is a quote
>     from press release:
>
>     The POP and IMAP components of the CommuniGate Pro server not only
>     support the latest Internet standards and all not-yet-standardized
>     protocol extensions, but they also allow for simultaneous
>     symmetric access to mailboxes. The high performance of IMAP
>     built-in search engine results in huge benefits for individuals
>     and organizations dealing with multi-thousand-message mailboxes.

There is nothing wrong with the implementation, specification wise,
but if the server is slower to search for a read flag on a 100MB
article compared to searching for a read flag on 1KB file, the server
could be optimized.

>     I think that problem is in our server - it's 650Mhz AMD with IDE
>     hard drives.
>
> But I'm suspicious that it's not the server since other mail clients
> don't seem to have any particular problems with speed.

Few (if any) other clients use the SEARCH command like Gnus does, so
they don't trigger the same problem.

Re: nnimap slowness

[Krzysztof Jędruczyk]

Simon Josefsson <jas@extundo.com> writes:

> Gnus do send several SEARCH commands, and this is quite inefficient,
> but I believe fixing this would require changing the Gnus backend
> interface, and that means work.  Several servers implement flag
> searching without opening all articles though, so many people don't
> have this problem.
>

I don't think this has anything to do with the way gnus communicates
with IMAP server. I use Gnus at work, where I have dual-boot
Win2k/FreeBSD box. On Windoze big messages download very slowly, but
when I reboot to FreeBSD (both OS' have GNU-Emacs 21.2, Oort Gnus 0.15
installed) download is an order of magnitude faster. I tried
increasing TCP window size in Windows (I did little tcpdump inspection
and suspected that to be a problem), but it didn't help at all. So I
just let it be this way. But agreed - this is a major pain-in-the-ass
for w32+gnus+IMAP users...

Does anybody know if this problem also happens in XEmacs/W32 setup?

-- 
Best Regards,
        Krzysztof Jedruczyk

Re: nnimap slowness

[Simon Josefsson]

beaker@iavmb.pl (Krzysztof Jędruczyk) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Gnus do send several SEARCH commands, and this is quite inefficient,
>> but I believe fixing this would require changing the Gnus backend
>> interface, and that means work.  Several servers implement flag
>> searching without opening all articles though, so many people don't
>> have this problem.
>
> I don't think this has anything to do with the way gnus communicates
> with IMAP server. I use Gnus at work, where I have dual-boot
> Win2k/FreeBSD box. On Windoze big messages download very slowly, but
> when I reboot to FreeBSD (both OS' have GNU-Emacs 21.2, Oort Gnus 0.15
> installed) download is an order of magnitude faster. I tried
> increasing TCP window size in Windows (I did little tcpdump inspection
> and suspected that to be a problem), but it didn't help at all. So I
> just let it be this way. But agreed - this is a major pain-in-the-ass
> for w32+gnus+IMAP users...

Perhaps it is two unrelated problems, since David said it became slow
on large attachments which suggest the server do open the messages for
SEARCH commands.  But I wouldn't be surprised if the network stuff in
Emacs for Windows was inefficient somehow.  Report it as a Emacs bug?

Re: nnimap slowness

[Lars Magne Ingebrigtsen]

beaker@iavmb.pl (Krzysztof Jędruczyk) writes:

> On Windoze big messages download very slowly, but when I reboot to
> FreeBSD (both OS' have GNU-Emacs 21.2, Oort Gnus 0.15 installed)
> download is an order of magnitude faster.

I've seen suboptimal behavior when using `accept-process-output'
while waiting for network stuff.  nntp.el uses a timeout of 100msecs,
while imap.el uses a 1s timeout.

But you're seeing slowness on Windows?  Never mind.  Windows Emacs
doesn't even have msec timeouts.

Pretend I didn't say anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap slowness

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've seen suboptimal behavior when using `accept-process-output'
> while waiting for network stuff.  nntp.el uses a timeout of 100msecs,
> while imap.el uses a 1s timeout.

By the way, there's now a `nnheader-accept-process-output' function
that does a "short" timeout that can be used, well, anywhere.  It
should do the right thing under both Windows and operating systems.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap slowness

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> beaker@iavmb.pl (Krzysztof Jędruczyk) writes:
>
>> On Windoze big messages download very slowly, but when I reboot to
>> FreeBSD (both OS' have GNU-Emacs 21.2, Oort Gnus 0.15 installed)
>> download is an order of magnitude faster.
>
> I've seen suboptimal behavior when using `accept-process-output'
> while waiting for network stuff.  nntp.el uses a timeout of 100msecs,
> while imap.el uses a 1s timeout.

I added a imap-read-timeout.

Re: nnimap slowness

[Krzysztof Jędruczyk]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But you're seeing slowness on Windows?  Never mind.  Windows Emacs
> doesn't even have msec timeouts.

I can actually confirm that it's GNU Emacs/W32 problem. On friday I
installed XEmacs and gnus/imap worked as fast as I would expect it
to (I had installed the same cvs-head-as-of-friday version). Too bad
XEmacs for Windows doesn't support MULE yet, because it keeps using it
out of the question for me...

-- 
Best Regards,
        Krzysztof Jędruczyk

Re: nnimap slowness

[David Abrahams]

beaker@iavmb.pl (Krzysztof Jędruczyk) writes:

>> But you're seeing slowness on Windows?  Never mind.  Windows Emacs
>> doesn't even have msec timeouts.
>
> I can actually confirm that it's GNU Emacs/W32 problem. On friday I
> installed XEmacs and gnus/imap worked as fast as I would expect it
> to (I had installed the same cvs-head-as-of-friday version). 

Well, that's fabulous.  At least maybe I can be happy doing all my
GNUs stuff under xemacs... if I can figure out how to get past

     "This application has failed to start because cygXpm-noX4.dll
     was not found."

and, of course, making my .emacs file portable to XEmacs.  Anyone
know where there's a guide to that?

> Too bad XEmacs for Windows doesn't support MULE yet, because it
> keeps using it out of the question for me...

Hmm, did you report this difference in speed to the GNU Emacs
developers?

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[David Abrahams]

David Abrahams <dave@boost-consulting.com> writes:

> beaker@iavmb.pl (Krzysztof Jêdruczyk) writes:
> 
> >> But you're seeing slowness on Windows?  Never mind.  Windows Emacs
> >> doesn't even have msec timeouts.
> >
> > I can actually confirm that it's GNU Emacs/W32 problem. On friday I
> > installed XEmacs and gnus/imap worked as fast as I would expect it
> > to (I had installed the same cvs-head-as-of-friday version). 
> 
> Well, that's fabulous.  At least maybe I can be happy doing all my
> GNUs stuff under xemacs... if I can figure out how to get past
> 
>      "This application has failed to start because cygXpm-noX4.dll
>      was not found."
> 
> and, of course, making my .emacs file portable to XEmacs.  Anyone
> know where there's a guide to that?

OK, I dealt with all that, and my IMAP interactions indeed seem to be
much faster.  Now I'm seeing two weird effects in XEmacs/GNUs:

1. Every time I read a message, it gets an extra newline after it in
   my summary buffer!  Interestingly, the _other_ (unread) lines all seem
   to end in ^M according to what I'm seeing below, having copied/pasted
   from my summary buffer.

    *   17-Apr[  27: Aaron Moore         ] Re: Collaboration?
    *   17-Apr[  51: faisal vali         ] [boost] Re: Types and Programming Languages

    *   18-Apr[  53: Peter Dimov         ] Re: exception specifications
    *   18-Apr[  33: David Abrahams      ] Re: [boost] luabind
    *   18-Apr[ 128: Pete Becker         ] RE: Why integral_constant in type_traits?

    *       18-Apr<  27: Pete Becker         > 

    *R  18-Apr[  13: Jeremy Siek         ] boost-sandbox

    *   18-Apr[  65: ship-confirm@amazon.] Your Amazon.com order has shipped (#104-0735696-8985567)

    *RA 18-Apr[  34: Daveed Vandevoorde  ] Re: Haskell Metaprogramming

    *   18-Apr[  19: code_cold           ] [C++-sig] How to embed  python to c++

2. For some reason, delete-selection-mode (a.k.a. pending-delete-mode)
   does not always seem to take effect in message composition
   buffers.  It's very strange, because it seems to work unless I use
   <CR> to delete selected text.  Maybe this one is just an XEmacs
   quirk and nothing more...

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Kai Großjohann]

David Abrahams <dave@boost-consulting.com> writes:

> 1. Every time I read a message, it gets an extra newline after it in
>    my summary buffer!  Interestingly, the _other_ (unread) lines all seem
>    to end in ^M according to what I'm seeing below, having copied/pasted
>    from my summary buffer.

Sounds as if selective display is on.  Weird.
-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: nnimap slowness

[David Abrahams]

David Abrahams <dave@boost-consulting.com> writes:

> OK, I dealt with all that, and my IMAP interactions indeed seem to be
> much faster.  Now I'm seeing two weird effects in XEmacs/GNUs:
>
> 1. Every time I read a message, it gets an extra newline after it in
>    my summary buffer!  Interestingly, the _other_ (unread) lines all seem
>    to end in ^M according to what I'm seeing below, having copied/pasted
>    from my summary buffer.
>
>     *   17-Apr[  27: Aaron Moore         ] Re: Collaboration?
>     *   17-Apr[  51: faisal vali         ] [boost] Re: Types and Programming Languages
>
>     *   18-Apr[  53: Peter Dimov         ] Re: exception specifications
>     *   18-Apr[  33: David Abrahams      ] Re: [boost] luabind
>     *   18-Apr[ 128: Pete Becker         ] RE: Why integral_constant in type_traits?
>
>     *       18-Apr<  27: Pete Becker         > 
>
>     *R  18-Apr[  13: Jeremy Siek         ] boost-sandbox
>
>     *   18-Apr[  65: ship-confirm@amazon.] Your Amazon.com order has shipped (#104-0735696-8985567)
>
>     *RA 18-Apr[  34: Daveed Vandevoorde  ] Re: Haskell Metaprogramming
>
>     *   18-Apr[  19: code_cold           ] [C++-sig] How to embed  python to c++
>
> 2. For some reason, delete-selection-mode (a.k.a. pending-delete-mode)
>    does not always seem to take effect in message composition
>    buffers.  It's very strange, because it seems to work unless I use
>    <CR> to delete selected text.  Maybe this one is just an XEmacs
>    quirk and nothing more...

  3. When using `B c' to copy messages from my INBOX to other
     folders under XEmacs, openssl just hangs and I have to kill it
     in order to make any further progress.  Doing the same thing
     under GNU Emacs gives me no problems.

And, strangest of all:

  4. Some GNUs bindings I've come to rely on, e.g. `/ o' for
     gnus-summary-insert-old-articles seem to be completely missing!

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Kai Großjohann]

David Abrahams <dave@boost-consulting.com> writes:

>   4. Some GNUs bindings I've come to rely on, e.g. `/ o' for
>      gnus-summary-insert-old-articles seem to be completely missing!

Very weird indeed.  Are you using viper, perchance?  Viper would eat /.

-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: nnimap slowness

[David Abrahams]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>   4. Some GNUs bindings I've come to rely on, e.g. `/ o' for
>>      gnus-summary-insert-old-articles seem to be completely missing!
>
> Very weird indeed.  Are you using viper, perchance?  Viper would eat /.

No, this turned out to be much simpler: XEmacs was still using the
very old GNUs it ships with.  Stupid user error; I had messed up my
.emacs ;-)


-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[A.J. Rossini]

David Abrahams <dave@boost-consulting.com> writes:

> Well, that's fabulous.  At least maybe I can be happy doing all my
> GNUs stuff under xemacs... if I can figure out how to get past
>
>      "This application has failed to start because cygXpm-noX4.dll
>      was not found."

install the relevant library from cygwin (xpm-nox, or something like
that).  It's as obvious as I'm making it out to be, but it is in the
libs section of the installer.

best,
-tony

-- 
A.J. Rossini rossini@u.washington.edu http://software.biostat.washington.edu/ 
Biostatistics, U Washington and Fred Hutchinson Cancer Research Center 
FHCRC:Tu: 206-667-7025 (fax=4812)|Voicemail is pretty sketchy/use Email 
UW  : Th: 206-543-1044 (fax=3286)|Change last 4 digits of phone to FAX 
CONFIDENTIALITY NOTICE: This e-mail message and any attachments may be
confidential and privileged. If you received this message in error,
please destroy it and notify the sender. Thank you.

Re: nnimap slowness

[David Abrahams]

rossini@blindglobe.net (A.J. Rossini) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> Well, that's fabulous.  At least maybe I can be happy doing all my
>> GNUs stuff under xemacs... if I can figure out how to get past
>>
>>      "This application has failed to start because cygXpm-noX4.dll
>>      was not found."
>
> install the relevant library from cygwin (xpm-nox, or something like
> that).  It's as obvious as I'm making it out to be, but it is in the
> libs section of the installer.

Yeah, thanks, I figured that one out.

Also, other GNUs problems had to do with XEmacs using its version of
GNUs instead of the Oort 0.18 I've checked out of CVS.

However, actually trying to *install* ognus 0.18 for XEmacs proved
fatal.  I had to wipe out my XEmacs installation and start over.  It
kept looking for the missing directory /usr/local/i686-pc-cygwin/DOC
or something everytime my .emacs file used defadvice.

Now I've got it pointing at my CVS copy of GNUs (which I've been
careful *not* to byte-compile so that I can use it with XEmacs or
NTEmacs) and things seem to be basically happy (except, of course, for
messages like "you should byte-compile GNUs" ;-)).

Also, the pending-delete-mode behavior with RET is pretty hard to get
used to.  Select/hit-RET is my habitual way to break up quoted replies
and insert my own text.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Steinar Bang]

>>>>> David Abrahams <dave@boost-consulting.com>:

> Now I've got it pointing at my CVS copy of GNUs (which I've been
> careful *not* to byte-compile so that I can use it with XEmacs or
> NTEmacs) and things seem to be basically happy (except, of course,
> for messages like "you should byte-compile GNUs" ;-)).

Why not have two separate CVS checkouts, one for GNU Emacs, and one
for XEmacs?  It's what I do with the following in ~/.emacs (warning!
clumsy lisp by a non-lisper):

;; CVS version of Gnus:
(if (string-match "XEmacs" emacs-version)
    (progn
      (add-to-list 'load-path (expand-file-name "~sb/cvs/gnus.xemacs/lisp"))
      (let ((cvs-gnus-info-dir (expand-file-name "~sb/cvs/gnus.xemacs/texi")))
	(if (boundp 'Info-directory-list)
	    (add-to-list 'Info-directory-list cvs-gnus-info-dir)
	  (setq Info-directory-list (append
				     (list cvs-gnus-info-dir)
				     Info-default-directory-list)))))
  ;; GNU Emacs setup
  (add-to-list 'load-path (expand-file-name "~sb/cvs/gnus/lisp"))
  ;(load-library "bbdb-autoloads")
  (let ((cvs-gnus-info-dir (expand-file-name "~sb/cvs/gnus/texi")))
    (if (boundp 'Info-directory-list)
	(add-to-list 'Info-directory-list cvs-gnus-info-dir)
      (setq Info-directory-list (append
				 (list cvs-gnus-info-dir)
				 Info-default-directory-list)))))

A smarter way to handle that Info-directory-list would be
appreciated. 

Re: nnimap slowness

[Frank Schmitt]

David Abrahams <dave@boost-consulting.com> writes:

> However, actually trying to *install* ognus 0.18 for XEmacs proved
> fatal.  I had to wipe out my XEmacs installation and start over.  It
> kept looking for the missing directory /usr/local/i686-pc-cygwin/DOC
> or something everytime my .emacs file used defadvice.

You have to delete *.elc before issuing the make command when you
compiled for GNU Emacs before.

-- 
"You know the world is going crazy when the best rapper is a white guy,
the best golfer is a black guy, the Swiss hold the America's Cup, France
is accusing the US of arrogance, and Germany doesn't want to go to war."
          -- Unknown author

Re: nnimap slowness

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> To find out what causes slowness for small data, you must tcpdump or
> ethereal the stream to see which side is delaying the transfer, or if
> it is the network that has high latency.  ELP data isn't that useful
> in this case.

I got myself a copy of Ethereal and captured packets from a `M-G', but
I have no clue yet as to how to get any useful information from what
I'm seeing...

...I hate to keep harping on this, but it really has the look/feel of
some inefficiency in what GNUs is doing.

Looking at the packets one more time it sure looks very much as though
GNUs is receiving the same sequence of packets (about 27K worth) twice
from the IMAP server, though it's pretty hard for me to tell
anything, especially peering through SSL encoding.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap slowness

[Jody Klymak]

David Abrahams <dave@boost-consulting.com> writes:
>
>           imap read: 774K
>
> with the numbers spinning upwards at a rate of only about 15K per
> second I grow highly doubtful that it's doing anything other than
> reading data from my IMAP server.
>

I use windows XP (if that matters) and I can verify this behaviour. I
have at times despaired of gnus and switched to another mail-reader
(whose name I won't mention) because of this problem.  A large Mb size
document will take minutes to download.  If I am on dialup, it will
take forever.  I think that other mailreaders do not try and download
the whole attachment until it is asked for.  But thats just a guess.

I'll be happy to profile something as well.

Cheers,   Jody

Re: nnimap slowness

[Niklas Morberg]

Jody Klymak <jklymak@coas.oregonstate.edu> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>>
>>           imap read: 774K
>>
>> with the numbers spinning upwards at a rate of only about 15K per
>> second I grow highly doubtful that it's doing anything other than
>> reading data from my IMAP server.
>>
>
> I use windows XP (if that matters) and I can verify this behaviour. 

Windows 2000 and same problem here. I have a dual-boot
system and when I use Linux and the exact same version of
gnus, .gnus.el, and emacs downloading is much snappier.

I switch to Outlook whenever I get messages with large
attachments.

Niklas