Making IMAP fasterer

Making IMAP fasterer

[Jaspn Tibbitts]

I'm on vacation and instead of my usual method of ssh-ing into my work
machine and running emacs in a screen session, I thought I'd try setting
up gnus locally.  And it works, but it's pretty slow.  I'm running
Fedora 24, with emacs 25.0.94 the version of gnus which comes with it
(5.13).

I think that the issue is that the path to my IMAP server has a bit of
latency, which isn't surprising considering I'm on a Norwegian island
not far from the arctic circle.  I have always had gnus-asynchronous
set, and I remember in the past it used to flash messages about the
articles it was prefetching.  I haven't seen that in a while, and it
doesn't look like gnus is doing any article prefetching.  So selecting
an article takes a couple of seconds while it does a bunch of round
trips.

In addition, gnus-group-get-new-news always tells me "nnimap read NNNk
from imap.math.uh.edu (initial sync of 4 groups; please wait)" while it
fetches about 400k in total.  This takes about ten seconds every time i
hit 'g', which I'm trying to condition myself not to do.

So, does anyone have any tips for making it run a bit smoother?  My
.gnus file is below.  (Sadly, all of the bbdb stuff there is pointless
because I can't make that work either.  But that's a topic for another
email.)

 - J<

;; GNUS definitions and setup  -*- emacs-lisp -*-
(require 'bbdb)
(bbdb-initialize 'gnus 'message)
(require 'gnus-icalendar)
(gnus-icalendar-setup)

;;; GNUS Envornment variables
(setq
 mm-text-html-renderer              'shr
 mm-discouraged-alternatives        '("text/html" "text/richtext")
;mm-inline-override-types           (setq mm-discouraged-alternatives
;					  '("text/html"
;					    "text/richtext"))
 message-signature                  t
 message-sendmail-f-is-evil         t
 rmail-dont-reply-to-names          "tibbs"
 gnus-ignored-from-address          "tibbs"
 gnus-treat-date-lapsed             'head
 gnus-treat-highlight-headers       'head
 gnus-treat-highlight-citation      't
 gnus-treat-highlight-signature     't
 gnus-treat-fill-long-lines         nil
 gnus-read-active-file              'some
 gnus-check-new-newsgroups          'ask-server
 gnus-asynchronous                  t
 gnus-keep-backlog                  50
 gnus-uncacheable-groups            "^nnml"
 gnus-use-demon                     t
 gnus-use-article-prefetch          200
 gnus-prefetched-article-deletion-strategy '(exit)
 gnus-summary-check-current         t
 gnus-uu-do-not-unpack-archives     t
 gnus-score-exact-adapt-limit       8
 gnus-summary-zcore-fuzz            15
 gnus-verbose                       100
 gnus-use-cache                     t
 gnus-cache-directory               "~/News/cache"
 gnus-decay-scores                  t
 gnus-default-adaptive-score-alist  '((gnus-kill-file-mark)
				     (gnus-unread-mark)
				     (gnus-read-mark
				      (from  3) (subject  30))
				     (gnus-catchup-mark
				      (subject -10))
				     (gnus-killed-mark
				      (from -1) (subject -20))
				     (gnus-expirable-mark
				      (from -2) (subject -15))
				     (gnus-del-mark
				      (from -2) (subject -15))
				     )
 gnus-adaptive-word-no-group-words t
 gnus-article-save-directory       "~/News"
 gnus-auto-center-summary          t
 gnus-auto-mail-to-author          t
 gnus-auto-select-first            nil
 gnus-auto-select-same             nil
 gnus-break-pages                  t
 gnus-default-distribution         "world"
 gnus-fetch-old-headers            nil
 gnus-file-save-name               'gnus-plain-save-name
 gnus-followup-to                  t
 gnus-ignored-headers              (concat
				    "^Path:\\|^Posting-Version:\\|^Article-I.D.:"
				    "\\|^Expires:\\|^Date-Received:\\|^Control:"
				    "\\|^Xref:\\|^Posted:\\|^Relay-Version:"
				    "\\|^Message-ID:\\|^Nf-ID:\\|^Nf-From:"
				    "\\|^Sender:\\|^Nntp-Posting-Host:"
				    "\\|Nntp-Posting-User:\\|^Originator:"
				    )
 gnus-interactive-post             t
 gnus-kill-file-name               "KILL"
 gnus-score-file-name              "SCORE"
 gnus-large-newsgroup              200
 gnus-local-domain                 "math.uh.edu"
 gnus-local-timezone               "CST"
 gnus-message-archive-method       `(nnfolder "archive"
					      (nnfolder-directory "~/Mail/Outgoing/")
					      (nnfolder-active-file "~/Mail/Outgoing/active")
					      (nnfolder-inhibit-expiry t)
					      (nnfolder-get-new-mail nil))
 gnus-message-archive-group        '((format-time-string
				      (if (message-news-p) "%Y/news-%m-%B" "%Y/mail-%m-%B")
				      (current-time)))
 gnus-novice-user                  t
 gnus-score-expiry-days            28
 gnus-secondary-select-methods     '((nnml "")); (nnslashdot ""))
 gnus-select-method                '(nnimap "imap.math.uh.edu")
;         (nnimap-stream gssapi)
;         (nnimap-authenticator gssapi))
 nnimap-fetch-partial-articles     t

 gnus-show-all-headers             nil
 gnus-summary-default-interest     0
 gnus-summary-gather-subject-limit 'fuzzy
 gnus-article-mode-line-format     "Gnus: %g/%A %S"
 gnus-summary-make-false-root      'adopt
 gnus-summary-mode-line-format     "Gnus: %g %Z"
 gnus-thread-hide-killed           t
 gnus-thread-hide-subtree          t
 gnus-thread-ignore-subject        t
 gnus-thread-indent-level          4
 gnus-thread-score-function        'max
 gnus-thread-sort-functions    	   '(gnus-thread-sort-by-total-score)
 gnus-use-adaptive-scoring         '(line)
 gnus-use-cross-reference          'neither
 gnus-use-full-window              t
 gnus-use-header-prefetch          t
 gnus-use-long-file-name           t
 gnus-build-sparse-threads         nil
 gnus-tree-line-format             "%[%]"
 gnus-use-trees                    t
 gnus-tree-minimize-window         nil
 nnmail-split-methods              nil
 bbdb-send-mail-style              'gnus
 bbdb-complete-mail-full-completion t
 bbdb-completion-list              'primary-or-name
 bbdb-complete-mail-allow-cycling  t
 bbdb-offer-save                   1
 bbdb-use-pop-up                   t
 bbdb-electric                     t
 bbdb-popup-target-lines           1

 mail-sources                      '((directory :path "/home/tibbs/Mail/spool/"
					       :suffix ".spool")
				     )

 mail-source-directory             (expand-file-name "~/Mail/tmp")
 nnmail-procmail-directory         (expand-file-name "~/Mail/spool")
 nnml-get-new-mail                 t
 nntp-buggy-select                 nil
)

(gnus-add-configuration
 '(article
   '(vertical 1.0
	      (horizontal 0.25
			  (summary 1.0 point)
			  (tree 0.25))
	      (article 1.0)
	      )))

(add-hook 'news-inews-hook 'ispell-message)
(add-hook 'message-mode-hook 'flyspell-mode)
;(remove-hook 'gnus-summary-prepare-exit-hook 'gnus-summary-expire-articles)

;(gnus-demon-add-handler 'gnus-demon-scan-mail 15 5)
;(gnus-demon-add-handler 'gnus-group-expire-all-groups nil 60)
(gnus-start-date-timer 60)

Re: Making IMAP fasterer

[Steinar Bang]

>>>>> Jaspn Tibbitts <tibbs@math.uh.edu>:

> I think that the issue is that the path to my IMAP server has a bit of
> latency, which isn't surprising considering I'm on a Norwegian island
> not far from the arctic circle.

Which one?

Re: Making IMAP fasterer

[Steinar Bang]

>>>>> Jaspn Tibbitts <tibbs@math.uh.edu>:

> I think that the issue is that the path to my IMAP server has a bit of
> latency, which isn't surprising considering I'm on a Norwegian island
> not far from the arctic circle.  I have always had gnus-asynchronous
> set,

FWIW my value og gnus-asynchronous is nil.

> and I remember in the past it used to flash messages about the
> articles it was prefetching.  I haven't seen that in a while, and it
> doesn't look like gnus is doing any article prefetching.  So selecting
> an article takes a couple of seconds while it does a bunch of round
> trips.

I do no preselect, so this happens for me as well on a slow line.

However I have set gnus-agent to cache all articles read, so the next
time is much faster.

I never use agent prefetch because I've found its behaviour
annoying when I first started using agent, and have never had any
incentive to revisit it (there is a size cutoff that at the time left
out many of the articles I wanted to read, and I've never looked into
adjusting that to something useful).

> In addition, gnus-group-get-new-news always tells me "nnimap read NNNk
> from imap.math.uh.edu (initial sync of 4 groups; please wait)" while it
> fetches about 400k in total.  This takes about ten seconds every time i
> hit 'g', which I'm trying to condition myself not to do.

Hm... that this happens the first time isn't any suprprise, but that it
happens every time is.  Do you use the agent?

> So, does anyone have any tips for making it run a bit smoother?

I dunno, use the agent maybe?  Seems to work for me, and have done so
since the late ninties.

Re: Making IMAP fasterer

[Eric Abrahamsen]

Jaspn Tibbitts <tibbs@math.uh.edu> writes:

> I'm on vacation and instead of my usual method of ssh-ing into my work
> machine and running emacs in a screen session, I thought I'd try setting
> up gnus locally.  And it works, but it's pretty slow.  I'm running
> Fedora 24, with emacs 25.0.94 the version of gnus which comes with it
> (5.13).
>
> I think that the issue is that the path to my IMAP server has a bit of
> latency, which isn't surprising considering I'm on a Norwegian island
> not far from the arctic circle.  I have always had gnus-asynchronous
> set, and I remember in the past it used to flash messages about the
> articles it was prefetching.  I haven't seen that in a while, and it
> doesn't look like gnus is doing any article prefetching.  So selecting
> an article takes a couple of seconds while it does a bunch of round
> trips.
>
> In addition, gnus-group-get-new-news always tells me "nnimap read NNNk
> from imap.math.uh.edu (initial sync of 4 groups; please wait)" while it
> fetches about 400k in total.  This takes about ten seconds every time i
> hit 'g', which I'm trying to condition myself not to do.

That's a bit odd, it sounds like the groups are syncing, but the data
isn't getting saved properly or something, so it resyncs every time.

What IMAP server are you connecting to? How many groups are there? Is it
always four groups that do the initial sync? Can you tell if they're the
same four groups every time?

One thing you might try, if you're comfortable with edebugging, is going
into the lisp/gnus/nnimap.el file, finding the function
`nnimap-retrieve-group-data-early', and stepping through it. Initial
sync only happens in the `if' clause towards the bottom of the function
(probably line 1446 in your version). You could step through the
function and see which of the active|uidvalidity|unexist vars are nil at
that stage.

Who knows, maybe that will be enough to tip someone here off to the
nature of the problem.

Yours,
Eric

Re: Making IMAP fasterer

[Jaspn Tibbitts]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Jaspn Tibbitts <tibbs@math.uh.edu> writes:

> That's a bit odd, it sounds like the groups are syncing, but the data
> isn't getting saved properly or something, so it resyncs every time.

My .newsrc.eld dates from, oh, gnus 0.6 if not older.  Probably from
when gnus first started writing out newsrc.eld.

> What IMAP server are you connecting to? How many groups are there?

It's Cyrus.  Sadly an older version, 2.3.16 with a couple of security
patches.

> Is it always four groups that do the initial sync? Can you tell if
> they're the same four groups every time?

It's always four.  I'm not sure how I'd tell if it's the same four.

> One thing you might try, if you're comfortable with edebugging

Yeah, I can do that.  Might take me a bit to swap all of that emacs
knowledge back in but I'll see what I can come up with.

Would it be worth building a development version of emacs so I can get
the current gnus code?

Also, is it possible to just purge .newsrc.eld of everything besides my
subscriptions and scoring and group parameters and such, and have it
just resync everything?  I honestly have no idea how much old cruft is
in there.

 - J<

Re: Making IMAP fasterer

[Jaspn Tibbitts]

Steinar Bang <sb@dod.no> writes:

>>>>>> Jaspn Tibbitts <tibbs@math.uh.edu>:

> I do no preselect, so this happens for me as well on a slow line.

Yeah, I was trying to have it do some readahead.  And I know it used to
but at some point (probably not recently) it stopped.

> However I have set gnus-agent to cache all articles read, so the next
> time is much faster.

Except that I'm not really interested in looking at old mail; I'd just
like reading new mail to be a bit faster.

> Hm... that this happens the first time isn't any suprprise, but that it
> happens every time is.  Do you use the agent?

I don't think so.  It's always possible I turned it on at some point in
the last, oh, 25+ years that I've used gnus, but nothing matches /agent/
in that .gnus I attached.

 - J<

Re: Making IMAP fasterer

[Eric Abrahamsen]

Jaspn Tibbitts <tibbs@math.uh.edu> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Jaspn Tibbitts <tibbs@math.uh.edu> writes:
>
>> That's a bit odd, it sounds like the groups are syncing, but the data
>> isn't getting saved properly or something, so it resyncs every time.
>
> My .newsrc.eld dates from, oh, gnus 0.6 if not older.  Probably from
> when gnus first started writing out newsrc.eld.
>
>> What IMAP server are you connecting to? How many groups are there?
>
> It's Cyrus.  Sadly an older version, 2.3.16 with a couple of security
> patches.

Here's where I hope someone with more intimate knowledge of IMAP servers
might jump in. I do know that Cyrus, particularly older versions of
Cyrus, are quirky, and it's not impossible that that's interacting badly
with Gnus in some way.

>> Is it always four groups that do the initial sync? Can you tell if
>> they're the same four groups every time?
>
> It's always four.  I'm not sure how I'd tell if it's the same four.

I don't either. But my guess is that it is the same four, and the
problem is in the data storage in your newsrc.eld.

>> One thing you might try, if you're comfortable with edebugging
>
> Yeah, I can do that.  Might take me a bit to swap all of that emacs
> knowledge back in but I'll see what I can come up with.
>
> Would it be worth building a development version of emacs so I can get
> the current gnus code?

No, if you're already using Emacs 25, the nnimap code won't have changed
significantly.

> Also, is it possible to just purge .newsrc.eld of everything besides my
> subscriptions and scoring and group parameters and such, and have it
> just resync everything?  I honestly have no idea how much old cruft is
> in there.

The .newsrc.eld file is a delicate creature, but I think this will end
up being the solution. Deleting marks for non-imap servers might cause
you difficulty, but if it's only imap servers then gnus should behave
correctly: initial syncs for all the groups, then normal behavior from
there on out. Theoretically. Don't take my word for it: wait until
someone else chimes in, then back up your file, cut out the imap marks,
and try re-syncing. I'd also shut off the asynchronous stuff first --
try to make the resync as "clean" as possible.

If I were you, I'd also do the edebug test first. Who knows, you might
see something illuminating.

Eric

Re: Making IMAP fasterer

[Steinar Bang]

>>>>> Jaspn Tibbitts <tibbs@math.uh.edu>:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> What IMAP server are you connecting to? How many groups are there?

> It's Cyrus.  Sadly an older version, 2.3.16 with a couple of security
> patches.

FWIW I went from UoW imapd to cyrus in 1999 or 2000 or thereabouts,
because of quirks with the UoW imapd, then had a lot of issues for the
years I used cyrus (e.g. lost all of my marks twice during upgrades).

However, I switched to dovecot in 2008 and have had trouble free IMAP
operation since.
 nntp://news.gmane.org/gmane.mail.imap.dovecot/32706
 http://permalink.gmane.org/gmane.mail.imap.dovecot/32706

Re: Making IMAP fasterer

[Steinar Bang]

>>>>> Jaspn Tibbitts <tibbs@math.uh.edu>:

> In addition, gnus-group-get-new-news always tells me "nnimap read NNNk
> from imap.math.uh.edu (initial sync of 4 groups; please wait)" while
> it fetches about 400k in total.  This takes about ten seconds every
> time i hit 'g', which I'm trying to condition myself not to do.

Ok, now I've seen this as well.  I'm on vacation, and have been using a
(supposedly) 4G ICE.net router, that is a replacement for my old 450MHz
ICE.net router, that no longer works, because they have switched off the
450MHz ntwork.

So far the new router has been a disappointment, and I've ended up using
campsites' free wi-fi or my cellphone instead... but anyway: I have now
tested slooow connections and it can be a pain to watch nnimap fetch the
same data time after time.

As you say, it says "initial sync", even though it fetches the same data
multiple times, so a guess is that it wasn't *intended* to be fetched
multiple times...?

Re: Making IMAP fasterer

[Jason Tibbitts]

Steinar Bang <sb@dod.no> writes:

>>>>>> Jaspn Tibbitts <tibbs@math.uh.edu>:

> So far the new router has been a disappointment, and I've ended up using
> campsites' free wi-fi or my cellphone instead... but anyway: I have now
> tested slooow connections and it can be a pain to watch nnimap fetch the
> same data time after time.

It's good to know that I'm not having some weird completely unique
problem.

However, I haven't really found the time to dig into it, because I had
better network latency in Trondheim and now that I'm in Bergen things
are fast enough that it isn't really an issue.  Plus I've trained myself
to not hit 'g' so often.

> As you say, it says "initial sync", even though it fetches the same data
> multiple times, so a guess is that it wasn't *intended* to be fetched
> multiple times...?

Are you always seeing it fetch four groups?  Or do you see a different
number?

 - J<

Re: Making IMAP fasterer

[Steinar Bang]

>>>>> Jason Tibbitts <tibbs@math.uh.edu>:

> Are you always seeing it fetch four groups?  Or do you see a different
> number?

For me it's just one group it fetches.

But the difference here could be either in my setup, or in the old Gnus
I'm running (the final git gnus before merging with emacs).

Re: Making IMAP fasterer

[Dave Abrahams]

on Sat Jul 09 2016, Jaspn Tibbitts <tibbs-AT-math.uh.edu> wrote:

> I'm on vacation and instead of my usual method of ssh-ing into my work
> machine and running emacs in a screen session, I thought I'd try setting
> up gnus locally.  And it works, but it's pretty slow.  I'm running
> Fedora 24, with emacs 25.0.94 the version of gnus which comes with it
> (5.13).
>
> I think that the issue is that the path to my IMAP server has a bit of
> latency, which isn't surprising considering I'm on a Norwegian island
> not far from the arctic circle.  I have always had gnus-asynchronous
> set, and I remember in the past it used to flash messages about the
> articles it was prefetching.  I haven't seen that in a while, and it
> doesn't look like gnus is doing any article prefetching.  So selecting
> an article takes a couple of seconds while it does a bunch of round
> trips.
>
> In addition, gnus-group-get-new-news always tells me "nnimap read NNNk
> from imap.math.uh.edu (initial sync of 4 groups; please wait)" while it
> fetches about 400k in total.  This takes about ten seconds every time i
> hit 'g', which I'm trying to condition myself not to do.
>
> So, does anyone have any tips for making it run a bit smoother?  

I use http://onmsg.github.io

-- 
-Dave