Speeding up IMAP parsing

Speeding up IMAP parsing

[Lars Magne Ingebrigtsen]

Using `read' to read the flags would be really fast and create very
little garbage.  The look like this:

* 140 FETCH (FLAGS (\Recent) UID 140).

So I need to change the syntax table so that backspace becomes a
word-constituent character:

    (let ((syntax-table (copy-syntax-table emacs-lisp-mode-syntax-table)))
      (modify-syntax-entry ?\\ "w" syntax-table)
      (set-syntax-table syntax-table))

But placing point before the parenthesis and calling

(read (current-buffer))

just gives me

(FLAGS (Recent) UID 141)

Doesn't `read' use the syntax table?  Or is some other magic necessary? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Speeding up IMAP parsing

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Doesn't `read' use the syntax table?  Or is some other magic necessary? 

Quoth the raven:

---------
   Syntax tables are used only for moving across text, not for the Emacs
Lisp reader.  Emacs Lisp uses built-in syntactic rules when reading Lisp
expressions, and these rules cannot be changed.
---------

Oh, well.  It's not really important -- "Deleted" is as clear as
"\Deleted".

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Speeding up IMAP parsing

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Oh, well.  It's not really important -- "Deleted" is as clear as
> "\Deleted".

But couldn't "Deleted" be a custom flag?

What about doing a search-and-replace ahead of time, changing every "\"
to "XS" (X is an escape character and B is for backslash) and changing
every "X" to "XX"?

Sounds annoying to deal with...

Dan

Re: Speeding up IMAP parsing

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

>> Oh, well.  It's not really important -- "Deleted" is as clear as
>> "\Deleted".
>
> But couldn't "Deleted" be a custom flag?

Yes, it could.

> What about doing a search-and-replace ahead of time, changing every "\"
> to "XS" (X is an escape character and B is for backslash) and changing
> every "X" to "XX"?

Looking at the rfc, the % character isn't valid in the flags, but it has
char syntax for the `read' command, so substing \ with % before reading
should do the trick (and be really fast).

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Speeding up IMAP parsing

[Dan Christensen]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Looking at the rfc, the % character isn't valid in the flags, but it has
> char syntax for the `read' command, so substing \ with % before reading
> should do the trick (and be really fast).

I did a timing comparison between my Dec 2009 version of Gnus and
current git when entering an IMAP group with about 6500 articles.

Dec 2009:  11.1 seconds
git:       10.8 seconds

Great!

With the patch below applied, the time reduces further to 10.4 seconds.
But I don't know if it will break things for others.  (I've used it for
several months now without noticing a problem.)

Interestingly, with my Dec 2009 version, the patch below reduces the
time from 11.1 to 10.5 seconds, a bigger change than with git, and
almost matching the git+patch time.

Did the older nnimap.el use the read trick?  If not, then it must have
been more efficient in some other way to come so close to the time
that the current version uses.

Dan


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: time-date.patch --]
[-- Type: text/x-diff, Size: 1590 bytes --]

diff --git a/lisp/time-date.el b/lisp/time-date.el
index 5dbd08c..01132bc 100644
--- a/lisp/time-date.el
+++ b/lisp/time-date.el
@@ -97,20 +97,20 @@ and type 2 is the list (HIGH LOW MICRO)."
 (autoload 'timezone-make-date-arpa-standard "timezone")
 
 ;;;###autoload
+;; `parse-time-string' isn't sufficiently general or robust.  It fails
+;; to grok some of the formats that timezone does (e.g. dodgy
+;; post-2000 stuff from some Elms) and either fails or returns bogus
+;; values.  timezone-make-date-arpa-standard should help.
 (defun date-to-time (date)
   "Parse a string DATE that represents a date-time and return a time value.
 If DATE lacks timezone information, GMT is assumed."
   (condition-case ()
-      (apply 'encode-time
-	     (parse-time-string
-	      ;; `parse-time-string' isn't sufficiently general or
-	      ;; robust.  It fails to grok some of the formats that
-	      ;; timezone does (e.g. dodgy post-2000 stuff from some
-	      ;; Elms) and either fails or returns bogus values.  Lars
-	      ;; reverted this change, but that loses non-trivially
-	      ;; often for me.  -- fx
-	      (timezone-make-date-arpa-standard date)))
-    (error (error "Invalid date: %s" date))))
+      (apply 'encode-time (parse-time-string date))
+    (error (condition-case ()
+	       (apply 'encode-time 
+		      (parse-time-string
+		       (timezone-make-date-arpa-standard date)))
+	     (error (error "Invalid date: %s" date))))))
 
 ;; Bit of a mess.  Emacs has float-time since at least 21.1.
 ;; This file is synced to Gnus, and XEmacs packages may have been written

Re: Speeding up IMAP parsing

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> With the patch below applied, the time reduces further to 10.4 seconds.
> But I don't know if it will break things for others.  (I've used it for
> several months now without noticing a problem.)

Thanks; I've applied the patch and I guess we'll see...

> Interestingly, with my Dec 2009 version, the patch below reduces the
> time from 11.1 to 10.5 seconds, a bigger change than with git, and
> almost matching the git+patch time.
>
> Did the older nnimap.el use the read trick?  If not, then it must have
> been more efficient in some other way to come so close to the time
> that the current version uses.

Group entry doesn't use the `read' trick -- it's just flags parsing and
stuff.  I haven't really looked at the nnimap group entry efficiency;
I've mainly concentrated on `g' speed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Speeding up IMAP parsing

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> With the patch below applied, the time reduces further to 10.4 seconds.
>> But I don't know if it will break things for others.  (I've used it for
>> several months now without noticing a problem.)
>
> Thanks; I've applied the patch and I guess we'll see...

Ok!

> Group entry doesn't use the `read' trick -- it's just flags parsing and
> stuff.  I haven't really looked at the nnimap group entry efficiency;
> I've mainly concentrated on `g' speed.

Oh, right, I was confused.  My main IMAP server is local, and `g' is
nice and snappy.  I'll report some timing info later in the thread
you just started.

I enter groups much more often than I run `g', though, so there are
still some areas in which you can work your magic!  

Dan

Re: Speeding up IMAP parsing

[Ted Zlatanov]

On Wed, 22 Sep 2010 02:29:10 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Using `read' to read the flags would be really fast and create very
LMI> little garbage.

Sounds like a custom C function could be useful if this is a bottleneck.

Ted

Re: Speeding up IMAP parsing

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Sounds like a custom C function could be useful if this is a bottleneck.

The Emacs admins don't really seem keen on having anything in C... 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen