Debugger entered--Lisp error: (overflow-error "13419098521433281274")

Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Tassilo Horn]

Hi all,

since a few minutes, I get this error when hitting `g' in *Group*.

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (overflow-error "13419098521433281274")
  read(#<buffer *nnimap mail.messagingengine.com nil  *nntpd**>)
  nnimap-parse-flags(...)
  nnimap-finish-retrieve-group-infos(...)
  gnus-finish-retrieve-group-infos(...)
  gnus-read-active-for-groups(...)
  gnus-get-unread-articles(nil)
  gnus-group-get-new-news(nil)
  call-interactively(gnus-group-get-new-news nil nil)
--8<---------------cut here---------------end--------------->8---

That number is bigger than `most-positive-fixnum' on my 64bit system, so
it cannot be `read'. :-(

The contents in that buffer around that number are as follows:

--8<---------------cut here---------------start------------->8---
* OK [CLOSED] Ok
* 2971 EXISTS
* 1 RECENT
* FLAGS (%Answered %Flagged %Draft %Deleted %Seen hasnoatt hasatt hasnotd hastd gnus-forward gnus-expire)
* OK [PERMANENTFLAGS ()] Ok
* OK [UNSEEN 2971] Ok
* OK [UIDVALIDITY 1103381936] Ok
* OK [UIDNEXT 12510] Ok
* OK [HIGHESTMODSEQ 4858] Ok
* OK [URLMECH INTERNAL] Ok
* 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))
3901 OK [READ-WRITE] Completed
* OK [CLOSED] Ok
* 0 EXISTS
* 0 RECENT
--8<---------------cut here---------------end--------------->8---

Hm, after restarting gnus (which fetched several megabytes from that
imap server), the issue seems to have vanished...  But I'm still
interested what that number actually means?

Bye,
Tassilo
-- 
Sent from my Emacs

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Tassilo Horn]

Damn!  Now I get that error (with another big number) as soon as I try
to start Gnus.  I even don't get the group list anymore, so it's
rendering Gnus totally unfunctional for me. :-(

Any help?

Bye,
Tassilo

On Wed, 02 Mar 2011 16:28 +0100, "Tassilo Horn" <tassilo@member.fsf.org>
wrote:
> Hi all,
> 
> since a few minutes, I get this error when hitting `g' in *Group*.
> 
> --8<---------------cut here---------------start------------->8---
> Debugger entered--Lisp error: (overflow-error "13419098521433281274")
>   read(#<buffer *nnimap mail.messagingengine.com nil  *nntpd**>)
>   nnimap-parse-flags(...)
>   nnimap-finish-retrieve-group-infos(...)
>   gnus-finish-retrieve-group-infos(...)
>   gnus-read-active-for-groups(...)
>   gnus-get-unread-articles(nil)
>   gnus-group-get-new-news(nil)
>   call-interactively(gnus-group-get-new-news nil nil)
> --8<---------------cut here---------------end--------------->8---
> 
> That number is bigger than `most-positive-fixnum' on my 64bit system, so
> it cannot be `read'. :-(
> 
> The contents in that buffer around that number are as follows:
> 
> --8<---------------cut here---------------start------------->8---
> * OK [CLOSED] Ok
> * 2971 EXISTS
> * 1 RECENT
> * FLAGS (%Answered %Flagged %Draft %Deleted %Seen hasnoatt hasatt hasnotd
> hastd gnus-forward gnus-expire)
> * OK [PERMANENTFLAGS ()] Ok
> * OK [UNSEEN 2971] Ok
> * OK [UIDVALIDITY 1103381936] Ok
> * OK [UIDNEXT 12510] Ok
> * OK [HIGHESTMODSEQ 4858] Ok
> * OK [URLMECH INTERNAL] Ok
> * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))
> 3901 OK [READ-WRITE] Completed
> * OK [CLOSED] Ok
> * 0 EXISTS
> * 0 RECENT
> --8<---------------cut here---------------end--------------->8---
> 
> Hm, after restarting gnus (which fetched several megabytes from that
> imap server), the issue seems to have vanished...  But I'm still
> interested what that number actually means?
> 
> Bye,
> Tassilo
> -- 
> Sent from my Emacs
> 
> 

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Andreas Schwab]

Tassilo Horn <tassilo@member.fsf.org> writes:

> But I'm still interested what that number actually means?

This is an opaque (unsigned 64-bit) value updated by the server whenever
a metadata item is modified (RFC4551).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Tassilo Horn]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> But I'm still interested what that number actually means?
>
> This is an opaque (unsigned 64-bit) value updated by the server
> whenever a metadata item is modified (RFC4551).

That's a bit unfortunate.  Today I've encountered 2 of these values that
were too big to be `read' by emacs.

Looking at `nnimap-parse-flags' I can see that the value of UIDVALIDITY
is extracted by string-matching and stored as string, because it might
be bigger than an emacs integer.  Probably the same has to be done for
the FETCH lines.

Here's the offending code:

--8<---------------cut here---------------start------------->8---
	  (while (re-search-forward "^\\* [0-9]+ FETCH " start t)
	    (setq elems (read (current-buffer)))
	    (push (cons (cadr (memq 'UID elems))
			(cadr (memq 'FLAGS elems)))
		  articles))
--8<---------------cut here---------------end--------------->8---

Of course, it is convenient to `read' the complete list in lines like

  * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))

but it fails for large numbers.  And if the RFC says those are 64bit
unsigned ints, well, then Gnus has to handle those.

Bye,
Tassilo

  ...who wonders why he got this error twice today after using Gnus with
  IMAP servers (including that problematic one) for several years with
  no such issue at all...

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Ted Zlatanov]

On Wed, 02 Mar 2011 20:24:17 +0100 Tassilo Horn <tassilo@member.fsf.org> wrote: 

TH> That's a bit unfortunate.  Today I've encountered 2 of these values that
TH> were too big to be `read' by emacs.

TH> Looking at `nnimap-parse-flags' I can see that the value of UIDVALIDITY
TH> is extracted by string-matching and stored as string, because it might
TH> be bigger than an emacs integer.  Probably the same has to be done for
TH> the FETCH lines.

TH> Here's the offending code:
TH> 	  (while (re-search-forward "^\\* [0-9]+ FETCH " start t)
TH> 	    (setq elems (read (current-buffer)))
TH> 	    (push (cons (cadr (memq 'UID elems))
TH> 			(cadr (memq 'FLAGS elems)))
TH> 		  articles))

TH> Of course, it is convenient to `read' the complete list in lines like

TH>   * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))

TH> but it fails for large numbers.  And if the RFC says those are 64bit
TH> unsigned ints, well, then Gnus has to handle those.

Gnus has no way to do it when Emacs can't.  The calc package has some
facilities for big numbers but yeah, (read) will not DTRT with input
like that.  I've asked for it on emacs-devel.

To fix it now, I would replace such large numbers with a truncated
version, which can be done safely with a regex before the reader sees
the input.  But I'll let Lars decide, he knows that code best.

Ted

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Tassilo Horn]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> TH> Here's the offending code:
> TH> 	  (while (re-search-forward "^\\* [0-9]+ FETCH " start t)
> TH> 	    (setq elems (read (current-buffer)))
> TH> 	    (push (cons (cadr (memq 'UID elems))
> TH> 			(cadr (memq 'FLAGS elems)))
> TH> 		  articles))
>
> TH> Of course, it is convenient to `read' the complete list in lines like
>
> TH>   * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))
>
> TH> but it fails for large numbers.  And if the RFC says those are 64bit
> TH> unsigned ints, well, then Gnus has to handle those.
>
> Gnus has no way to do it when Emacs can't.  The calc package has some
> facilities for big numbers but yeah, (read) will not DTRT with input
> like that.  I've asked for it on emacs-devel.
>
> To fix it now, I would replace such large numbers with a truncated
> version, which can be done safely with a regex before the reader sees
> the input.  But I'll let Lars decide, he knows that code best.

But that huge number is not used by gnus, anyway.  It is `read' but only
the UID and FLAGS values are saved.  So IMHO, the proper fix is to put a
condition-case around the `read' that picks the UID and FLAGS with
string-matching if an overflow-error occurs.

Bye,
Tassilo
-- 
Sent from my Emacs

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Andreas Schwab]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
> Hi Ted,
>
>> TH> Here's the offending code:
>> TH> 	  (while (re-search-forward "^\\* [0-9]+ FETCH " start t)
>> TH> 	    (setq elems (read (current-buffer)))
>> TH> 	    (push (cons (cadr (memq 'UID elems))
>> TH> 			(cadr (memq 'FLAGS elems)))
>> TH> 		  articles))
>>
>> TH> Of course, it is convenient to `read' the complete list in lines like
>>
>> TH>   * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))
>>
>> TH> but it fails for large numbers.  And if the RFC says those are 64bit
>> TH> unsigned ints, well, then Gnus has to handle those.
>>
>> Gnus has no way to do it when Emacs can't.  The calc package has some
>> facilities for big numbers but yeah, (read) will not DTRT with input
>> like that.  I've asked for it on emacs-devel.
>>
>> To fix it now, I would replace such large numbers with a truncated
>> version, which can be done safely with a regex before the reader sees
>> the input.  But I'll let Lars decide, he knows that code best.
>
> But that huge number is not used by gnus, anyway.  It is `read' but only
> the UID and FLAGS values are saved.  So IMHO, the proper fix is to put a
> condition-case around the `read' that picks the UID and FLAGS with
> string-matching if an overflow-error occurs.

That won't work since you won't be able to read the rest either (since
it is read as a single sexp).  A possible fix would be to insert a space
into the middle of any number sequence that is longer than 9 digits.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Tassilo Horn]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> But that huge number is not used by gnus, anyway.  It is `read' but
>> only the UID and FLAGS values are saved.  So IMHO, the proper fix is
>> to put a condition-case around the `read' that picks the UID and
>> FLAGS with string-matching if an overflow-error occurs.
>
> That won't work since you won't be able to read the rest either (since
> it is read as a single sexp).  A possible fix would be to insert a
> space into the middle of any number sequence that is longer than 9
> digits.

I don't get you.  My idea was to try a `read', and if that errors, use
`search-forward' and `looking-at' to get the UID and FLAGS values
without having to `read' them as lisp objects.

Bye,
Tassilo

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Ted Zlatanov]

On Wed, 02 Mar 2011 21:36:23 +0100 Tassilo Horn <tassilo@member.fsf.org> wrote: 

TH> Ted Zlatanov <tzz@lifelogs.com> writes:

TH> Of course, it is convenient to `read' the complete list in lines like
TH> * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))
TH> but it fails for large numbers.  And if the RFC says those are 64bit
TH> unsigned ints, well, then Gnus has to handle those.
>> 
>> Gnus has no way to do it when Emacs can't.  The calc package has some
>> facilities for big numbers but yeah, (read) will not DTRT with input
>> like that.  I've asked for it on emacs-devel.
>> 
>> To fix it now, I would replace such large numbers with a truncated
>> version, which can be done safely with a regex before the reader sees
>> the input.  But I'll let Lars decide, he knows that code best.

TH> But that huge number is not used by gnus, anyway.  It is `read' but only
TH> the UID and FLAGS values are saved.  So IMHO, the proper fix is to put a
TH> condition-case around the `read' that picks the UID and FLAGS with
TH> string-matching if an overflow-error occurs.

Yeah, that would work too, but it's a hack either way.  My way would be
less work since we just rewrite that number to 1 or whatever, which is
one extra line of code.  If you want to submit a patch for either hack
go ahead, otherwise I'll wait for Lars to fix it.

Ted

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Tassilo Horn]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

>>> To fix it now, I would replace such large numbers with a truncated
>>> version, which can be done safely with a regex before the reader
>>> sees the input.  But I'll let Lars decide, he knows that code best.
>
> TH> But that huge number is not used by gnus, anyway.  It is `read'
> TH> but only the UID and FLAGS values are saved.  So IMHO, the proper
> TH> fix is to put a condition-case around the `read' that picks the
> TH> UID and FLAGS with string-matching if an overflow-error occurs.
>
> Yeah, that would work too, but it's a hack either way.  My way would
> be less work since we just rewrite that number to 1 or whatever, which
> is one extra line of code.  If you want to submit a patch for either
> hack go ahead, otherwise I'll wait for Lars to fix it.

I've now combined both approaches.  A condition-case handler simply
deletes the big number and `read's again.  That fixes the issue for me.
I added a FIXME and comment, too.  Maybe Lars has some better idea how
to deal with that.

Since that number occurs as value of MODSEQ (which Gnus doesn't use), is
the HIGHESTMODSEQ (which Gnus reads and stores as number) also a
candidate for overflowing?  It probably is, so it might be better to
handle it as string just like the UIDVALIDITY.

Bye,
Tassilo
-- 
Sent from my Emacs

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Ted Zlatanov]

On Thu, 03 Mar 2011 09:11:18 +0100 Tassilo Horn <tassilo@member.fsf.org> wrote: 

TH> I've now combined both approaches.  A condition-case handler simply
TH> deletes the big number and `read's again.  That fixes the issue for me.
TH> I added a FIXME and comment, too.  Maybe Lars has some better idea how
TH> to deal with that.

Cool, thanks.

TH> Since that number occurs as value of MODSEQ (which Gnus doesn't use), is
TH> the HIGHESTMODSEQ (which Gnus reads and stores as number) also a
TH> candidate for overflowing?  It probably is, so it might be better to
TH> handle it as string just like the UIDVALIDITY.

I agree and would suggest at this point a generic Gnus approach to
reading large numbers rather than one-off hacks.

Ted

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Lars Magne Ingebrigtsen]

Tassilo Horn <tassilo@member.fsf.org> writes:

> But that huge number is not used by gnus, anyway.  It is `read' but only
> the UID and FLAGS values are saved.  So IMHO, the proper fix is to put a
> condition-case around the `read' that picks the UID and FLAGS with
> string-matching if an overflow-error occurs.

The MODSEQ number is used when doing QRESYNC...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Lars Magne Ingebrigtsen]

Tassilo Horn <tassilo@member.fsf.org> writes:

> * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))

I think the best way to handle this is to make the MODSEQ into a string
if it's more than mumble number of characters long.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Ted Zlatanov]

On Sat, 05 Mar 2011 11:55:05 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Tassilo Horn <tassilo@member.fsf.org> writes:
>> * 2971 FETCH (FLAGS (%Recent) UID 12509 MODSEQ (13419098521433281274))

LMI> I think the best way to handle this is to make the MODSEQ into a string
LMI> if it's more than mumble number of characters long.

...so basically replace (\d+) with "\1" if that's bigger than MAXINT?

Ted

Re: Debugger entered--Lisp error: (overflow-error "13419098521433281274")

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> I think the best way to handle this is to make the MODSEQ into a string
> LMI> if it's more than mumble number of characters long.
>
> ...so basically replace (\d+) with "\1" if that's bigger than MAXINT?

I've now had a peek at the code, and we basically don't use MODSEQ from
FETCH at all.  We use HIGHESTMODSEQ from SELECT, though, which has the
same problems.

So this is what I think should be done:

1) The parser should just remove "\\bMODSEQ ([0-9])" totally before
parsing FETCH

2) HIGHESTMODSEQ should be read as a string

3) The thing that compares old to new HIGHESTMODSEQ should take both
numbers and strings, but compare things as strings

I'll make those changes now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen