Serious IMAP problems because Gnus doesn't behave RFC compliant

Serious IMAP problems because Gnus doesn't behave RFC compliant

[Elias Oltmanns]

Hi all,

it would appear that Gnus currently by design cannot possibly comply
with RFC 3501. The problem is that according to RFC 3501 a UID may be
any 32bit unsigned integer, whereas ELisp only supports 28bit signed
integers---on my system anyway.

This has the rather unpleasant consequence that I can't even open one of
my IMAP mailboxes in Gnus since messages appear to have negative UIDs
just because of an integer overflow. So, the question is: What can be
done about it?

Regards,

Elias

Re: Serious IMAP problems because Gnus doesn't behave RFC compliant

[Simon Josefsson]

Elias Oltmanns <eo@nebensachen.de> writes:

> Hi all,
>
> it would appear that Gnus currently by design cannot possibly comply
> with RFC 3501. The problem is that according to RFC 3501 a UID may be
> any 32bit unsigned integer, whereas ELisp only supports 28bit signed
> integers---on my system anyway.

Wasn't the bit size increased recently?  But still, on 32-bit systems it
still isn't the full 32 bits.  On 64-bit systems, though, I believe
elisp integers can hold close to 64-bits.

> This has the rather unpleasant consequence that I can't even open one of
> my IMAP mailboxes in Gnus since messages appear to have negative UIDs
> just because of an integer overflow. So, the question is: What can be
> done about it?

Implement bignum in elisp, translate UIDs to locally maintained lower
article numbers (nn)imap.el (messy!), run a translating IMAP-proxy, or
buy a 64-bit machine. :)

If all UIDs in the mailbox is high, you might solve this by substracting
a fixed value from all UIDs in that mailbox.  I think even this will
also be quite messy to implement.

I think XEmacs supports bignum nowadays?

/Simon

Re: Serious IMAP problems because Gnus doesn't behave RFC compliant

[Greg Troxel]

Simon Josefsson <simon@josefsson.org> writes:

> Elias Oltmanns <eo@nebensachen.de> writes:
>
>> Hi all,
>>
>> it would appear that Gnus currently by design cannot possibly comply
>> with RFC 3501. The problem is that according to RFC 3501 a UID may be
>> any 32bit unsigned integer, whereas ELisp only supports 28bit signed
>> integers---on my system anyway.
>
> Wasn't the bit size increased recently?  But still, on 32-bit systems it
> still isn't the full 32 bits.  On 64-bit systems, though, I believe
> elisp integers can hold close to 64-bits.

The notion that correct behavior depends on the machine architecture is
unsettling, particularly in a lisp-like language.  As I'm sure you know,
Scheme simply deals with numbers of arbitrary size.

I find it surprising that emacs doesn't have a 'int64_t' type, even if
it's only 62.

>> This has the rather unpleasant consequence that I can't even open one of
>> my IMAP mailboxes in Gnus since messages appear to have negative UIDs
>> just because of an integer overflow. So, the question is: What can be
>> done about it?
>
> Implement bignum in elisp,

> translate UIDs to locally maintained lower article numbers (nn)imap.el
> (messy!),

but maybe most feasible

> run a translating IMAP-proxy,

bogus - that leaves gnus not standards-compliant.

> or buy a 64-bit machine. :)

really bogus!

> I think XEmacs supports bignum nowadays?

then maybe that can be ported.

Re: Serious IMAP problems because Gnus doesn't behave RFC compliant

[Reiner Steib]

On Wed, Oct 03 2007, Simon Josefsson wrote:

> Elias Oltmanns <eo@nebensachen.de> writes:
>> it would appear that Gnus currently by design cannot possibly comply
>> with RFC 3501. The problem is that according to RFC 3501 a UID may be
>> any 32bit unsigned integer, whereas ELisp only supports 28bit signed
>> integers---on my system anyway.
>
> Wasn't the bit size increased recently?

It has been increased in Emacs 22 (compared to Emacs 21).  Below[1]
are some data from January 2007.

> But still, on 32-bit systems it still isn't the full 32 bits.  On
> 64-bit systems, though, I believe elisp integers can hold close to
> 64-bits.

Maybe the recent thread "maximum buffer size exceeded" on emacs-devel
<http://thread.gmane.org/gmane.emacs.help/47195/focus=77865> could be
interesting.

>> This has the rather unpleasant consequence that I can't even open one of
>> my IMAP mailboxes in Gnus since messages appear to have negative UIDs
>> just because of an integer overflow. 

Could we add a test for this and display a warning?  Additionally we
should document this problem somewhere (in the manual?), I think.

Bye, Reiner.

[1] Current Emacs 22 branch (future Emacs 22.2)

,----
| ELISP> emacs-version
| "22.1.50.7"
| ELISP> most-positive-fixnum
| 268435455
`----

From <news:v9k5zki7br.fsf@marauder.physik.uni-ulm.de>
(<http://www.google.de/groups?as_umsgid=v9k5zki7br.fsf@marauder.physik.uni-ulm.de&hl=en>).

,----
| ELISP> emacs-version
| "21.3.1"
| ELISP> system-configuration
| "x86_64-suse-linux"
| ELISP> (require 'cl)
| cl
| ELISP> most-positive-fixnum
| 576460752303423487
`----
,----
| ELISP> emacs-version
| "22.0.92.1"
| ELISP> system-configuration
| "x86_64-unknown-linux-gnu"
| ELISP> most-positive-fixnum
| 1152921504606846975
`----
,----
| ELISP> emacs-version 
| "21.5  (beta21) \"corn\" (+CVS-20050720) XEmacs Lucid" 
| ELISP> system-configuration 
| "i386-suse-linux" 
| ELISP> most-positive-fixnum 
| 1073741823 
`----
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Serious IMAP problems because Gnus doesn't behave RFC compliant

[Simon Josefsson]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

>>> This has the rather unpleasant consequence that I can't even open one of
>>> my IMAP mailboxes in Gnus since messages appear to have negative UIDs
>>> just because of an integer overflow. 
>
> Could we add a test for this and display a warning?  Additionally we
> should document this problem somewhere (in the manual?), I think.

Sounds like a good idea.  imap.el uses string-to-integer in a few
places, possibly that could be replaced with a imap-string-to-integer
that would give a warning if the conversion would overflow.  I don't
have time to implement or test this though.

/Simon

Re: Serious IMAP problems because Gnus doesn't behave RFC compliant

[Daniel Pittman]

Greg Troxel <gdt@work.lexort.com> writes:
> Simon Josefsson <simon@josefsson.org> writes:
>> Elias Oltmanns <eo@nebensachen.de> writes:
>>
>>> it would appear that Gnus currently by design cannot possibly comply
>>> with RFC 3501. The problem is that according to RFC 3501 a UID may
>>> be any 32bit unsigned integer, whereas ELisp only supports 28bit
>>> signed integers---on my system anyway.
>>
>> Wasn't the bit size increased recently?  But still, on 32-bit systems
>> it still isn't the full 32 bits.  On 64-bit systems, though, I
>> believe elisp integers can hold close to 64-bits.
>
> The notion that correct behavior depends on the machine architecture
> is unsettling, particularly in a lisp-like language.  As I'm sure you
> know, Scheme simply deals with numbers of arbitrary size.
>
> I find it surprising that emacs doesn't have a 'int64_t' type, even if
> it's only 62.

Yah, it is a bit annoying.  The CVS repository is --> that way, please
fix it.  I would be very happy with you. ;)

Regards,
        Daniel
-- 
Daniel Pittman <daniel@cybersource.com.au>           Phone: 03 9621 2377
Level 4, 10 Queen St, Melbourne             Web: http://www.cyber.com.au
Cybersource: Australia's Leading Linux and Open Source Solutions Company

Re: Serious IMAP problems because Gnus doesn't behave RFC compliant

[Ken Raeburn]

On Oct 1, 2007, at 3:50, Elias Oltmanns wrote:
> it would appear that Gnus currently by design cannot possibly comply
> with RFC 3501. The problem is that according to RFC 3501 a UID may be
> any 32bit unsigned integer, whereas ELisp only supports 28bit signed
> integers---on my system anyway.
>
> This has the rather unpleasant consequence that I can't even open  
> one of
> my IMAP mailboxes in Gnus since messages appear to have negative UIDs
> just because of an integer overflow. So, the question is: What can be
> done about it?

Would floating point values give you the precision you need?  They  
use the C type "double" in Emacs.  I don't know if the C standard has  
a requirement for the minimum number of digits in the mantissa, but I  
suspect most representations if not all would support precise  
representation of up to 2**32-1.  You'd probably want to check for  
round-off errors, though, just to be paranoid, unless you can confirm  
that 2**32-1 will always fit.

Ken