Problems with attachments in S/MIME encrypted messages from Outlook

Problems with attachments in S/MIME encrypted messages from Outlook

[Dave Goldberg]

At work, we recently moved to an all Microsoft email solution (yuck!)
from the corporate IT perspective.  That is, the mail servers are
Exchange but users are expected to use Outlook.  Those of us who use
other mail/calendar clients are on our own, though they did enable
IMAP and iCal so we're not shut out.  S/MIME is used for signing and
encrypting as it was in our previous environment.  Lately I've been
receiving encrypted messages that contain attachments and Gnus is
unable to deal with them.  I get this error when attempting to save or
view the attachment:

Error while decoding: (error Illegal `=' character while decoding base64 22139)

and the resulting file is corrupt.  That last number is not always the
same; it appears to be the position of the illegal character, though
by my count it's off a bit (more on that below).  Since I couldn't
find that error message in any lisp code, I guessed it was built in to
XEmacs.  So I tried loading the base64 stuff from mail-lib but other
than no error being reported, the results were the same - a corrupt
file.

Evolution and Thunderbird have no trouble with these attachments.
Further, if I run openssl by hand to get the message into a textual
format, I can extract the attachment's base64 block and successfully
convert it so this appears to be an error parsing the MIME structure
of the message and somehow embedding the Illegal '=' character into
the base64 stream.  Unfortunately I do not see any obvious way to
debug this inside of XEmacs.  The messages are encrypted and the
initial decryption results in a single base64 encoded block with
headers like this:

    Content-Type: application/x-pkcs7-mime; name=smime.p7m; smime-type=signed-data
    Content-Transfer-Encoding: base64
    Content-Disposition: attachment; filename=smime.p7m

A subsequent run of openssl to verify the signature is required to get
the message into readable form (well, at least that's how I've been
doing it).  In otherwords, I run

openssl smime -decrypt -recip mykeyfile -in message | \
openssl smime -verify -noverify 

to get a MIME structure like this (text removed and attachment
truncated)

    Content-Type: multipart/mixed;
    boundary="----=_NextPart_000_000F_01C5DAF3.7FAD2950"

    This is a multi-part message in MIME format.

    ------=_NextPart_000_000F_01C5DAF3.7FAD2950
    Content-Type: text/plain;
    charset="us-ascii"
    Content-Transfer-Encoding: 7bit



    some text


    ------=_NextPart_000_000F_01C5DAF3.7FAD2950
    Content-Type: application/vnd.ms-excel;
    name="test.xls"
    Content-Transfer-Encoding: base64
    Content-Disposition: attachment;
    filename="test.xls"

    [lots of stuff elided]
    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==

    ------=_NextPart_000_000F_01C5DAF3.7FAD2950--

Note the = at the end of the last line.  I'm guessing those are the
illegal characters yet they don't seem to cause me any grief when I
process the file manually or, any grief to other clients.  However, I
really don't know whether the message I'm creating from my command
line is the same as that created when Gnus runs openssl since I'm not
certain exactly what command line it runs.  I've attempted to use the
debugger a bit, but I don't know where to break into it and all
attempts have ended up with me spending too much time single stepping,
and then missing the key event when I try to let some functions go.
I'm happy to try to track this down (almost desperate to do so, in
fact :-) but I'm hoping someone can give me some guidance on where to
look.

Thanks,

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: Problems with attachments in S/MIME encrypted messages from Outlook

[Simon Josefsson]

Dave Goldberg <david.goldberg6@verizon.net> writes:

> At work, we recently moved to an all Microsoft email solution (yuck!)
> from the corporate IT perspective.  That is, the mail servers are
> Exchange but users are expected to use Outlook.  Those of us who use
> other mail/calendar clients are on our own, though they did enable
> IMAP and iCal so we're not shut out.  S/MIME is used for signing and
> encrypting as it was in our previous environment.  Lately I've been
> receiving encrypted messages that contain attachments and Gnus is
> unable to deal with them.  I get this error when attempting to save or
> view the attachment:
>
> Error while decoding: (error Illegal `=' character while decoding base64 22139)

Does M-x toggle-debug-on-error RET give any clues to where this
happens?

> and the resulting file is corrupt.  That last number is not always the
> same; it appears to be the position of the illegal character, though
> by my count it's off a bit (more on that below).  Since I couldn't
> find that error message in any lisp code, I guessed it was built in to
> XEmacs.  So I tried loading the base64 stuff from mail-lib but other
> than no error being reported, the results were the same - a corrupt
> file.

The base64 decoder in emacs is a bit picky, I recall.  But there could
be some other problem (CRLF?).

Re: Problems with attachments in S/MIME encrypted messages from Outlook

[Dave Goldberg]

>>>>> On Wed, 22 Mar 2006 10:53:21 +0100, Simon Josefsson <jas@extundo.com> said:

>> Error while decoding: (error Illegal `=' character while decoding base64 22139)

> Does M-x toggle-debug-on-error RET give any clues to where this
> happens?

Unfortunately no.  The decode "succeeds" with the error message just
flashing in the minibuffer and stored in the message log.

-- 
Dave Goldberg
david.goldberg6@verizon.net

'g' not checking nnimap groups

[gdt]

I'm running Gnus from cvs head this morning with emacs 21.3 (on
NetBSD/i386).

I am mostly reading mail, and have an nnimap secondary select method,
but also one newsgroup at news.mit.edu.

I have turned off the agent on my nnimap select method; I found (with
an earlier release) that articles deleted from the imap folders still
appeared locally, and I consider this bad behavior, or at least not
what I want.

The 'g' command in the *Group* buffer behaves inconsistently.
Sometimes it checks all of my nnimap groups, but often it just checks
nnimap+work.lexort.com:lists.ding   This is the most recently added,
but not the last in the *Group* buffer.

In *imap-log*, I get this on hitting g:

539 UID SEARCH UNSEEN UNDELETED
* SEARCH
539 OK Search completed.
540 UID SEARCH SEEN
* SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
540 OK Search completed.
541 UID SEARCH FLAGGED
* SEARCH 1 2 41
541 OK Search completed.
542 UID SEARCH ANSWERED
* SEARCH 4
542 OK Search completed.
543 UID SEARCH RECENT
* SEARCH
543 OK Search completed.
544 FETCH 1,* UID
* 1 FETCH (UID 1)
* 84 FETCH (UID 84)
544 OK Fetch completed.
545 STATUS "lists.ding" (UNSEEN)
* STATUS "lists.ding" (UNSEEN 0)
545 OK Status completed.
546 EXPUNGE
546 OK Expunge completed.

This indicates to me that the imap connection was already selected,
and that only that one group was checked.

If I do ^U 1 g, I get the following:

555 STATUS "Inbox" (uidvalidity uidnext unseen)
* STATUS "Inbox" (UIDNEXT 351 UIDVALIDITY 1141579755 UNSEEN 0)
555 OK Status completed.

and ^U 2 g gets me:

556 STATUS "misc.gdt-cvs" (uidvalidity uidnext unseen)
557 STATUS "family.foobar" (uidvalidity uidnext unseen)
558 STATUS "Inbox" (uidvalidity uidnext unseen)
* STATUS "misc.gdt-cvs" (UIDNEXT 5 UIDVALIDITY 1142461155 UNSEEN 0)
556 OK Status completed.
* STATUS "family.foobar" (UIDNEXT 2 UIDVALIDITY 1141827703 UNSEEN 0)
557 OK Status completed.
* STATUS "Inbox" (UIDNEXT 351 UIDVALIDITY 1141579755 UNSEEN 0)
558 OK Status completed.


Sometimes, g checks all my nnimap groups, but usually it doesn't.

Questions arising:

  Am I correct in thinking g should query all groups?

  Is there some hairy IMAP register-for-notifications scheme in use,
  and gone wrong?


-- 
	Greg Troxel <gdt@work.lexort.com>

Re: 'g' not checking nnimap groups

[Simon Josefsson]

gdt@work.lexort.com writes:

> I'm running Gnus from cvs head this morning with emacs 21.3 (on
> NetBSD/i386).
>
> I am mostly reading mail, and have an nnimap secondary select method,
> but also one newsgroup at news.mit.edu.
>
> I have turned off the agent on my nnimap select method; I found (with
> an earlier release) that articles deleted from the imap folders still
> appeared locally, and I consider this bad behavior, or at least not
> what I want.
>
> The 'g' command in the *Group* buffer behaves inconsistently.
> Sometimes it checks all of my nnimap groups, but often it just checks
> nnimap+work.lexort.com:lists.ding   This is the most recently added,
> but not the last in the *Group* buffer.

Maybe there is some server confusion.  When this occur, do you have
multiple copies of your nnimap server in the server buffer?  (^ from
the group buffer.)

> In *imap-log*, I get this on hitting g:
>
> 539 UID SEARCH UNSEEN UNDELETED
> * SEARCH
> 539 OK Search completed.
> 540 UID SEARCH SEEN
> * SEARCH 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84
> 540 OK Search completed.
> 541 UID SEARCH FLAGGED
> * SEARCH 1 2 41
> 541 OK Search completed.
> 542 UID SEARCH ANSWERED
> * SEARCH 4
> 542 OK Search completed.
> 543 UID SEARCH RECENT
> * SEARCH
> 543 OK Search completed.
> 544 FETCH 1,* UID
> * 1 FETCH (UID 1)
> * 84 FETCH (UID 84)
> 544 OK Fetch completed.
> 545 STATUS "lists.ding" (UNSEEN)
> * STATUS "lists.ding" (UNSEEN 0)
> 545 OK Status completed.
> 546 EXPUNGE
> 546 OK Expunge completed.
>
> This indicates to me that the imap connection was already selected,
> and that only that one group was checked.

Right, although the above is the "slow" mail check.  If the server
definitions are in order, it should only use STATUS.

> If I do ^U 1 g, I get the following:
>
> 555 STATUS "Inbox" (uidvalidity uidnext unseen)
> * STATUS "Inbox" (UIDNEXT 351 UIDVALIDITY 1141579755 UNSEEN 0)
> 555 OK Status completed.

The uidvalidity look potentially too large for elisp integers,
although I don't think large uidvalidity should be a problem.

> and ^U 2 g gets me:
>
> 556 STATUS "misc.gdt-cvs" (uidvalidity uidnext unseen)
> 557 STATUS "family.foobar" (uidvalidity uidnext unseen)
> 558 STATUS "Inbox" (uidvalidity uidnext unseen)
> * STATUS "misc.gdt-cvs" (UIDNEXT 5 UIDVALIDITY 1142461155 UNSEEN 0)
> 556 OK Status completed.
> * STATUS "family.foobar" (UIDNEXT 2 UIDVALIDITY 1141827703 UNSEEN 0)
> 557 OK Status completed.
> * STATUS "Inbox" (UIDNEXT 351 UIDVALIDITY 1141579755 UNSEEN 0)
> 558 OK Status completed.
>
> Sometimes, g checks all my nnimap groups, but usually it doesn't.

Weird.

> Questions arising:
>
>   Am I correct in thinking g should query all groups?

Yes.

>   Is there some hairy IMAP register-for-notifications scheme in use,
>   and gone wrong?

No.  Nnimap should invoke a STATUS command on every group that is
passed to the nnimap backend.  Maybe you can debug things and figure
out why Gnus doesn't hand a list of all groups to nnimap, or why
nnimap doesn't check all the groups in the list.

Re: 'g' not checking nnimap groups

[gdt]

  [inconsistent nnimap checking on 'g' - slow check, only checking one
  group]

I have found part of the problem.  I had subscribed to
nnimap+myserver:Inbox with U, because that's how thunderbird rendered
Inbox which is magic with dovecot.  But, when I use the server buffer
and browse, I find it's INBOX.  So I killed Inbox and subscribed to
INBOX.  I'm still getting the slow check, but it's checking
consistently.

Perhaps there needs to be some canonicalization funtion, assuming IMAP
is case insenstive.  Or, dovecot should be less permissive.

Probably there is some error with INBOX/Inbox that isn't handled
correctly.

I'll report back to the list if I figure anything else out.

-- 
	Greg Troxel <gdt@work.lexort.com>

Re: 'g' not checking nnimap groups

[Simon Josefsson]

gdt@work.lexort.com writes:

>   [inconsistent nnimap checking on 'g' - slow check, only checking one
>   group]
>
> I have found part of the problem.  I had subscribed to
> nnimap+myserver:Inbox with U, because that's how thunderbird rendered
> Inbox which is magic with dovecot.  But, when I use the server buffer
> and browse, I find it's INBOX.  So I killed Inbox and subscribed to
> INBOX.  I'm still getting the slow check, but it's checking
> consistently.
>
> Perhaps there needs to be some canonicalization funtion, assuming IMAP
> is case insenstive.  Or, dovecot should be less permissive.
>
> Probably there is some error with INBOX/Inbox that isn't handled
> correctly.

Yes, that's probably true.  INBOX in IMAP is special, since it is
case-insensitive, other mailboxes are case-sensitive.  Gnus is
case-sensitive.  It might have been that nnimap doesn't map some value
it receive from the server (for "INBOX") to the Gnus "Inbox" group
correctly.

Still, you shouldn't have the slow checks.  What does 'G e' on one of
your groups say?  It should be "nnimap:foo" where "foo" is your nnimap
server name.  If it is (nnimap "foo"), change it to "nnimap:foo".

Re: 'g' not checking nnimap groups

[gdt]

I have in .emacs

(setq gnus-secondary-select-methods
      '((nnimap "my.logical.domain"
		(nnimap-address "my.server.edu")
		(nnimap-port 993)
		(nnimap-stream ssl)
		(nnimap-authenticator login)
		)))

and G e showed (nnimap "my.logical.domain")

imap-log when doing g gets lots of blocks like

3171 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft gnus-expire)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft gnus-expire \*)] Flags permitted.
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1141579755] UIDs valid
* OK [UIDNEXT 414] Predicted next UID
3171 OK [READ-WRITE] Select completed.
3172 UID SEARCH UNSEEN UNDELETED
* SEARCH
3172 OK Search completed.
3173 UID SEARCH SEEN
* SEARCH 377 386 392 413
3173 OK Search completed.
3174 UID SEARCH FLAGGED
* SEARCH 377 386
3174 OK Search completed.
3175 UID SEARCH ANSWERED
* SEARCH 392
3175 OK Search completed.
3176 UID SEARCH KEYWORD gnus-expire
* SEARCH
3176 OK Search completed.
3177 UID SEARCH RECENT
* SEARCH
3177 OK Search completed.
3178 FETCH 1,* UID
* 1 FETCH (UID 377)
* 4 FETCH (UID 413)
3178 OK Fetch completed.
3179 STATUS "INBOX" (UNSEEN)
* STATUS "INBOX" (UNSEEN 0)
3179 OK Status completed.
3180 EXPUNGE
3181 SELECT "family.spot"
3180 OK Expunge completed.
 

I had a number of groups with bad method, and after hunting them down
from *imap-log* I now get just STATUS queries.


So I guess there's a bug where (nnimap "foo") does a slow status
check.  And, it seems that if there is such a group sometimes the
other groups on the server do not get checked.

Thanks very much for the advice.

-- 
	Greg Troxel <gdt@work.lexort.com>

Re: 'g' not checking nnimap groups

[Simon Josefsson]

gdt@work.lexort.com writes:

> I have in .emacs
>
> (setq gnus-secondary-select-methods
>       '((nnimap "my.logical.domain"
> 		(nnimap-address "my.server.edu")
> 		(nnimap-port 993)
> 		(nnimap-stream ssl)
> 		(nnimap-authenticator login)
> 		)))
>
> and G e showed (nnimap "my.logical.domain")

That is the problem then.

> imap-log when doing g gets lots of blocks like
...
> I had a number of groups with bad method, and after hunting them down
> from *imap-log* I now get just STATUS queries.

Yup.

> So I guess there's a bug where (nnimap "foo") does a slow status
> check.  And, it seems that if there is such a group sometimes the
> other groups on the server do not get checked.

Actually, the bug is how (nnimap "foo") ever got into your setup.

The server (nnimap "foo") isn't the same as (nnimap "foo" (nnimap-port
...) ...), and Gnus will treat them as different servers.
Consequently, (nnimap "foo") is treated as an ephemeral server, and
checked using the slow mechanism (I forgot the reason as to why, but
it is probably because it is simpler to optimize the primary/secondary
case).

Re: 'g' not checking nnimap groups

[gdt]

Simon Josefsson <jas@extundo.com> writes:

> Actually, the bug is how (nnimap "foo") ever got into your setup.

I agree that's a bug.  I think I did U and then subscribed to a group,
before I realized the preferred way was to use server mode and browse.

> The server (nnimap "foo") isn't the same as (nnimap "foo" (nnimap-port
> ...) ...), and Gnus will treat them as different servers.
> Consequently, (nnimap "foo") is treated as an ephemeral server, and
> checked using the slow mechanism (I forgot the reason as to why, but
> it is probably because it is simpler to optimize the primary/secondary
> case).

But, gnus used the same imap server connection.  The "foo" in the
wrong group spec isn't the same as the IMAP server, so it must have
used the nnimap select method in gnus-secondary-select-methods to find
it.

On my list of things to look into if I find spare time:

  add path from UNSEEN count from STATUS to count of new articles in
  summmary

  investigate nnimap/agent cache coherency, ensuring articles
  deleted/moved by other clients are removed from the agent.

-- 
	Greg Troxel <gdt@work.lexort.com>

Re: 'g' not checking nnimap groups

[Simon Josefsson]

gdt@work.lexort.com writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Actually, the bug is how (nnimap "foo") ever got into your setup.
>
> I agree that's a bug.  I think I did U and then subscribed to a group,
> before I realized the preferred way was to use server mode and browse.

I've tried that, and I can't seem to reproduce it.  But others have
said similar things.  Maybe it only happens sometimes.

>> The server (nnimap "foo") isn't the same as (nnimap "foo" (nnimap-port
>> ...) ...), and Gnus will treat them as different servers.
>> Consequently, (nnimap "foo") is treated as an ephemeral server, and
>> checked using the slow mechanism (I forgot the reason as to why, but
>> it is probably because it is simpler to optimize the primary/secondary
>> case).
>
> But, gnus used the same imap server connection.  The "foo" in the
> wrong group spec isn't the same as the IMAP server, so it must have
> used the nnimap select method in gnus-secondary-select-methods to find
> it.

Actually, I think Gnus doesn't, but nnimap then locate the same server
connection using the server name only.

But it was a long time since I looked at this in detail, so I could be
wrong, and the only way to find out is to play with edebug and see
what happens.

> On my list of things to look into if I find spare time:
>
>   add path from UNSEEN count from STATUS to count of new articles in
>   summmary
>
>   investigate nnimap/agent cache coherency, ensuring articles
>   deleted/moved by other clients are removed from the agent.

Fixing those would be very good, many people have problems related to
similar issues.  I don't think fixing them is that complicated.