ProtonMail Bridge Patch

ProtonMail Bridge Patch

[Jakub Ječmínek]

Hello,
I'm not sure if this is the correct place to post this but I wish there
was this information somewhere like a week ago.

I recently switched my mail provider to ProtonMail which lets you use
IMAP/SMTP protocol only if you run something called ProtonMail Bridge on
your machine. This software basically acts as a local IMAP/SMTP server
and communicates with the upstream servers in a secured fashion (or at
least this is the rationale).

The problem is that Gnus and Gluon - IMAP server embedded inside
ProtonMail Bridge - doesn't work well together. Gnus (nnimap back end)
was acting like a lunatic and it took me a while before I found the real
reason for its behaviour.

Gnus expects that the messages are returned from a FETCH request in a
ascending (predictable?) order but Gluon responds each time
differently. Example communication might look like this:

Gnus: FETCH 1:3 ...
Gluon: * 3 RESPONSE ...
Gluon: * 1 RESPONSE ...
Gluon: * 2 RESPONSE ...

This is apparently a valid response according to RFC 3501. But luckily
for us there's server option inside Gluon which let's you disable
parallelism and ultimately fix message shuffling. Here's the patch for
proton-bridge package:

From 23f9c69a1552af1f946687f55de901488a2c9a38 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jakub=20Je=C4=8Dm=C3=ADnek?= <kuba@kubajecminek.cz>
Date: Fri, 16 Feb 2024 16:07:21 +0100
Subject: [PATCH] Gnus Fix: disable parallelism

---
 internal/services/imapsmtpserver/imap.go | 1 +
 1 file changed, 1 insertion(+)

diff --git a/internal/services/imapsmtpserver/imap.go b/internal/services/imapsmtpserver/imap.go
index 63888b51..358173c6 100644
--- a/internal/services/imapsmtpserver/imap.go
+++ b/internal/services/imapsmtpserver/imap.go
@@ -120,6 +120,7 @@ func newIMAPServer(
 		gluon.WithReporter(reporter),
 		gluon.WithUIDValidityGenerator(uidValidityGenerator),
 		gluon.WithPanicHandler(panicHandler),
+		gluon.WithDisableParallelism(),
 	)
 	if err != nil {
 		return nil, err
-- 
2.42.0

-- 
Kuba Ječmínek
<http://kubajecminek.cz>

Re: ProtonMail Bridge Patch

[Eric Abrahamsen]

Jakub Ječmínek <kuba@kubajecminek.cz> writes:

> Hello,
> I'm not sure if this is the correct place to post this but I wish there
> was this information somewhere like a week ago.
>
> I recently switched my mail provider to ProtonMail which lets you use
> IMAP/SMTP protocol only if you run something called ProtonMail Bridge on
> your machine. This software basically acts as a local IMAP/SMTP server
> and communicates with the upstream servers in a secured fashion (or at
> least this is the rationale).
>
> The problem is that Gnus and Gluon - IMAP server embedded inside
> ProtonMail Bridge - doesn't work well together. Gnus (nnimap back end)
> was acting like a lunatic and it took me a while before I found the real
> reason for its behaviour.
>
> Gnus expects that the messages are returned from a FETCH request in a
> ascending (predictable?) order but Gluon responds each time
> differently. Example communication might look like this:
>
> Gnus: FETCH 1:3 ...
> Gluon: * 3 RESPONSE ...
> Gluon: * 1 RESPONSE ...
> Gluon: * 2 RESPONSE ...

> This is apparently a valid response according to RFC 3501.

Thanks a lot for this note! I'm glad you found a solution, but if
out-of-order FETCH UIDs are valid according to the RFC, Gnus should also
handle that.

I assume the problem is in `nnimap-transform-headers'? Nothing in there
jumps out at me immediately. Or maybe the problem is that the function
doesn't sort the parsed headers, and Gnus' next step expects the headers
to be in sorted order?

Thanks
Eric

Re: ProtonMail Bridge Patch

[Jakub Ječmínek]

"Eric Abrahamsen" <eric@ericabrahamsen.net> writes:

> Thanks a lot for this note! I'm glad you found a solution, but if
> out-of-order FETCH UIDs are valid according to the RFC, Gnus should also
> handle that.
>
> I assume the problem is in `nnimap-transform-headers'? Nothing in there
> jumps out at me immediately. Or maybe the problem is that the function
> doesn't sort the parsed headers, and Gnus' next step expects the headers
> to be in sorted order?

I might be wrong of course but I think that the latter and that there's
good chance sorting the headers would fix this issue.

-- 
Kuba Ječmínek
<http://kubajecminek.cz>

Re: ProtonMail Bridge Patch

[Eric Abrahamsen]

Jakub Ječmínek <kuba@kubajecminek.cz> writes:

> "Eric Abrahamsen" <eric@ericabrahamsen.net> writes:
>
>> Thanks a lot for this note! I'm glad you found a solution, but if
>> out-of-order FETCH UIDs are valid according to the RFC, Gnus should also
>> handle that.
>>
>> I assume the problem is in `nnimap-transform-headers'? Nothing in there
>> jumps out at me immediately. Or maybe the problem is that the function
>> doesn't sort the parsed headers, and Gnus' next step expects the headers
>> to be in sorted order?
>
> I might be wrong of course but I think that the latter and that there's
> good chance sorting the headers would fix this issue.

I still haven't found anything that looks like a hard requirement that
headers be sorted, but that doesn't mean it isn't there.

I also wasn't able to find anything explicit in RFC 3501 or 9051 about
the order of FETCH responses -- for my information, can you point me in
the right direction?

Thanks,
Eric

Re: ProtonMail Bridge Patch

[Jakub Ječmínek]

"Eric Abrahamsen" <eric@ericabrahamsen.net> writes:

>> I might be wrong of course but I think that the latter and that there's
>> good chance sorting the headers would fix this issue.
>
> I still haven't found anything that looks like a hard requirement that
> headers be sorted, but that doesn't mean it isn't there.

That was only my guess (based on very shallow understanding of the Gnus
internals). I can try to implement sorting and see if that helps. This
is what I found inside Gnus manual but I'm not sure if that's relevant
for this issue:

[...] article numbers must be assigned in order of arrival in the group;
this is not necessarily the same as the date of the message.

> I also wasn't able to find anything explicit in RFC 3501 or 9051 about
> the order of FETCH responses -- for my information, can you point me in
> the right direction?

That's the thing. There's nothing explicit in RFC 3501 about the message
order so AFAIK the consensus is that UIDs don't have to be sorted.

-- 
Kuba Ječmínek (http://kubajecminek.cz)

Re: ProtonMail Bridge Patch

[Eric Abrahamsen]

Jakub Ječmínek <kuba@kubajecminek.cz> writes:

> "Eric Abrahamsen" <eric@ericabrahamsen.net> writes:
>
>>> I might be wrong of course but I think that the latter and that there's
>>> good chance sorting the headers would fix this issue.
>>
>> I still haven't found anything that looks like a hard requirement that
>> headers be sorted, but that doesn't mean it isn't there.
>
> That was only my guess (based on very shallow understanding of the Gnus
> internals). I can try to implement sorting and see if that helps. This
> is what I found inside Gnus manual but I'm not sure if that's relevant
> for this issue:
>
> [...] article numbers must be assigned in order of arrival in the group;
> this is not necessarily the same as the date of the message.

Huh. Yeah, I absolutely believe it's true, I just wish I could pinpoint
where.

>> I also wasn't able to find anything explicit in RFC 3501 or 9051 about
>> the order of FETCH responses -- for my information, can you point me in
>> the right direction?
>
> That's the thing. There's nothing explicit in RFC 3501 about the message
> order so AFAIK the consensus is that UIDs don't have to be sorted.

Oh I see. Well that's fair enough, and I don't think there's any
intrinsic reason why Gnus should be imposing its internal restrictions
on external servers. The assumption is just that Gnus article numbers
and IMAP UIDs are the same idea -- monotonically-increasing integer
identifiers that don't get reused within a group -- so we could map them
one-to-one. But obviously it's more complicated than that.

Re: ProtonMail Bridge Patch

[Bjørn Mork]

Jakub Ječmínek <kuba@kubajecminek.cz> writes:
> "Eric Abrahamsen" <eric@ericabrahamsen.net> writes:
>
>> I also wasn't able to find anything explicit in RFC 3501 or 9051 about
>> the order of FETCH responses -- for my information, can you point me in
>> the right direction?
>
> That's the thing. There's nothing explicit in RFC 3501 about the message
> order so AFAIK the consensus is that UIDs don't have to be sorted.

I was curiuos about this too and went looking.  I believe the definition
of "sequence-set" (which is what the ID range in the FETCH request is)
on https://datatracker.ietf.org/doc/html/rfc3501#page-90 suggests that
clients should expect replies in any order:

   Servers MAY coalesce overlaps and/or execute the sequence in any
   order.


Bjørn

Re: ProtonMail Bridge Patch

[Eric Abrahamsen]

Bjørn Mork <bjorn@mork.no> writes:

> Jakub Ječmínek <kuba@kubajecminek.cz> writes:
>> "Eric Abrahamsen" <eric@ericabrahamsen.net> writes:
>>
>>> I also wasn't able to find anything explicit in RFC 3501 or 9051 about
>>> the order of FETCH responses -- for my information, can you point me in
>>> the right direction?
>>
>> That's the thing. There's nothing explicit in RFC 3501 about the message
>> order so AFAIK the consensus is that UIDs don't have to be sorted.
>
> I was curiuos about this too and went looking.  I believe the definition
> of "sequence-set" (which is what the ID range in the FETCH request is)
> on https://datatracker.ietf.org/doc/html/rfc3501#page-90 suggests that
> clients should expect replies in any order:
>
>    Servers MAY coalesce overlaps and/or execute the sequence in any
>    order.

Thanks! I hadn't looked as far as the formal syntax section, that's helpful.

Re: ProtonMail Bridge Patch

[Jakub Ječmínek]

"Eric Abrahamsen" <eric@ericabrahamsen.net> writes:

> Huh. Yeah, I absolutely believe it's true, I just wish I could pinpoint
> where.

Ok, I think I've found it. The problem is that we're feeding unsorted
lists into a bunch of functions that require lists to be sorted (a lot
of functions do in 'gnus-range.el').

**Example 1:**
Inside the body of 'gnus-select-newsgroup' ('gnus-sum.el').

(setq gnus-newsgroup-unreads
	    (gnus-sorted-nintersection
	     gnus-newsgroup-unreads fetched-articles))

...

(gnus-update-missing-marks
  (gnus-sorted-difference articles fetched-articles))


**Example 2:**
Inside the body of 'nnimap-update-info' ('nnimap.el').

(let* ((unread (gnus-compress-sequence
		(gnus-set-difference
		 (gnus-set-difference
		  existing
		  (gnus-sorted-union
		   (cdr (assoc '%Seen flags))
		   (cdr (assoc '%Deleted flags))))
		 (cdr (assoc '%Flagged flags)))))
       (read (range-difference
	      (cons start-article high) unread)))

-- 
Kuba Ječmínek (http://kubajecminek.cz)

Re: ProtonMail Bridge Patch

[Eric Abrahamsen]

Jakub Ječmínek <kuba@kubajecminek.cz> writes:

> "Eric Abrahamsen" <eric@ericabrahamsen.net> writes:
>
>> Huh. Yeah, I absolutely believe it's true, I just wish I could pinpoint
>> where.
>
> Ok, I think I've found it. The problem is that we're feeding unsorted
> lists into a bunch of functions that require lists to be sorted (a lot
> of functions do in 'gnus-range.el').

Yes! That's it, thanks very much. The question remains where exactly to
address this, but it's good to know why it's behaving this way. I'll put
this on the list.

Thanks,
Eric

Re: ProtonMail Bridge Patch

[Jakub Ječmínek]

"Eric Abrahamsen" <eric@ericabrahamsen.net> writes:

> Yes! That's it, thanks very much. The question remains where exactly to
> address this, but it's good to know why it's behaving this way. I'll put
> this on the list.

I don't think we have good/elegant way to do it. I see two options:

1. Implement custom sorting in nnimap.el back end.

This would require substantial work because we'd have to modify all the
definitions where we perform FETCH & SEARCH (AFAIK these two commands
are the only affected).

2. Modify the functions in gnus-range.el to accept unsorted lists

I don't know the internals good enough to tell if this would be
enough. There might be other parts in Gnus which also expect sorted
headers/messages/lists/etc that I'm not aware of.

-- 
Kuba Ječmínek (http://kubajecminek.cz)

Re: ProtonMail Bridge Patch

[Eric Abrahamsen]

Jakub Ječmínek <kuba@kubajecminek.cz> writes:

> "Eric Abrahamsen" <eric@ericabrahamsen.net> writes:
>
>> Yes! That's it, thanks very much. The question remains where exactly to
>> address this, but it's good to know why it's behaving this way. I'll put
>> this on the list.
>
> I don't think we have good/elegant way to do it. I see two options:
>
> 1. Implement custom sorting in nnimap.el back end.
>
> This would require substantial work because we'd have to modify all the
> definitions where we perform FETCH & SEARCH (AFAIK these two commands
> are the only affected).

It's possible that all that would be necessary is to add a routine at
the bottom of `nnimap-transform-headers', to sort the buffer text. The
SEARCH command is handled differently, I believe.

> 2. Modify the functions in gnus-range.el to accept unsorted lists
>
> I don't know the internals good enough to tell if this would be
> enough. There might be other parts in Gnus which also expect sorted
> headers/messages/lists/etc that I'm not aware of.

I don't think modifying gnus-range is a good idea -- if that code
requires sorted ranges, we should just make sure it's always given
sorted ranges.

Re: ProtonMail Bridge Patch

[Jakub Ječmínek]

"Eric Abrahamsen" <eric@ericabrahamsen.net> writes:

> It's possible that all that would be necessary is to add a routine at
> the bottom of `nnimap-transform-headers', to sort the buffer text.

There's also `nnimap-request-group-scan' and probably other functions as
well.

-- 
Kuba Ječmínek (http://kubajecminek.cz)