Re: Does Gnus read and sync the read with IMAP?

Re: Does Gnus read and sync the read with IMAP?

[Bob Newell]

> I noticed that when I read a mail on my phone the read statusis still unread when I fetch my mails from my imap server is still unread in
> Gnus.
> How can I make Gnus to fetch the read status from Imap?

This could depend on your email client on your phone.  At what
point does it sync back to the IMAP server to say that you've
read the mail?  When the client closes?  At certain intervals?
You might try closing the client and see if that helps.

I see the opposite effect.  I access my gmail directly from
gnus with IMAP.  When I read a mail on gnus it isn't
immediately marked as read on gmail, and appears in the web
interface or on my phone client as unopened sometimes, until I
close down gnus.

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Does Gnus read and sync the read with IMAP?

[Björn Bidar]

Bob Newell <bobnewell@bobnewell.net> writes:

>> I noticed that when I read a mail on my phone the read statusis
>> still unread when I fetch my mails from my imap server is still
>> unread in
>> Gnus.
>> How can I make Gnus to fetch the read status from Imap?
>
> This could depend on your email client on your phone.  At what
> point does it sync back to the IMAP server to say that you've
> read the mail?  When the client closes?  At certain intervals?
> You might try closing the client and see if that helps.

The mails that were marked as new in Gnus were in the cur folder of the
imap folder there were from so yes my email client marked them as read.

I closed the client in between I think. I will look how Gnus marks them
as read on my imap server.

Re: Does Gnus read and sync the read with IMAP?

[Bob Newell]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

On Thu, Oct 19, 2023 at 12:14 PM Björn Bidar <bjorn.bidar@thaodan.de> wrote:

> Bob Newell <bobnewell@bobnewell.net> writes:
>
> >> I noticed that when I read a mail on my phone the read statusis
> >> still unread when I fetch my mails from my imap server is still
> >> unread in
> >> Gnus.
> >> How can I make Gnus to fetch the read status from Imap?
> >
> > This could depend on your email client on your phone.  At what
> > point does it sync back to the IMAP server to say that you've
> > read the mail?  When the client closes?  At certain intervals?
> > You might try closing the client and see if that helps.
>
> The mails that were marked as new in Gnus were in the cur folder of the
> imap folder there were from so yes my email client marked them as read.
>

The thing is, your /client/ may show them as read but may or may not have
uploaded that "being read" information to the IMAP server.  Do try
deliberately closing the client on your phone and see if it helps.  You
have to be sure (at least on Android) to actually exit the client and not
just switch to another app.

I closed the client in between I think. I will look how Gnus marks them
> as read on my imap server.
>


-- 
Bob Newell
Honolulu, Hawai`i

[-- Attachment #2: Type: text/html, Size: 2233 bytes --]

Re: Does Gnus read and sync the read with IMAP?

[Michael Heerdegen]

Bob Newell <bobnewell@bobnewell.net> writes:

> The thing is, your /client/ may show them as read but may or may not
> have uploaded that "being read" information to the IMAP server.  Do
> try deliberately closing the client on your phone and see if it helps.
> You have to be sure (at least on Android) to actually exit the client
> and not just switch to another app.

Does Gnus update the "being read" information from the server also for
messages it already knows (i.e., has seen in a prior session)?

Michael.

Re: Does Gnus read and sync the read with IMAP?

[Bob Newell]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Bob Newell <bobnewell@bobnewell.net> writes:
>
>> The thing is, your /client/ may show them as read but may or may not
>> have uploaded that "being read" information to the IMAP server.  Do
>> try deliberately closing the client on your phone and see if it helps.
>> You have to be sure (at least on Android) to actually exit the client
>> and not just switch to another app.
>
> Does Gnus update the "being read" information from the server also for
> messages it already knows (i.e., has seen in a prior session)?

This goes well beyond my very limited knowledge so I did an
experiment.  I went to the gmail web interface and found an
already-opened message in the All Mail folder (really a label)
that I knew for certain I had originally opened in gnus, so
therefore gnus would have known about the message.  I marked
it unread (on the web) and moved it back to INBOX.

Then I started up gnus and gnus saw the message as an unread
INBOX message.  Based on this it appears the answer is yes,
gnus did update the status of this previously seen email.
However I don't know how much if any status information gnus
actually keeps (and I don't use the registry) so this may or
may not be meaningless.  It certainly seems as if, when
reopening the group subsequent to shutting down gnus at some
point, gnus gets a fresh set of information.

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Does Gnus read and sync the read with IMAP?

[Michael Heerdegen]

Bob Newell <bobnewell@bobnewell.net> writes:

> This goes well beyond my very limited knowledge so I did an
> experiment.  I went to the gmail web interface and found an
> already-opened message in the All Mail folder (really a label)
> that I knew for certain I had originally opened in gnus, so
> therefore gnus would have known about the message.  I marked
> it unread (on the web) and moved it back to INBOX.
>
> Then I started up gnus and gnus saw the message as an unread
> INBOX message.  Based on this it appears the answer is yes,
> gnus did update the status of this previously seen email.
> However I don't know how much if any status information gnus
> actually keeps (and I don't use the registry) so this may or
> may not be meaningless.  It certainly seems as if, when
> reopening the group subsequent to shutting down gnus at some
> point, gnus gets a fresh set of information.

Thanks for doing this experiment.  I hoped for that result.

AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR
".newsrc.el" in older Emacs versions).  And indeed, among other things
there seems to be information about "seen" status of messages in that
file, as well as saved status, ticked and dormant status, all of that
Gnus specific stuff, but nothing related to the "read" (or "old")
status.  So I guess the "read" status is fetched every time when you
open the group, along with the rest (list of existing messages etc).


Thx,

Michael.

Re: Does Gnus read and sync the read with IMAP?

[Björn Bidar]

Bob Newell <bobnewell@bobnewell.net> writes:

> On Thu, Oct 19, 2023 at 12:14 PM Björn Bidar <bjorn.bidar@thaodan.de>
> wrote:
>
>> Bob Newell <bobnewell@bobnewell.net> writes:
>>
>> >> I noticed that when I read a mail on my phone the read statusis
>> >> still unread when I fetch my mails from my imap server is still
>> >> unread in
>> >> Gnus.
>> >> How can I make Gnus to fetch the read status from Imap?
>> >
>> > This could depend on your email client on your phone.  At what
>> > point does it sync back to the IMAP server to say that you've
>> > read the mail?  When the client closes?  At certain intervals?
>> > You might try closing the client and see if that helps.
>>
>> The mails that were marked as new in Gnus were in the cur folder of
>> the
>> imap folder there were from so yes my email client marked them as
>> read.
>>
>
> The thing is, your /client/ may show them as read but may or may not
> have
> uploaded that "being read" information to the IMAP server.  Do try
> deliberately closing the client on your phone and see if it helps.
> You
> have to be sure (at least on Android) to actually exit the client and
> not
> just switch to another app.
>
> I closed the client in between I think. I will look how Gnus marks
> them

I have access to the imap server. If the mail in the maildir is no longer in the new
directory but in the cur directory then the client has send the read
status to the server.

In my case (Sailfish OS mail) updates the read status after the mail has
been read.

>> as read on my imap server.
>>

Re: Does Gnus read and sync the read with IMAP?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Bob Newell <bobnewell@bobnewell.net> writes:
>
>> This goes well beyond my very limited knowledge so I did an
>> experiment.  I went to the gmail web interface and found an
>> already-opened message in the All Mail folder (really a label)
>> that I knew for certain I had originally opened in gnus, so
>> therefore gnus would have known about the message.  I marked
>> it unread (on the web) and moved it back to INBOX.
>>
>> Then I started up gnus and gnus saw the message as an unread
>> INBOX message.  Based on this it appears the answer is yes,
>> gnus did update the status of this previously seen email.
>> However I don't know how much if any status information gnus
>> actually keeps (and I don't use the registry) so this may or
>> may not be meaningless.  It certainly seems as if, when
>> reopening the group subsequent to shutting down gnus at some
>> point, gnus gets a fresh set of information.
>
> Thanks for doing this experiment.  I hoped for that result.
>
> AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR
> ".newsrc.el" in older Emacs versions).  And indeed, among other things
> there seems to be information about "seen" status of messages in that
> file, as well as saved status, ticked and dormant status, all of that
> Gnus specific stuff, but nothing related to the "read" (or "old")
> status.  So I guess the "read" status is fetched every time when you
> open the group, along with the rest (list of existing messages etc).

Here's the mapping between Gnus' marks and IMAP flags, I believe things
like "gnus-save" are set as user flags on servers that support that:

(defvar nnimap-mark-alist
  '((read "\\Seen" %Seen)
    (tick "\\Flagged" %Flagged)
    (reply "\\Answered" %Answered)
    (expire "gnus-expire")
    (dormant "gnus-dormant")
    (score "gnus-score")
    (save "gnus-save")
    (download "gnus-download")
    (forward "gnus-forward")))

Re: Does Gnus read and sync the read with IMAP?

[Bob Newell]

> (defvar nnimap-mark-alist
>   '((read "\\Seen" %Seen)
>     (tick "\\Flagged" %Flagged)
>     (reply "\\Answered" %Answered)
>     (expire "gnus-expire")
>     (dormant "gnus-dormant")
>     (score "gnus-score")
>     (save "gnus-save")
>     (download "gnus-download")
>     (forward "gnus-forward")))

Eric, thanks; this is very helpful.  And it leaves a
question/comment or maybe a multi-part one :)

1. Yes as you say not all IMAP servers support all the
gnus-ish additions.

2. When an already read/seen article becomes unread again, as
in the case I tried out, it appears gnus recognizes the
change.

3. But here's the big one, and one that I thought I understood
but seem to get inconsistent results.  At what point does gnus
update the IMAP server?  In other words, I read an article.
In the summary buffer that article definitely is now shown as
read.  But that doesn't propagate instantly to the IMAP
server.  I can see this by (say) opening a gmail web interface
and noting that the article I just read on gnus is still
unread on gmail web.  (I will try a few test cases on my own
to see what I can learn.)

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Does Gnus read and sync the read with IMAP?

[Bob Newell]

> 3. But here's the big one, and one that I thought I understood
> but seem to get inconsistent results.  At what point does gnus
> update the IMAP server?  In other words, I read an article.
> In the summary buffer that article definitely is now shown as
> read.  But that doesn't propagate instantly to the IMAP
> server.  I can see this by (say) opening a gmail web interface
> and noting that the article I just read on gnus is still
> unread on gmail web.  (I will try a few test cases on my own
> to see what I can learn.)

Appendix: At least with gmail, the IMAP server appears to be
pretty consistently updated with 'read' status from gnus when
leaving the group.  So (say) q-uitting the INBOX group updates
the INBOX status on the server, as evidenced by the web
interface.

But there are still some inconsistencies with the All Mail
folder.  To wit, if I read an email in gnus INBOX, and then archive
it to the All Mail folder (from gnus) /before/ I exit INBOX,
then the mail remains 'unread' in All Mail on the web
interface.  Furthermore if I reload All Mail on gnus, that
same email shows up now as unread.

Although this is getting pretty gmail specific, it makes some
sense.  To gmail, INBOX and All Mail are simply labels and an
email is not actually moved from one to the other; /all/ mail
/always/ exists in All Mail and 'moving' from INBOX simply
means removing the INBOX label.  (The exception is Trash and
Spam.)

But gnus treats these as actual groups (call them folders if
you wish).  Now, when the email is moved out of INBOX to All
Mail, and you exit INBOX, I'm guessing that the All Mail
'read' status isn't updated because gnus views that as a truly
separate group.  It would have updated INBOX but the email is
already gone.  Gnus doesn't know, hey this is gmail, they do
things the gmail way.

Similarly the email in question has yet to be seen in the gnus
All Mail group, so when that group is entered/refreshed, the
email comes up as a new unseen (unread) item.

This points to an issue with gnus and gmail, namely that gnus
follows a traditional paradigm which doesn't reflect how gmail
actually works.  I do not view this as a big issue, however,
and fixing it would require, I think, massive rework well
beyond the limited added value.

I think things are a lot simpler with traditional email which
has true IMAP folders.  But as a practical matter, if a few
emails are shown as unread, whether in gnus or another client,
it isn't a show-stopper.

-- 
Bob Newell
Honolulu, Hawai`i

- Via GNU/Linux/Emacs/Gnus/BBDB

Re: Does Gnus read and sync the read with IMAP?

[Adam Sjøgren]

Bob writes:

> At what point does gnus update the IMAP server? In other words, I
> read an article. In the summary buffer that article definitely is
> now shown as read.

Gnus updates the IMAP server when you exit the summary buffer (q).

To find out exactly what happens when, you can set
nnimap-record-commands in your nnimap server configuration, then the
commands will be logged in a buffer called "*imap log*".


  Best regards,

    Adam

-- 
 "None of them kicks go boom                                Adam Sjøgren
  None of them basslines fill the room"                asjo@koldfront.dk

Re: Does Gnus read and sync the read with IMAP?

[Björn Bidar]

Bob Newell <bobnewell@bobnewell.net> writes:

>> 3. But here's the big one, and one that I thought I understood
>> but seem to get inconsistent results.  At what point does gnus
>> update the IMAP server?  In other words, I read an article.
>> In the summary buffer that article definitely is now shown as
>> read.  But that doesn't propagate instantly to the IMAP
>> server.  I can see this by (say) opening a gmail web interface
>> and noting that the article I just read on gnus is still
>> unread on gmail web.  (I will try a few test cases on my own
>> to see what I can learn.)

>
> Although this is getting pretty gmail specific, it makes some
> sense.  To gmail, INBOX and All Mail are simply labels and an
> email is not actually moved from one to the other; /all/ mail
> /always/ exists in All Mail and 'moving' from INBOX simply
> means removing the INBOX label.  (The exception is Trash and
> Spam.)
>

These folders also exist in other imap servers such as dovecot.
I'm not sure if they are standard but dovecot creates them by
grouping all folders with for example the seen flag.


> But gnus treats these as actual groups (call them folders if
> you wish).  Now, when the email is moved out of INBOX to All
> Mail, and you exit INBOX, I'm guessing that the All Mail
> 'read' status isn't updated because gnus views that as a truly
> separate group.

I know GNUS can create virtual groups but I don't think
that's transferred of to these Groups coming from Imap folders.

https://doc.dovecot.org/configuration_manual/namespace/#core_setting-namespace/mailbox/special_use

Based upon RFC 6154.

> I think things are a lot simpler with traditional email which
> has true IMAP folders.  But as a practical matter, if a few
> emails are shown as unread, whether in gnus or another client,
> it isn't a show-stopper.

It's not about inconsistency is an issue I think.

Re: Does Gnus read and sync the read with IMAP?

[James Thomas]

Michael Heerdegen wrote:

> AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR
> ".newsrc.el" in older Emacs versions).  And indeed, among other things
> there seems to be information about "seen" status of messages in that
> file, as well as saved status, ticked and dormant status, all of that
> Gnus specific stuff, but nothing related to the "read" (or "old")
> status.  So I guess the "read" status is fetched every time when you
> open the group, along with the rest (list of existing messages etc).

Just FYI: For those who use the agent, (info "(gnus) Agent and flags")
is also relevant in this regard.

--

Re: Does Gnus read and sync the read with IMAP?

[Adam Sjøgren]

Michael writes:

> AFAIU Gnus stores such stuff in a file named ".newsrc.eld" (AFAIR
> ".newsrc.el" in older Emacs versions).

Looks like the 'd' has been there at least since 1997; gnus-start.el:

    eec82323c2e3 (Lars Ingebrigtsen 1997-04-16 2768)      (setq buffer-file-name
    eec82323c2e3 (Lars Ingebrigtsen 1997-04-16 2769)            (concat gnus-current-startup-file ".eld"))

and before that; gnus.el:

    aa088210dd (Lars Magne Ingebrigtsen 1997-03-04 01:26:40 +0000 9645)        (write-region 1 (point-max) 
    aa088210dd (Lars Magne Ingebrigtsen 1997-03-04 01:26:40 +0000 9646)                      (concat gnus-current-startup-file ".eld") 
    aa088210dd (Lars Magne Ingebrigtsen 1997-03-04 01:26:40 +0000 9647)                      nil 'nomesg))

which is as far back the git repository I had lying around seems to
go:

    commit aa088210dd9c90f00466b59c58f60f14b4544e69 (HEAD, tag: v0-5)
    Author: Lars Magne Ingebrigtsen <larsi@gnus.org>
    Date:   Tue Mar 4 01:26:40 1997 +0000

        Initial revision


   :-),

    Adam

-- 
 "Do you think the people who make these decisions          Adam Sjøgren
  about what features a project will have are the      asjo@koldfront.dk
  *engineers*? Because that's adorable."

Re: Does Gnus read and sync the read with IMAP?

[Eric Abrahamsen]

Björn Bidar <bjorn.bidar@thaodan.de> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> On Oct 20, 2023, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>
>>> Dan Christensen <jdc@uwo.ca> writes:
>>>
>>>> Here's a followup question: how can I make Gnus update the marks
>>>> for
>>>> *all* IMAP groups.
>>>
>>> I once asked Lars about this -- since the IMAP server should be the
>>> source of truth for all information about the messages, why does
>>> Gnus
>>> store its own marks data for IMAP groups at all? Why not just
>
> How many imap server support imap metadata? Clients such as Thunderbird
> already safe their metadata in imap.
>
>>> refresh
>>> from the server at each startup? I think he said that for many large
>>> groups that would end up being prohibitively slow.
>>
>> I agree that it would be too slow for many people, especially with
>> remote IMAP servers.  So I'd like an optional setting.  But I don't
>> think it should have anything to do with Gnus shutdown and startup.
>
> Why would it be to slow? Other mail clients fetch the read
> status for each mail when they update the state of each
> imap mailbox.

I have never tried it, so I don't know if it would be slow or not. But
this wouldn't just update the read status, this would remove all
existing marks and refresh them from the server: ticks, replied,
forwarded, everything. I could imagine that being pretty slow.

Re: Does Gnus read and sync the read with IMAP?

[Björn Bidar]

Dan Christensen <jdc@uwo.ca> writes:

> On Oct 20, 2023, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> Here's a followup question: how can I make Gnus update the marks
>>> for
>>> *all* IMAP groups.
>>
>> I once asked Lars about this -- since the IMAP server should be the
>> source of truth for all information about the messages, why does
>> Gnus
>> store its own marks data for IMAP groups at all? Why not just

How many imap server support imap metadata? Clients such as Thunderbird
already safe their metadata in imap.

>> refresh
>> from the server at each startup? I think he said that for many large
>> groups that would end up being prohibitively slow.
>
> I agree that it would be too slow for many people, especially with
> remote IMAP servers.  So I'd like an optional setting.  But I don't
> think it should have anything to do with Gnus shutdown and startup.

Why would it be to slow? Other mail clients fetch the read
status for each mail when they update the state of each
imap mailbox.

> *Every* time I hit `g', I want the marks to be refreshed from the IMAP
> server.  Presumably this is the same code that runs when Gnus starts
> up,
> so it would improve the behaviour at startup as well.
>
> Dan

Re: Does Gnus read and sync the read with IMAP?

[Dan Christensen]

On Oct 20, 2023, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Here's a followup question:  how can I make Gnus update the marks for
>> *all* IMAP groups.
>
> I once asked Lars about this -- since the IMAP server should be the
> source of truth for all information about the messages, why does Gnus
> store its own marks data for IMAP groups at all? Why not just refresh
> from the server at each startup? I think he said that for many large
> groups that would end up being prohibitively slow.

I agree that it would be too slow for many people, especially with
remote IMAP servers.  So I'd like an optional setting.  But I don't
think it should have anything to do with Gnus shutdown and startup.
*Every* time I hit `g', I want the marks to be refreshed from the IMAP
server.  Presumably this is the same code that runs when Gnus starts up,
so it would improve the behaviour at startup as well.

Dan

Re: Does Gnus read and sync the read with IMAP?

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> On Oct 19, 2023, Björn Bidar <bjorn.bidar@thaodan.de> wrote:
>
>> How can I make Gnus to fetch the read status from Imap?
>
> In the *Group* buffer, you can make Gnus update the marks from the IMAP
> server using `M-g'.
>
> Here's a followup question:  how can I make Gnus update the marks for
> *all* IMAP groups.  I've tried various incantations.  `g' doesn't work,
> nor does `M-g' on a topic containing the IMAP groups.  In fact, I'd
> like Gnus to always refresh IMAP groups in the thorough way that `M-g'
> on the group does, since I'm accessing a local IMAP server.

I once asked Lars about this -- since the IMAP server should be the
source of truth for all information about the messages, why does Gnus
store its own marks data for IMAP groups at all? Why not just refresh
from the server at each startup? I think he said that for many large
groups that would end up being prohibitively slow. I guess that might be
true, I don't know. I do think that Gnus should be much, much more
willing to dump its marks store and refresh if anything at all seems to
be getting out of whack.

Maybe we could put in a customization option that would clear out Gnus'
nnimap group marks at shutdown, which would force a full refresh at the
next startup. That feels like a pretty major change, I'm not sure what
the knock-on consequences might be.

The other common problem (that Bob Newell ran into recently) is that
Gnus' article numbers are purely an internal convention, and don't have
any real correspondence to external reality, yet are relied upon pretty
heavily within Gnus to "make sense" as an indicator of sequence. When
they get out of whack you are SOL. I don't see a great solution to that.

Eric

Re: Does Gnus read and sync the read with IMAP?

[Dan Christensen]

On Oct 19, 2023, Björn Bidar <bjorn.bidar@thaodan.de> wrote:

> How can I make Gnus to fetch the read status from Imap?

In the *Group* buffer, you can make Gnus update the marks from the IMAP
server using `M-g'.

Here's a followup question:  how can I make Gnus update the marks for
*all* IMAP groups.  I've tried various incantations.  `g' doesn't work,
nor does `M-g' on a topic containing the IMAP groups.  In fact, I'd
like Gnus to always refresh IMAP groups in the thorough way that `M-g'
on the group does, since I'm accessing a local IMAP server.

Dan

Does Gnus read and sync the read with IMAP?

[Björn Bidar]

Hey,

I noticed that when I read a mail on my phone the read statusis still unread when I fetch my mails from my imap server is still unread in
Gnus.
How can I make Gnus to fetch the read status from Imap?

Br,

BJÖRN