nnimap article editing

nnimap article editing

[Didier Verna]

       Hi there,

nnimap doesn't support article editing, but it seems that I can "upload"
messages, because for instance, I've just "B m"'ed a whole group from a
local account to gmail.

So am I correct in assuming that we could implement nnimap article
editing by simply downloading the article, editing it, destroying the
old version and uploading the new one ?

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

EPITA/LRDE, 14-16 rue Voltaire, 94276 Le Kremlin-Bicêtre, France
Tel. +33 (0)1 44 08 01 85       Fax. +33 (0)1 53 14 59 22

Re: nnimap article editing

[David Engster]

Didier Verna <didier@xemacs.org> writes:
>nnimap doesn't support article editing

Yes, because the IMAP protocol doesn't allow it.

> So am I correct in assuming that we could implement nnimap article
> editing by simply downloading the article, editing it, destroying the
> old version and uploading the new one ?

That's how other mail clients do it, e.g. for removing attachments from
articles to save space on the server. The problem with this approach is
that the edited article gets a new UID from the server (and hence a new
article number in Gnus), which can lead to problems regarding sorting,
caching, calculation of article numbers, etc.

-David

Re: nnimap article editing

[Ted Zlatanov]

On Thu, 09 Oct 2008 13:52:02 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Didier Verna <didier@xemacs.org> writes:

>> So am I correct in assuming that we could implement nnimap article
>> editing by simply downloading the article, editing it, destroying the
>> old version and uploading the new one ?

DE> That's how other mail clients do it, e.g. for removing attachments from
DE> articles to save space on the server. The problem with this approach is
DE> that the edited article gets a new UID from the server (and hence a new
DE> article number in Gnus), which can lead to problems regarding sorting,
DE> caching, calculation of article numbers, etc.

This, if implemented, would work for any backend that accepts articles,
not just nnimap.

I don't see a problem with sorting or caching.  Expire the old article,
that's all.  The new one should have the same Message-ID but otherwise
can be completely different.  It doesn't even need the same Message-ID,
it can be treated as a followup if that breaks any functionality.  Users
can deal with this.  We could use the Gnus registry to track things if
necessary, but I can't think of a scenario that requires it.

Ted

Re: nnimap article editing

[David Engster]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Thu, 09 Oct 2008 13:52:02 +0200 David Engster <deng@randomsample.de> wrote: 
> DE> Didier Verna <didier@xemacs.org> writes:
>>> So am I correct in assuming that we could implement nnimap article
>>> editing by simply downloading the article, editing it, destroying the
>>> old version and uploading the new one ?
>
> DE> That's how other mail clients do it, e.g. for removing attachments from
> DE> articles to save space on the server. The problem with this approach is
> DE> that the edited article gets a new UID from the server (and hence a new
> DE> article number in Gnus), which can lead to problems regarding sorting,
> DE> caching, calculation of article numbers, etc.
>
> This, if implemented, would work for any backend that accepts articles,
> not just nnimap.

OK, so you're talking about implementing an alternative to article
editing through delete-and-create, without touching nnimap at all? I
understood the original posting to implement
nnimap-request-replace-article, which would be necessary for editing
articles in nnimap like e.g. in nnml groups, which just cannot be done
for IMAP, at least not according to the specification given in the
manual ("remove ARTICLE (which is a number) from GROUP and insert BUFFER
there instead").

> I don't see a problem with sorting or caching.  Expire the old article,
> that's all.  The new one should have the same Message-ID but otherwise
> can be completely different. 

The problem I see is that the article number would change while the
article is displayed. Hence, after editing, Gnus would have to at least
quit the summary buffer, rescan the group, and then re-enter the summary
to display the edited article. Users who use the *-by-number functions
for sorting will probably see edited articles as most recent, but that's
indeed a small issue. As for caching (and the agent), I don't know if
they depend in any way on a mapping between UID and Message-ID? If not,
it shouldn't be a problem. In any case, I admit having an article
editing solution like this is better than none at all.

-David

Re: nnimap article editing

[Ted Zlatanov]

On Thu, 09 Oct 2008 16:58:51 +0200 David Engster <deng@randomsample.de> wrote: 

DE> Ted Zlatanov <tzz@lifelogs.com> writes:
>> This, if implemented, would work for any backend that accepts articles,
>> not just nnimap.

DE> OK, so you're talking about implementing an alternative to article
DE> editing through delete-and-create, without touching nnimap at all? I
DE> understood the original posting to implement
DE> nnimap-request-replace-article, which would be necessary for editing
DE> articles in nnimap like e.g. in nnml groups, which just cannot be
DE> done for IMAP, at least not according to the specification given in
DE> the manual ("remove ARTICLE (which is a number) from GROUP and
DE> insert BUFFER there instead").

Right, it would be BACKEND-request-replace-or-supersede-article
(whatever name people like) which would fall back to
BACKEND-request-replace-article if that was available.

>> I don't see a problem with sorting or caching.  Expire the old article,
>> that's all.  The new one should have the same Message-ID but otherwise
>> can be completely different. 

DE> The problem I see is that the article number would change while the
DE> article is displayed. Hence, after editing, Gnus would have to at least
DE> quit the summary buffer, rescan the group, and then re-enter the summary
DE> to display the edited article. 

I think that's reasonable, and probably can be done faster once the
basic operation works reliably.

DE> Users who use the *-by-number functions for sorting will probably
DE> see edited articles as most recent, but that's indeed a small
DE> issue. 

I think it's actually the expected behavior--users would expect the sort
to be by modification time, not creation time.  We can use the registry
to link the Message-ID to the creation time if necessary, it's not hard,
but I doubt many will care.

DE> As for caching (and the agent), I don't know if they depend in any
DE> way on a mapping between UID and Message-ID? If not, it shouldn't be
DE> a problem. In any case, I admit having an article editing solution
DE> like this is better than none at all.

I think the only part of Gnus that really cares about the Message-ID is
the registry.  Everything else goes by article number (based on what
I've seen in the code).  Maybe someone that knows the agent and caching
can explain better.

Ted

Re: nnimap article editing

[Reiner Steib]

On Thu, Oct 09 2008, Ted Zlatanov wrote:

> The new one should have the same Message-ID but otherwise
> can be completely different.  It doesn't even need the same Message-ID,
> it can be treated as a followup if that breaks any functionality.

Creating a new Message-ID would break threading.  The edited article
should have the same Message-ID, as we do for nnml, etc.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnimap article editing

[Ted Zlatanov]

On Thu, 09 Oct 2008 19:36:23 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Thu, Oct 09 2008, Ted Zlatanov wrote:
>> The new one should have the same Message-ID but otherwise
>> can be completely different.  It doesn't even need the same Message-ID,
>> it can be treated as a followup if that breaks any functionality.

RS> Creating a new Message-ID would break threading.  The edited article
RS> should have the same Message-ID, as we do for nnml, etc.

Do you know of anything that would break if two messages have the same
Message-ID?  I don't, so I agree with you that we should stick with the
same Message-ID, but my point was that if there is such functionality
that depends on unique Message-IDs we will have to fix it or change the
Message-ID of the new message.

Ted

Re: nnimap article editing

[Christoph Conrad]

Hi Ted,

> Do you know of anything that would break if two messages have the same
> Message-ID?

E.g. cancelling messages and superseding.

Kind regards,
Christoph

Re: nnimap article editing

[Ted Zlatanov]

On Thu, 09 Oct 2008 22:02:00 +0200 Christoph Conrad <christoph.conrad@gmx.de> wrote: 

CC> Hi Ted,
>> Do you know of anything that would break if two messages have the same
>> Message-ID?

CC> E.g. cancelling messages and superseding.

I don't think cancelling applies to writeable backends such as nnimap,
does it?  I thought it was just for nntp, and the code in
message-cancel-message only works where (message-news-p) is true.  So I
think cancelling won't break.

message-supersede seems to just copy the message into a buffer with a
Supersedes header set to the old Message-ID.  A new Message-ID gets
generated as usual for new messages.  This actually may be how we
implement generic article editing anyhow--it seems like an easy way to
tell Gnus that the new article is an edit of the old one.  

When superseding happens more than once (through actual calls of
message-supersede or through generic article editing), multiple
Supersedes headers would accumulate, and will be merged into one (the
header, according to the RFC 4021, can have 1* message IDs).  This
rewrite is not hard and can happen inside message-supersede.  Thus the
trail of edits can be preserved.

Ted

Re: nnimap article editing

[Christoph Conrad]

Hi Ted,

CC> E.g. cancelling messages and superseding.

> I don't think cancelling applies to writeable backends such as nnimap,
> does it? I thought it was just for nntp, and the code in
> message-cancel-message only works where (message-news-p) is true. So I
> think cancelling won't break.

Sounds ok. I must confess that i did not follow the whole thread. My
posting was overhasty.

Kind regards,
Christoph

Re: nnimap article editing

[Reiner Steib]

On Thu, Oct 09 2008, Ted Zlatanov wrote:

> On Thu, 09 Oct 2008, Reiner Steib <reinersteib+gmane@imap.cc> wrote: 
> RS> Creating a new Message-ID would break threading.  The edited
> RS> article should have the same Message-ID, as we do for nnml, etc.
>
> Do you know of anything that would break if two messages have the same
> Message-ID?  

I'd be surprised if there are any problems with duplicate MIDs in mail
back ends.  It's quite a normal situation, cf. (info
"(gnus)Duplicates").  You can also generate multiple copies of a
message (with the same MID) with `gnus-summary-copy-article' (B c) in
the same group.  I never tried threading in this situation (multiple
parents with the same MID).

> I don't, so I agree with you that we should stick with the same
> Message-ID, but my point was that if there is such functionality
> that depends on unique Message-IDs we will have to fix it or change
> the Message-ID of the new message.

I'm not aware of it, but I might be missing something.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnimap article editing

[Ted Zlatanov]

On Thu, 09 Oct 2008 22:45:59 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Thu, Oct 09 2008, Ted Zlatanov wrote:
>> On Thu, 09 Oct 2008, Reiner Steib <reinersteib+gmane@imap.cc> wrote: 
RS> Creating a new Message-ID would break threading.  The edited
RS> article should have the same Message-ID, as we do for nnml, etc.
>> 
>> Do you know of anything that would break if two messages have the same
>> Message-ID?  

RS> I'd be surprised if there are any problems with duplicate MIDs in mail
RS> back ends.  It's quite a normal situation, cf. (info
RS> "(gnus)Duplicates").  You can also generate multiple copies of a
RS> message (with the same MID) with `gnus-summary-copy-article' (B c) in
RS> the same group.  I never tried threading in this situation (multiple
RS> parents with the same MID).

OK, so without using Supersede: it will work thus:

1) get article contents in a buffer, keep Message-ID
2) let user edit, then on submit...
3) spool article as new and mark original expired (or read maybe?)

This will be called if BACKEND doesn't support direct editing.  So
nnmaildir and nnml for instance will not be affected.

Ted

Re: nnimap article editing

[Reiner Steib]

On Fri, Oct 10 2008, Ted Zlatanov wrote:

> 1) get article contents in a buffer, keep Message-ID
> 2) let user edit, then on submit...
> 3) spool article as new and mark original expired (or read maybe?)

I think the behavior should be as close to the local back ends (nnml,
nnfolder, nnmaildir, ...) as possible.  I'd expect the old article to
be deleted, but maybe marking it as expirable might be sufficient
(especially during a testing period of this feature).

A typical usage for article editing is to strip or externalize large
attachments.  If the original article isn't deleted, the space is not
released (think of IMAP quotas).

But other users may have a different usage pattern, so it might make
sense to make it customizable and/or prompt after editing.

> This will be called if BACKEND doesn't support direct editing.  So
> nnmaildir and nnml for instance will not be affected.

ACK.

Please keep in mind not to commit new features to the trunk until we
decided about the Gnus development during Emacs feature freeze, see
<http://thread.gmane.org/87wsi91d8d.fsf@marauder.physik.uni-ulm.de>.
(I didn't have time to push this issue further after my vacations.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nnimap article editing

[Ted Zlatanov]

On Fri, 10 Oct 2008 20:14:12 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> Please keep in mind not to commit new features to the trunk until we
RS> decided about the Gnus development during Emacs feature freeze, see
RS> <http://thread.gmane.org/87wsi91d8d.fsf@marauder.physik.uni-ulm.de>.
RS> (I didn't have time to push this issue further after my vacations.)

I will not commit anything until I hear back from you.

Thanks
Ted

Gnus CVS checkins (was: nnimap article editing)

[Ted Zlatanov]

On Fri, 10 Oct 2008 20:14:12 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> Please keep in mind not to commit new features to the trunk until we
RS> decided about the Gnus development during Emacs feature freeze, see
RS> <http://thread.gmane.org/87wsi91d8d.fsf@marauder.physik.uni-ulm.de>.
RS> (I didn't have time to push this issue further after my vacations.)

Can you make a decision or discuss this?

I'm waiting on:

- diagnostic messages for starttls.el plus one new function to be used
  by smtpmail.el

- diagnostic messages for nnrss item hashing

I committed the following:

- ietf-drums fix of broken parsing

- auth-source fix to cache password and login (works around EPA issues
  on Mac OS X)

Thanks
Ted

Re: Gnus CVS checkins

[Reiner Steib]

On Mon, Nov 03 2008, Ted Zlatanov wrote:

> On Fri, 10 Oct 2008 20:14:12 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 
>
> RS> Please keep in mind not to commit new features to the trunk until we
> RS> decided about the Gnus development during Emacs feature freeze, see
> RS> <http://thread.gmane.org/87wsi91d8d.fsf@marauder.physik.uni-ulm.de>.
> RS> (I didn't have time to push this issue further after my vacations.)
>
> Can you make a decision or discuss this?

I've sent a (personal) mail to Lars asking for his opinion on Oct 29,
but he didn't answer yet.

> I'm waiting on:
>
> - diagnostic messages for starttls.el plus one new function to be used
>   by smtpmail.el

smtpmail.el is not part of Gnus.  If the Emacs maintainers don't
object, it's fine with me to install it now.

> - diagnostic messages for nnrss item hashing

Dunno about this.

> I committed the following:
>
> - ietf-drums fix of broken parsing
>
> - auth-source fix to cache password and login (works around EPA issues
>   on Mac OS X)

IIUC, the latter two are bug fixes.  So it was okay to commit them.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus CVS checkins

[Ted Zlatanov]

On Mon, 03 Nov 2008 21:19:17 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Mon, Nov 03 2008, Ted Zlatanov wrote:
>> On Fri, 10 Oct 2008 20:14:12 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 
>> 
RS> Please keep in mind not to commit new features to the trunk until we
RS> decided about the Gnus development during Emacs feature freeze, see
RS> <http://thread.gmane.org/87wsi91d8d.fsf@marauder.physik.uni-ulm.de>.
RS> (I didn't have time to push this issue further after my vacations.)
>> 
>> Can you make a decision or discuss this?

RS> I've sent a (personal) mail to Lars asking for his opinion on Oct 29,
RS> but he didn't answer yet.

>> I'm waiting on:
>> 
>> - diagnostic messages for starttls.el plus one new function to be used
>> by smtpmail.el

RS> smtpmail.el is not part of Gnus.  If the Emacs maintainers don't
RS> object, it's fine with me to install it now.

OK.

>> - diagnostic messages for nnrss item hashing

RS> Dunno about this.

It finds out what RSS fields need to be ignored (otherwise you get
duplicates).  The fields vary per feed and bugs can come and go in the
remote server's feed generator, so it's valuable to discover the actual
data in a message.

Ted