new back-end interface to fix article move/delete problems

new back-end interface to fix article move/delete problems

[Joe Wells]

Dear fellow Gnusians (of any color, but Red preferred),

I have recently done a great deal of reimplementation to fix some problems
in using the mail-back-end commands to move, crosspost, and delete
articles that are crossposted inside the mail back end.  This work
involves a new back-end interface *-request-modify-article which subsumes
the functionality of the *-request-move-article, *-request-accept-article,
and *-request-replace-article interfaces.  I have shown my changes to Lars
and he seems happy with them.

Unfortunately, I have only implemented the back-end portion of my changes
for the nnml back end.  So I would like to recruit Gnus-loving people to
implement this interface for the other back ends.  I have written most of
the nnml implementation in a manner that does not actually depend on nnml.
Unfortunately, I don't understand nnoo well enough to make these functions
available in that way.  I also don't use the other back ends which makes
it difficult for me to test my changes.

I have enclosed a description of the new back-end interface and some of
the changes I have made below.

-- 
Joe Wells <jbw@cs.bu.edu>

----------------------------------------------------------------------
SPECIFICATION OF *-request-modify-article INTERFACE

`(nnchoke-request-modify-article
             SERVER OLD-GROUP-ARTICLE NEW-GROUP-ARTICLE BUFFER
   &optional DONT-SAVE DONT-ADD-EXTRAS-TO-NEW)'

Replaces the article specified by SERVER and OLD-GROUP-ARTICLE with one
specified by NEW-GROUP-ARTICLE and BUFFER (and SERVER).  This may involve
removing the article from some old groups and adding (crossposting) it to
some new groups.

SERVER must be a string identifying an opened virtual server in the nnchoke
back end.

OLD-GROUP-ARTICLE is an alist whose members are of the form (GROUP-NAME
. ARTICLE-NUMBER).  The group names must be distinct from each other and
must not include any server prefix.  Each group must already exist in the
server.  Collectively, the group names and article numbers must belong to
a single cross-posted article in the server.  OLD-GROUP-ARTICLE is
automatically extended to include all other existing group/article pairs
reachable by Xref header links from articles already mentioned in
OLD-GROUP-ARTICLE.  (Call these additional group/article pairs the
"extra" pairs.)  OLD-GROUP-ARTICLE may be empty (indicating that a new
article is being created).

NEW-GROUP-ARTICLE is an alist whose members are of the form (GROUP-NAME
. ARTICLE-NUMBER) or (GROUP-NAME . nil).  The group names must be distinct
from each other and must not include any server prefix.  If any of the
named groups do not exist, they will be created.  For each (GROUP-NAME
. XXX) pair in NEW-GROUP-ARTICLE, one of the following must be true:
  1. (GROUP-NAME . XXX) is a member of OLD-GROUP-ARTICLE.
  2. GROUP-NAME does not appear in OLD-GROUP-ARTICLE and either:
     a. XXX is an article number not already used in group GROUP-NAME.
     b. XXX is nil (indicating an unused article number in group
        GROUP-NAME should be allocated).
NEW-GROUP-ARTICLE may be empty (causing only deletions of the article to
occur).  NEW-GROUP-ARTICLE is automatically extended so as to include the
"extra" group/article pairs mentioned above.  NEW-GROUP-ARTICLE may be
just t or contain the element t in addition to its ordinary alist pairs.
In this case, a list of groups will be added in place of the t based on
nnmail-split-methods and the contents of BUFFER.  For each additional
group not already in NEW-GROUP-ARTICLE, a new pair (GROUP-NAME . ART-NUM)
or (GROUP-NAME . nil) is added, depending on whether such a pair is in
OLD-GROUP-ARTICLE.

Either BUFFER names a buffer containing a correctly formatted article or
BUFFER is nil.  The "Xref" and "Lines" headers do not need to contain
valid data since they will be recomputed.  The "Message-ID" header will
also be replaced if OLD-GROUP-ARTICLE is not nil.  If a "Newsgroups"
header exists, it will be adjusted to agree with the new "Xref" header.
If BUFFER is nil and NEW-GROUP-ARTICLE is non-nil, then OLD-GROUP-ARTICLE
must be non-nil, in which case one of the articles (all probably
identical) it specifies will be used.

If optional parameter DONT-SAVE is non-nil, avoids saving the active file
and NOV database files if possible.

If optional parameter DONT-ADD-EXTRAS-TO-NEW is non-nil, the "extra"
group article pairs are not added to NEW-GROUP-ALIST.

Returns no data (i.e., doesn't do anything to nntp-server-buffer).  The
return value is nil if the operation fails.  The reason for a failure may
be recovered via nnchoke-status-message.  If the operation succeeds, the
return value depends on whether NEW-GROUP-ARTICLE is non-nil.  If
NEW-GROUP-ARTICLE is nil, the return value is t, otherwise the return
value will be the same as NEW-GROUP-ARTICLE except the nil values will be
replaced by the article numbers actually used.  This function
destructively modifies NEW-GROUP-ARTICLE and the contents of BUFFER.

----------------------------------------------------------------------
EXAMPLES OF HOW TO USE THE *-request-modify-article INTERFACE

In these examples, assume that at the start there exists an article
crossposted to groups in the "serv" virtual server of the nnchoke back end
as given by this Xref header:

  Xref: foo.bar.com abcdef:412 ghijkl:398 mnopqrs:43

  (nnchoke-request-modify-article 
             SERVER OLD-GROUP-ARTICLE NEW-GROUP-ARTICLE BUFFER
   &optional DONT-SAVE DONT-ADD-EXTRAS-TO-NEW)'

To add an additional crossposting of the article in group "zyxwvu":

  (nnchoke-request-modify-article "serv" '(("abcdef" . 412))
                                  '(("abcdef" . 412) ("zyxwvu")) nil)

To remove all copies of the article:

  (nnchoke-request-modify-article "serv" '(("abcdef" . 412)) nil nil nil t)

To replace the crossposting in group "abcdef" by one in "zyxwvu" without
affecting the other crosspostings:

  (nnchoke-request-modify-article "serv" '(("abcdef" . 412)) '(("zyxwvu")) nil)

To replace all of the crosspostings of the article by one in "zyxwvu":

  (nnchoke-request-modify-article "serv" '(("abcdef" . 412))
                                  '(("zyxwvu")) nil nil t)

To replace all of the crosspostings of the article by crosspostings
determined by consulting nnmail-split-methods:

  (nnchoke-request-modify-article "serv" '(("abcdef" . 412)) t nil nil t)

To add additional crosspostings determined by consulting
nnmail-split-methods without affecting the existing crosspostings:

  (nnchoke-request-modify-article "serv" '(("abcdef" . 412))
                                  '(t ("abcdef" . 412)) nil)

With any of these examples, one can supply a buffer for the 4th argument
to simultaneously change the body and/or headers of the article.

To add a new article to the system (or copy an old article) into group
"zyxwvu" using the headers and body in BUFFER:

  (nnchoke-request-modify-article "serv" nil '(("zyxwvu")) BUFFER)

To add a new article to the system (or copy an old article) using groups
determined by consulting nnmail-split-methods and using the headers and
body in BUFFER:

  (nnchoke-request-modify-article "serv" nil t BUFFER)

----------------------------------------------------------------------
DESCRIPTION OF CHANGES (in pseudo-ChangeLog format)

gnus-sum.el:

  New Summary mode keybindings:

    B D       gnus-summary-backend-delete-map
    B D d     gnus-summary-delete-article
    B D a     gnus-summary-delete-article-crosspostings-also
    B D o     gnus-summary-delete-article-other-crosspostings-only

  gnus-summary-move-article:

    Reimplemented using new *-request-modify-article interface.  No longer
    uses *-request-move or *-request-accept.

    Doc string is now complete, including correct description of crosspost
    and copy behavior specified by the ACTION parameter.

    Fixes these problems/bugs:

      New articles were not crossposted with the source article (or
      preexisting articles already crossposted with the source article)
      when possible, even when ACTION was crosspost.  (There was code to
      do this, it was just broken in a number of different ways.) 

      Xref headers in article files were sometimes left different from the
      value in the NOV file.  (This happened for pre-existing articles
      that were crossposted with the source article.)

      Corrupt Xref headers were generated.  (Spaces were missing.)

      Marks in the wrong group could be updated after a move.

      Marks were not propagated when ACTION was crosspost.

      Would propagate marks when ACTION was copy.

      Would leave dangling Xref links.  (The Xref header would point to
      nonexistent articles.  This was not a serious problem since Gnus is
      accustomed to dealing with this in the case of expired articles.)

      If the code for handling Xref links when crossposting had worked at
      all, it could have done crossposts across multiple back ends and
      created Xref lines which mentioned the same newsgroup twice!  These
      would be serious semantic errors.
      
      Did not update the various copies of an article's header vector in
      active summary buffers.
      
      Did not flush the old version of an article (with the old Xref
      header) from various caches (backlog, async prefetch,
      original-article-buffer).  *** Still not fixed for the "cache". ***
      
      Did not check that at most one of TO-GROUP and SELECT-METHOD were
      non-nil.

    I'm still working on fixing these problems:

      Marks are never propagated to new groups created as part of the
      move.

      Does not update the Xref header in the Article buffer if viewing the
      article in question.
      
  gnus-summary-delete-article-crosspostings-also:
    New function.

  gnus-summary-delete-article-other-crosspostings-only:
    New function.

  gnus-summary-delete-article:

    Added optional parameter CROSSPOSTINGS to support previous two
    functions.

    Reimplemented using *-request-modify-article for two reasons:
      1. Supporting previous two functions.
      2. Fix bug where Xref headers were not kept up-to-date.  (This is
         less important.)

    Doc string now explains return value.

nnml.el:

  nnml-request-move-article, nnml-request-accept-article,
  nnml-request-replace-article:

    Completely reimplemented using nnml-request-modify-article.
    
  nnml-request-modify-article:
    
    New function.

    Preserves Xref headers properly in both articles and NOV files.
    Can add new crosspostings to an existing article.

    Subsumes functionality of *-request-move-article,
    *-request-accept-article, and *-request-replace-article interfaces,
    but also allows much more with a very easy-to-use interface.  New
    interface is completely documented in the doc string in a form
    suitable for pasting into the manual section on back-end interfaces.
    (Interface specification included below at end of this file.)

    Fixes bug where LAST parameter of nnml-request-move-article or
    nnml-request-accept-article might be t but the NOV files and active
    list are not flushed to disk due to an error causing an abort before
    the flushing code is reached.

  nnml-validate-existing-gaa,
  nnml-validate-potential-gaa,
  nnml-verify-writable-gaa,
  nnml-create-gaa-groups,
  nnml-allocate-gaa-article-numbers,
  nnml-delete-gaa,
  nnml-gaa-difference,
  nnml-gaa-union,
  nnml-find-other-crossrefs-for-gaa,
  nnml-article-header,
  nnml-article-message-id:
    New functions used by nnml-request-modify-article.

  nnml-add-nov:
    Wrote complete doc string.
    
    Extended to also be able to set the NOV entry for an arbitrary article
    number, existing or not.  Still runs fast for append case.
 
gnus-int.el:

  gnus-request-modify-article:
    New function.  Written in the standard manner for a back-end interface
    wrapper.  Only non-trivial aspect: It changes its method argument into
    a server name (i.e., address) argument.

----------------------------------------------------------------------

Re: new back-end interface to fix article move/delete problems

[Paul Franklin]

[I've been meaning to respond for a while.]

Joe proposed some backend interface modifications to fix problems with
moving, etc. crossposted articles, which I'm assuming will make it
into qgnus.  I've attached an exerpt of the part relevant to me.

In general, these seem great!  However, I'm considering writing a
backend based on nnml (if I can figure out the nnoo stuff) which might
not be able to handle case 2a for an element of NEW-GROUP-ARTICLE.  So
I'm wondering when this case would get used.

(If you're wondering, my imaginary backend allows mail delivery
directly into group directories, so gnus might think the next article
number in sequence isn't used when in reality it is.  But I don't plan
to eliminate the getting new mail step; this would scan the current
mail spool, which now only contains enough information to update the
overview and active files.  I have no idea if I can get it to work;
the crutial characteristic is that both the mail delivery agent and
the gnus backend need to back off and keep incrementing their idea of
the last message number if it exists, and the mail delivery agent also
reads the backend's active file to avoid filling in holes.)

--Paul

>>>>> Joe Wells writes:

 > I have recently done a great deal of reimplementation to fix some problems
 > in using the mail-back-end commands to move, crosspost, and delete
 > articles that are crossposted inside the mail back end.  This work
 > involves a new back-end interface *-request-modify-article which subsumes
 > the functionality of the *-request-move-article, *-request-accept-article,
 > and *-request-replace-article interfaces.

 > [...]

 > ----------------------------------------------------------------------
 > SPECIFICATION OF *-request-modify-article INTERFACE

 > `(nnchoke-request-modify-article
 >              SERVER OLD-GROUP-ARTICLE NEW-GROUP-ARTICLE BUFFER
 >    &optional DONT-SAVE DONT-ADD-EXTRAS-TO-NEW)'

 > Replaces the article specified by SERVER and OLD-GROUP-ARTICLE with one
 > specified by NEW-GROUP-ARTICLE and BUFFER (and SERVER).  This may involve
 > removing the article from some old groups and adding (crossposting) it to
 > some new groups.

 > [...]

 > NEW-GROUP-ARTICLE is an alist whose members are of the form (GROUP-NAME
 > . ARTICLE-NUMBER) or (GROUP-NAME . nil).  The group names must be distinct
 > from each other and must not include any server prefix.  If any of the
 > named groups do not exist, they will be created.  For each (GROUP-NAME
 > . XXX) pair in NEW-GROUP-ARTICLE, one of the following must be true:
 >   1. (GROUP-NAME . XXX) is a member of OLD-GROUP-ARTICLE.
 >   2. GROUP-NAME does not appear in OLD-GROUP-ARTICLE and either:
 >      a. XXX is an article number not already used in group GROUP-NAME.
 >      b. XXX is nil (indicating an unused article number in group
 >         GROUP-NAME should be allocated).
 > NEW-GROUP-ARTICLE may be empty (causing only deletions of the article to
 > occur).  NEW-GROUP-ARTICLE is automatically extended so as to include the
 > "extra" group/article pairs mentioned above.  NEW-GROUP-ARTICLE may be
 > just t or contain the element t in addition to its ordinary alist pairs.
 > In this case, a list of groups will be added in place of the t based on
 > nnmail-split-methods and the contents of BUFFER.  For each additional
 > group not already in NEW-GROUP-ARTICLE, a new pair (GROUP-NAME . ART-NUM)
 > or (GROUP-NAME . nil) is added, depending on whether such a pair is in
 > OLD-GROUP-ARTICLE.

 > [...]

Re: new back-end interface to fix article move/delete problems

[Joe Wells]

>>>>> "Paul" == Paul Franklin <paul@cs.washington.edu> writes:

  Paul> Joe proposed some backend interface modifications to fix problems with
  Paul> moving, etc. crossposted articles, which I'm assuming will make it
  Paul> into qgnus.  ...

  Paul> ...  However, I'm considering writing a
  Paul> backend based on nnml (if I can figure out the nnoo stuff) which might
  Paul> not be able to handle case 2a for an element of NEW-GROUP-ARTICLE.  So
  Paul> I'm wondering when this case would get used.

The relevant part of the proposed back-end-interface:

  >> NEW-GROUP-ARTICLE is an alist whose members are of the form (GROUP-NAME
  >> . ARTICLE-NUMBER) or (GROUP-NAME . nil).  The group names must be distinct
  >> from each other and must not include any server prefix.  If any of the
  >> named groups do not exist, they will be created.  For each (GROUP-NAME
  >> . XXX) pair in NEW-GROUP-ARTICLE, one of the following must be true:
  >>   1. (GROUP-NAME . XXX) is a member of OLD-GROUP-ARTICLE.
  >>   2. GROUP-NAME does not appear in OLD-GROUP-ARTICLE and either:
  >>      a. XXX is an article number not already used in group GROUP-NAME.
  >>      b. XXX is nil (indicating an unused article number in group
  >>         GROUP-NAME should be allocated).

There are two goals behind possibility 2a (neither of which have been
implemented):

  1. An article is originally crossposted in groups A, B, and C as
     articles 5, 6, and 7 (respectively).  The user removes it from group
     C.  Later, the user asks for the the article to be crossposted in
     group C.  It should be possible to reuse the old number 7 rather than
     the number that would be given to a new article.

  2. The user asks for an article to be crossposted to group A.  The
     article was originally sent in 1994 (three years ago).  If the
     article is given the number in group A that would be given to a new
     article, then it may be listed in the summary buffer for that group
     after many articles that are actually newer than it.  If there were
     gaps in the sequence of existing article numbers in group A, it
     should be possible to use one of the numbers in the gaps instead, so
     that the article gets listed in a more natural order.

Goal number 2 is more questionable than goal number 1.

A reasonable restriction of the proposed back-end-interface would be to
require that a non-existing article number must be one that the article
used earlier that has since been deleted.  This would support goal 1 but
not goal 2.  It would also probably avoid the problems that Paul
envisioned.

-- 
Joe Wells <jbw@cs.bu.edu>

Re: new back-end interface to fix article move/delete problems

[Paul Franklin]

>>>>> Joe Wells writes:

 > A reasonable restriction of the proposed back-end-interface would be to
 > require that a non-existing article number must be one that the article
 > used earlier that has since been deleted.  This would support goal 1 but
 > not goal 2.  It would also probably avoid the problems that Paul
 > envisioned.

Yup.  I would also be perfectly happy with the lesser restriction that
it be less than the highest known article number (specified in the
active file), since I'm only worried about articles which might exist
above that highest known article number.

I just didn't want someone to decide to supply 1 more than the last
number in the active file because "I know what article number the
backend would use."

--Paul