Backend writing

Backend writing

[Sean Rima]

I am very new to both elisp and gnus, but until recently I used
Emacs/Gnus in conjuction with Fidogate/Inn to read Fidonet mail. Due
to a change in my system setup this is no longer possible and I am
thinking about writing a backend to read (and later) write Fidonet
netmails. These are stored as *.msg files, which I can see as not
being that difficult to read.

What I seek is a detailed FAQ on writing backends. Anyone know where I
could get one, part from the info files.

Sean
-- 
BBS and Fidonet News Portal at http://tcob1.uklinux.net
ICQ 679813 Fidonet 2:263/950
Boring sig really

Re: Backend writing

[Kai Großjohann]

On Fri, 03 Aug 2001, Sean Rima wrote:

> What I seek is a detailed FAQ on writing backends. Anyone know where
> I could get one, part from the info files.

I think everybody who has written a backend has just read the relevant
section from the info file (`Backend Interface' and its children).
And then they've played it by ear.

If you have a specific question, you can look in other backends which
are similar w.r.t. that specific question.

Or ask here.

kai
-- 
~/.signature: No such file or directory

Re: Backend writing

[Paul Jarc]

Sean Rima <sean@**Delete**.tcob1.net> writes:
> What I seek is a detailed FAQ on writing backends. Anyone know where I
> could get one, part from the info files.

If there's anything you need to know that isn't in the manual, it
probably ought to be added.  Ask questions here so the maintainers
know what to add.

One thing that I would have liked to find there, but didn't, is what
conditions article numbers must satisfy.  Numbers of new articles must
be greater than all existing numbers.  They should probably start from
1; they must not be zero or negative.  As long as you start from 1 and
increment by 1, don't worry about what happens when you reach 2^28
(the largest integer Elisp can handle); no one else does, and it
hasn't been a problem yet.  Unless you do something unusual like
nnmaildir does, an article must keep the same number for its entire
lifetime - once assigned to an article, a number cannot be reassigned
to another article.

nnfoo-request-scan is described as optional - Gnus can get by without
it - but it'll make your life easier if you implement it.  That was my
experience, anyway.

A good way too see what precise format things must be in in
nntp-server-buffer is to look at the contents of that buffer
(" *nntpd*") while using an existing backend.


paul

Re: Backend writing

[Kai Großjohann]

On Fri, 03 Aug 2001, Paul Jarc wrote:

> One thing that I would have liked to find there, but didn't, is what
> conditions article numbers must satisfy.

I have added the following to the end of the `Backend Interface'
node.  Please comment.  (Especially on the `???' part.)

/----
| Gnus identifies each message by way of group name and article number.
| A few remarks about these article numbers might be useful.  First of
| all, the numbers are positive integers.  Secondly, it is not allowed for
| later articles to `re-use' older article numbers.  That is, if a group
| has ever contained a message numbered 42, then no other message may get
| that number, or Gnus will get mightily confused.@footnote{??? Is this
| still true with the nnchoke-request-*-mark* functions?}  Third, article
| numbers must be assigned in order of arrival in the group; this is not
| necessarily the same as the date of the message.
| 
| The previous paragraph already mentions all the `hard' restrictions that
| article numbers must fulfill.  But it seems that it might be useful to
| assign @emph{consecutive} article numbers, for Gnus gets quite confused
| if there are holes in the article numbering sequence.  However, due to
| the `no-reuse' restriction, holes cannot be avoided altogether.
\----


kai
-- 
~/.signature: No such file or directory

Re: Backend writing

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>| Secondly, it is not allowed for later articles to `re-use' older
>| article numbers.

I would suggest "it is normally not possible [...] without confusing
Gnus", or something like that, to avoid giving the impression that
Gnus is capable of detecting and reporting this.

>| That is, if a group has ever contained a message numbered 42, then
>| no other message may get that number, or Gnus will get mightily
>| confused.@footnote{??? Is this still true with the
>| nnchoke-request-*-mark* functions?}

You can reassign article numbers, but it takes some extra work.  You
let Gnus notify your backend about article marks by implementing
nnchoke-request-set-mark.  That function must store marks somehow,
typically in a way similar to how articles themselves are stored.  The
stored marks must identify the articles they belong to by some other
identification that will never change during an article's lifetime.
Your backend must then hand the marks back to Gnus via
nnchoke-request-update-info, mapping the persistent article
identification to article numbers.  Your backend must maintain the
mapping between numbers and the persistent identification, so that
-request-set-mark will know which marks belong to which articles.  You
can reassign numbers at each call to -request-update-info, but
depending on the details of your backend, it may be easier to assign
them only when opening the server and scanning for new articles, and
to let that numbering remain in effect until the server is closed.

>| But it seems that it might be useful to assign @emph{consecutive}
>| article numbers, for Gnus gets quite confused if there are holes in
>| the article numbering sequence.

It's also useful to start from 1.  Otherwise, you'll run out of
article numbers earlier than you would have.

Another useful addition to the description of each function would be a
list of user actions that cause that function to be called.  A backend
author will probably know when to perform certain backend actions in
terms of user actions, but figuring out the mapping between user
actions and backend function calls takes some work.


paul

Re: Backend writing

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>| that number, or Gnus will get mightily confused.@footnote{??? Is this
>| still true with the nnchoke-request-*-mark* functions?}  Third, article

It's still true.

>| The previous paragraph already mentions all the `hard' restrictions that
>| article numbers must fulfill.  But it seems that it might be useful to
>| assign @emph{consecutive} article numbers, for Gnus gets quite confused
>| if there are holes in the article numbering sequence.  However, due to
>| the `no-reuse' restriction, holes cannot be avoided altogether.

I would say "is slowed down" instead of "gets quite confused".  I
don't think there are any problems, other than that the `gnus-range-*'
functions start to take very long time.  Some IMAP servers leave large
holes.

Re: Backend writing

[Kai Großjohann]

On Fri, 03 Aug 2001, Simon Josefsson wrote:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
>>| that number, or Gnus will get mightily confused.@footnote{??? Is
>>| this still true with the nnchoke-request-*-mark* functions?}
>>| Third, article
> 
> It's still true.

But haven't you just changed nnimap to ignore article renumberings?  I
suppose you wouldn't do that unless it's safe?

>>| The previous paragraph already mentions all the `hard'
>>| restrictions that article numbers must fulfill.  But it seems that
>>| it might be useful to assign @emph{consecutive} article numbers,
>>| for Gnus gets quite confused if there are holes in the article
>>| numbering sequence.  However, due to the `no-reuse' restriction,
>>| holes cannot be avoided altogether.
> 
> I would say "is slowed down" instead of "gets quite confused".  I
> don't think there are any problems, other than that the
> `gnus-range-*' functions start to take very long time.  Some IMAP
> servers leave large holes.

I meant that Gnus says `1276423 messages in this group, how many?' or
something like that when the user enters the group.

kai
-- 
~/.signature: No such file or directory

Re: Backend writing

[Kai Großjohann]

Thanks for the additional comments.  I hope I have incorporated them.
kai
-- 
~/.signature: No such file or directory

Re: Backend writing

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>> 
>>>| that number, or Gnus will get mightily confused.@footnote{??? Is
>>>| this still true with the nnchoke-request-*-mark* functions?}
>>>| Third, article
>> 
>> It's still true.
> 
> But haven't you just changed nnimap to ignore article renumberings?  I
> suppose you wouldn't do that unless it's safe?

I don't know if it's safe, that's what Oort is for. <g>

Basicly, nnimap makes sure the old NOV cache isn't used when a
uidvalidity has changed.  Nnimap also updates all Gnus marks when
entering a group so they should be OK.  The only thing left is the
active range, and that's updated by Gnus when you `g', `M-g' or
somesuch.  Possibly a uidvalidity change breaks the cache or the agent
if article numbers are actually re-used.  But in several cases IMAP
servers update UIDVALIDITY but still does not re-use article numbers,
so in those cases it's OK to ignore the error.

It gets sort of ugly if a backend has to update internal things of
Gnus's cache and agent.  Some sort of mechanism to handle renumberings
in the cache and agent would be nice.

>>>| The previous paragraph already mentions all the `hard'
>>>| restrictions that article numbers must fulfill.  But it seems that
>>>| it might be useful to assign @emph{consecutive} article numbers,
>>>| for Gnus gets quite confused if there are holes in the article
>>>| numbering sequence.  However, due to the `no-reuse' restriction,
>>>| holes cannot be avoided altogether.
>> 
>> I would say "is slowed down" instead of "gets quite confused".  I
>> don't think there are any problems, other than that the
>> `gnus-range-*' functions start to take very long time.  Some IMAP
>> servers leave large holes.
> 
> I meant that Gnus says `1276423 messages in this group, how many?' or
> something like that when the user enters the group.

That's another side-effect of the high/low article number story.  It's
documented, so maybe we could call it a feature.  For now at least.

Re: Backend writing

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>> But haven't you just changed nnimap to ignore article renumberings?  I
>> suppose you wouldn't do that unless it's safe?
> 
> I don't know if it's safe, that's what Oort is for. <g>

FWIW, nnmaildir has been renumbering articles each time you open the
server since 2001-04-11.  No problems yet.  But you seem to be dealing
with something that changes more frequently than that, right?

> Basicly, nnimap makes sure the old NOV cache isn't used when a
> uidvalidity has changed.  [...]  It gets sort of ugly if a backend
> has to update internal things of Gnus's cache and agent.

Does this mean nnimap is stepping outside the backend interface?

>>>>| for Gnus gets quite confused if there are holes in the article
>>>>| numbering sequence.
>>> 
>>> I would say "is slowed down" instead of "gets quite confused".
>> 
>> I meant that Gnus says `1276423 messages in this group, how many?' or
>> something like that when the user enters the group.

So maybe "the user gets quite confused"? :)


paul

Re: Backend writing

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> Basicly, nnimap makes sure the old NOV cache isn't used when a
> uidvalidity has changed.  [...]  It gets sort of ugly if a backend
> has to update internal things of Gnus's cache and agent.

Does this mean Gnus caches NOV data in memory?  I can't find anything
about that in the manual, so maybe I'm misunderstanding.


paul

Re: Backend writing

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> writes:
>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>>> But haven't you just changed nnimap to ignore article renumberings?  I
>>> suppose you wouldn't do that unless it's safe?
>> 
>> I don't know if it's safe, that's what Oort is for. <g>
> 
> FWIW, nnmaildir has been renumbering articles each time you open the
> server since 2001-04-11.  No problems yet.  But you seem to be dealing
> with something that changes more frequently than that, right?

Depends, some IMAP server seem to change the UIDVALIDITY value more or
less at will (UIDVALIDITY indicate that a renumbering might have taken
place).  Most don't.  And those that do sometimes doesn't actually
renumber anything, so it just "works" anyway.  The problem is when a
uidvalidity change is actually due to a real renumbering, and even
more problem arise if the renumbering re-use article numbers.  This
kind of IMAP server design is rare though, so in practice I don't
think it's a big problem.

I think nnmaildir is different though, don't you hide the renumbering
from Gnus (by translating back and forth between old and new
numbering)?  Nnimap doesn't, it simply present the servers article
numbers to Gnus and hopes for the best.  (And assuming uidvalidity
doesn't change, the IMAP design guarantee that Gnus will be happy.
Problem is when uidvalidity change...).

>> Basicly, nnimap makes sure the old NOV cache isn't used when a
>> uidvalidity has changed.  [...]  It gets sort of ugly if a backend
>> has to update internal things of Gnus's cache and agent.
> 
> Does this mean nnimap is stepping outside the backend interface?

Not this time, nnimap doesn't mess with the cache or agent.  So an
article cached by Gnus might become invalid if a renumbering has
happened.  It would be nice if nnimap could tell the cache/agent to
invalidate everything it knows about stored articles.

Re: Backend writing

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

>> Basicly, nnimap makes sure the old NOV cache isn't used when a
>> uidvalidity has changed.  [...]  It gets sort of ugly if a backend
>> has to update internal things of Gnus's cache and agent.
> 
> Does this mean Gnus caches NOV data in memory?  I can't find anything
> about that in the manual, so maybe I'm misunderstanding.

No, nnimap has it's own (disk) NOV cache.  There was talk about moving
the functionality into Gnus, because other backends might find a NOV
cache nice, but nothing became of it.  It shouldn't be difficult
though, I think.

Re: Backend writing

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> I think nnmaildir is different though, don't you hide the renumbering
> from Gnus (by translating back and forth between old and new
> numbering)?

I have external, non-numeric article identifications that never change
for a given article during its lifetime.  Gnus never sees those; I
translate them to article numbers.  That mapping does change from one
Gnus session to another, to keep the numbers small and consecutive.

> Nnimap doesn't, it simply present the servers article numbers to
> Gnus and hopes for the best.  (And assuming uidvalidity doesn't
> change, the IMAP design guarantee that Gnus will be happy.  Problem
> is when uidvalidity change...).

Ah, ok.  So the only IMAP-level identification of an article is
subject to change during an article's lifetime?  Or is there another,
non-volatile piece of identification that just isn't used?


paul

Re: Backend writing

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> prj@po.cwru.edu (Paul Jarc) writes:
>> Does this mean Gnus caches NOV data in memory?  I can't find anything
>> about that in the manual, so maybe I'm misunderstanding.
> 
> No, nnimap has it's own (disk) NOV cache.  There was talk about moving
> the functionality into Gnus, because other backends might find a NOV
> cache nice, but nothing became of it.

Ah, ok.  I was just wondering if I had wasted my time and RAM when I
made nnmaildir cache NOV data in memory.


paul

Re: Backend writing

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> writes:
>> I think nnmaildir is different though, don't you hide the renumbering
>> from Gnus (by translating back and forth between old and new
>> numbering)?
> 
> I have external, non-numeric article identifications that never change
> for a given article during its lifetime.  Gnus never sees those; I
> translate them to article numbers.  That mapping does change from one
> Gnus session to another, to keep the numbers small and consecutive.

Oh!  How do the cache/agent like this?  Maybe the cache/agent isn't
really useful together with nnmaildir though.  But I doubt it would
work, which is sort of bad.  You couldn't have the nnmaildir spool NFS
mounted and use the agent when you don't have network connectivity,
could you?  Do you also frob group info to renumber the marks?  When
do you do this?  Do you use nnchoke-request-update-info?

>> Nnimap doesn't, it simply present the servers article numbers to
>> Gnus and hopes for the best.  (And assuming uidvalidity doesn't
>> change, the IMAP design guarantee that Gnus will be happy.  Problem
>> is when uidvalidity change...).
> 
> Ah, ok.  So the only IMAP-level identification of an article is
> subject to change during an article's lifetime?  Or is there another,
> non-volatile piece of identification that just isn't used?

No.  The article number in IMAP is intended to not change during the
life-time of an article, but IMAP has a concept of UIDVALIDITY so
clients can find out if there has been a renumbering.  The client must
then delete all cached copies of the article.  Most server never
change uidvalidity unless something major happens, but some have a
more liberal interpretation..

Re: Backend writing

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> You couldn't have the nnmaildir spool NFS mounted and use the agent
> when you don't have network connectivity, could you?

I suppose not.  I don't use caching or the agent; when I did this, I
didn't know it would break things.  But people can use the agent on
some servers and not others, right?  Then, at least, they could just
run with the nnmaildir server closed whenever the NFS mount was
unavailable.

> Do you also frob group info to renumber the marks?  When do you do
> this?  Do you use nnchoke-request-update-info?

Yes.  Article numbers aren't stored on disk at all; -request-scan
creates the mapping anew each time you close and reopen the server,
and only keeps it in memory.  Then -request-update-info translates the
stored marks into a whole new group info.


paul

Re: Backend writing

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
>> But haven't you just changed nnimap to ignore article renumberings?  I
>> suppose you wouldn't do that unless it's safe?
> 
> I don't know if it's safe, that's what Oort is for. <g>

He, he.

> Basicly, nnimap makes sure the old NOV cache isn't used when a
> uidvalidity has changed.  Nnimap also updates all Gnus marks when
> entering a group so they should be OK.  The only thing left is the
> active range, and that's updated by Gnus when you `g', `M-g' or
> somesuch.  Possibly a uidvalidity change breaks the cache or the agent
> if article numbers are actually re-used.  But in several cases IMAP
> servers update UIDVALIDITY but still does not re-use article numbers,
> so in those cases it's OK to ignore the error.

My goodness.  I didn't think of the agent and the cache.  Hmpf.

kai
-- 
~/.signature: No such file or directory

Re: Backend writing

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> writes:
>> You couldn't have the nnmaildir spool NFS mounted and use the agent
>> when you don't have network connectivity, could you?
> 
> I suppose not.  I don't use caching or the agent; when I did this, I
> didn't know it would break things.  But people can use the agent on
> some servers and not others, right?  Then, at least, they could just
> run with the nnmaildir server closed whenever the NFS mount was
> unavailable.

Sure.

>> Do you also frob group info to renumber the marks?  When do you do
>> this?  Do you use nnchoke-request-update-info?
> 
> Yes.  Article numbers aren't stored on disk at all; -request-scan
> creates the mapping anew each time you close and reopen the server,
> and only keeps it in memory.  Then -request-update-info translates the
> stored marks into a whole new group info.

Hm, how are the permanent marks stored?  Doesn't maildir use integers
as (permanent) article identifiers?

Re: Backend writing

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> prj@po.cwru.edu (Paul Jarc) writes:
>> Article numbers aren't stored on disk at all; -request-scan creates
>> the mapping anew each time you close and reopen the server, and
>> only keeps it in memory.  Then -request-update-info translates the
>> stored marks into a whole new group info.
> 
> Hm, how are the permanent marks stored?  Doesn't maildir use integers
> as (permanent) article identifiers?

No.  Messages are delivered to separate files; to ensure a delivery
doesn't use the name of an existing message (or interfere with a
concurrent delivery), the filenames include a timestamp, the process
id of the delivering process, and the name of the host where the
delivering process is running.  (See
<URL:http://cr.yp.to/proto/maildir.html>.)  Example:
"996962912.20703.multivac.cwru.edu".  That message is considered to
have the 'tick mark if the file
".nnmaildir/marks/tick/996962912.20703.multivac.cwru.edu" exists in
the maildir.  (See
<URL:http://multivac.cwru.edu./nnmaildir/using/#marks>.)  The cr.yp.to
page describes another way of storing a more limited set of flags;
nnmaildir does not do things that way at present, but for better
interaction with other maildir readers, it might in the future (in
addition to the current way).


paul