Gnus and resilience to datafile corruption...

Gnus and resilience to datafile corruption...

[Daniel Pittman]

After losing my .newsrc.eld in a wonderful file-system corruption a
little while ago I need to go and reestablish a whole lot of group
properties for my nnml and nntp groups.

So, several things have become obvious to me through this:

Firstly, Gnus is *extremely* resilient for a piece of software. I had a
large number of corrupted metadata files in the system and, between the
.marks and `nnml-generate-nov-databases' function I recovered pretty
much everything.

What I didn't get back and would like to are:

* group properties

It would seem reasonable to create a .properties file, similar to the
.marks, that stored individual group properties in the nnml groups. This
would allow a second level of safety -- if they died, .newsrc.eld would
have the same information. The reverse, obviously, is also true.

Oh, and WIBNI you could modify group properties from within the summary
buffer? I keep needing to do that to reestablish the to-address property
for mailing lists.


* nntp group marks and properties

These are only stored in the .newsrc.eld file, resulting in my losing
all the information about them. WIBNI they were stored in the nnml
redundant and split style somewhere?

Ditto the properties, of course. :)


On the whole, though, I am amazed at the ease with which I recovered my
1.65 gigabytes of email and news configuration. Thank you very much to
everyone who has worked to make this such reliable software.

        Daniel

-- 
It is rather ridiculous to ask a man just about to be boiled in a pot and
eaten, at a purely religious feast, why he does not regard all religions as
equally friendly and fraternal.
        -- _The Everlasting Man_, 1925

Re: Gnus and resilience to datafile corruption...

[Paul Jarc]

Daniel Pittman <daniel@rimspace.net> wrote:
> It would seem reasonable to create a .properties file, similar to the
> .marks, that stored individual group properties in the nnml groups.

This could be done, sort of, but not very well without an enhancement
to the backend interface.  Backends can update Gnus's copy of the
group parameters, but backends aren't ever notified when the user
changes parameters a la -request-set-mark for marks, so they can't
tell whether their copy or Gnus's copy is more up to date.

> * nntp group marks and properties
>
> These are only stored in the .newsrc.eld file, resulting in my losing
> all the information about them. WIBNI they were stored in the nnml
> redundant and split style somewhere?

If you want redundancy, keep backups.  Duplicating information storage
is pointless when Gnus has no way of knowing that one form of storage
is more reliable than another.  It's just one more thing that can
break.

Note that you can specify all your group properties in the
gnus-parameters variable, if your .gnus is more reliable than your
.newsrc.eld.


paul

Re: Gnus and resilience to datafile corruption...

[Daniel Pittman]

On Sat, 04 May 2002, Paul Jarc wrote:
> Daniel Pittman <daniel@rimspace.net> wrote:
>> It would seem reasonable to create a .properties file, similar to the
>> .marks, that stored individual group properties in the nnml groups.
> 
> This could be done, sort of, but not very well without an enhancement
> to the backend interface.  Backends can update Gnus's copy of the
> group parameters, but backends aren't ever notified when the user
> changes parameters a la -request-set-mark for marks, so they can't
> tell whether their copy or Gnus's copy is more up to date.

Hrm. This should be practical to implement, yes? It would require
extending the interface to support a nnfoo `-request-set-properties'
method, then call that on the backend when it's implemented.

That should be backward compatible, yes? It also sounds practical to do,
so I will look at writing it up.

>> * nntp group marks and properties
>>
>> These are only stored in the .newsrc.eld file, resulting in my losing
>> all the information about them. WIBNI they were stored in the nnml
>> redundant and split style somewhere?
> 
> If you want redundancy, keep backups.  

I do, but not quite so up to date as the application specific ones that
NNML maintains. :)

> Duplicating information storage is pointless when Gnus has no way of
> knowing that one form of storage is more reliable than another. It's
> just one more thing that can break.

As my recent experience shows, there is some value in this duplication.
Gnus may not know the difference in reliability but /I/ do. I lost
vastly less information in NNML than in other backends as a result of
it's more decentralized organization.

In the end I wouldn't care if .newsrc.eld didn't exist -- confining the
data loss from filesystem corruption to individual groups rather than
*all* groups would be better as far as I am concerned.


The NNML technique for storing information has also proved more reliable
in the face of minor disk corruption, from bad blocks, as well as rapid
corruption from broken kernels.

> Note that you can specify all your group properties in the
> gnus-parameters variable, if your .gnus is more reliable than your
> .newsrc.eld.

It's not, because it's a different single file. It would have lived
through the filesystem corruption with no data loss but the bad block
would have killed it if it happened to be in that file.

Central storage is, by it's nature, less reliable than distributed
storage.

        Daniel

-- 
If you feel you have both feet planted on level ground,
then the university has failed you.
        -- Robert F. Goheen

Re: Gnus and resilience to datafile corruption...

[Paul Jarc]

Daniel Pittman <daniel@rimspace.net> wrote:
> It would require extending the interface to support a nnfoo
> `-request-set-properties' method, then call that on the backend when
> it's implemented.
>
> That should be backward compatible, yes?

Right.

>> If you want redundancy, keep backups.
>
> I do, but not quite so up to date as the application specific ones that
> NNML maintains. :)

(add-hook 'gnus-save-newsrc-hook
  (lambda ()
    (copy-file "~/.newsrc.eld" "~/.newsrc.eld.backup" 'ok-if-already-exists)))
Or split it up among several files, if you like.

> In the end I wouldn't care if .newsrc.eld didn't exist

I think that's a popular goal.


paul