self-contained nnml

self-contained nnml

[Simon Josefsson]

This patch makes nnml self-contained as far as marks are concerned.
Marks are stored in e.g. ~/Mail/misc/.marks.

Once the .marks file has been (automatically) created you should be
able to tar up a nnml server, move it to some other machine or back it
on tape up or whatever, and be able to plunk it back into a Gnus and
the marks should be discovered.  You could also be able to share a
nnml server between users, and have flags shared as well.

If you don't care about the above, this should NOT cause any problems.
But there may be bugs or design flaws.  Please report.  If you can't
even live with a small .marks file in the nnml directory and the
hopefully small performance penalty, you can disable it by toggling
`nnml-marks-is-evil'.  Erhm, perhaps that's not a good variable name,
who's Mark?  Maybe it should be Marx.  Nah, then it would have to be a
function rather than a variable. :-)

Known bug: If the .marks file is changed, you need to `g' (or `M-g')
before the new marks are noticed.  Simply entering the group is not
enough.  This is because `nnml-request-update-info' is not called when
the group is entered.  Maybe it should?  (FWIW, in nnimap,
`n-request-group' calls the equivalent of `n-request-update-info'.
But Gnus should do that.  Or not.)

Todo: Port to nnfolder etc.  It should be _really_ trivial.  Any
takers?

Caveat: I don't use nnml so I didn't test it.

2001-08-18  Simon Josefsson  <jas@extundo.com>

	Make nnml groups self-contained as far as marks are concerned.

	* nnml.el (nnml-request-delete-group): Delete marks file.
	(nnml-request-rename-group): Move marks file.
	(nnml-marks-file-name, nnml-marks-is-evil, nnml-marks): New server
	variables.
	(nnml-request-set-mark, nnml-request-update-info): New server
	functions.
	(nnml-save-marks, nnml-open-marks): New functions.

--- nnml.el.~6.9.~	Sat Aug 18 10:20:10 2001
+++ nnml.el	Sat Aug 18 13:12:56 2001
@@ -442,7 +442,8 @@
 	   (directory-files
 	    nnml-current-directory t
 	    (concat nnheader-numerical-short-files
-		    "\\|" (regexp-quote nnml-nov-file-name) "$")))
+		    "\\|" (regexp-quote nnml-nov-file-name) "$"
+		    "\\|" (regexp-quote nnml-marks-file-name) "$")))
 	  article)
       (while articles
 	(setq article (pop articles))
@@ -480,6 +481,10 @@
       (let ((overview (concat old-dir nnml-nov-file-name)))
 	(when (file-exists-p overview)
 	  (rename-file overview (concat new-dir nnml-nov-file-name))))
+      ;; Move .marks file.
+      (let ((marks (concat old-dir nnml-marks-file-name)))
+	(when (file-exists-p marks)
+	  (rename-file marks (concat new-dir nnml-marks-file-name))))
       (when (<= (length (directory-files old-dir)) 2)
 	(ignore-errors (delete-directory old-dir)))
       ;; That went ok, so we change the internal structures.
@@ -852,6 +857,85 @@
 	    force)
     (setq nnml-article-file-alist
 	  (nnheader-article-to-file-alist nnml-current-directory))))
+
+(defvoo nnml-marks-file-name ".marks")
+(defvoo nnml-marks-is-evil nil)
+(defvoo nnml-marks nil)
+
+(deffoo nnml-request-set-mark (group actions &optional server)
+  (nnml-possibly-change-directory group server)
+  (unless nnml-marks-is-evil
+    (nnml-open-marks group server)
+    (dolist (action actions)
+      (let ((range (nth 0 action))
+	    (what  (nth 1 action))
+	    (marks (nth 2 action)))
+	(assert (or (eq what 'add) (eq what 'del)) t
+		"Unknown request-set-mark action: %s" what)
+	(dolist (mark marks)
+	  (setq nnml-marks (nnimap-update-alist-soft
+			    mark
+			    (funcall (if (eq what 'add) 'gnus-range-add
+				       'gnus-remove-from-range)
+				     (cdr (assoc mark nnml-marks)) range)
+			    nnml-marks)))))
+    (nnml-save-marks group server)))
+
+(deffoo nnml-request-update-info (group info &optional server)
+  (nnml-possibly-change-directory group server)
+  (unless nnml-marks-is-evil
+    (nnml-open-marks group server)
+    ;; Update info using `nnml-marks'.
+    (mapcar (lambda (pred)
+	      (gnus-info-set-marks
+	       info
+	       (nnimap-update-alist-soft
+		(cdr pred)
+		(cdr (assq (cdr pred) nnml-marks))
+		(gnus-info-marks info))
+	       t))
+	    gnus-article-mark-lists)
+    (let ((seen (cdr (assq 'read nnml-marks))))
+      (gnus-info-set-read info
+			  (if (and (integerp (car seen))
+				   (null (cdr seen)))
+			      (list (cons (car seen) (car seen)))
+			    seen))))
+  info)
+
+(defun nnml-save-marks (group server)
+  (let ((file-name-coding-system nnmail-pathname-coding-system)
+	(file (expand-file-name nnml-marks-file-name
+				(nnmail-group-pathname group nnml-directory))))
+    (gnus-make-directory (file-name-directory file))
+    (with-temp-file file
+      (erase-buffer)
+      (princ nnml-marks (current-buffer))
+      (insert "\n"))))
+
+(defun nnml-open-marks (group server)
+  (with-temp-buffer
+    (let ((file (expand-file-name 
+		 nnml-marks-file-name 
+		 (nnmail-group-pathname group nnml-directory))))
+      (if (file-exists-p file)
+	  (setq nnml-marks (condition-case err
+			       (progn
+				 (nnheader-insert-file-contents file)
+				 (read (current-buffer)))
+			     (error (or (gnus-yes-or-no-p
+					 (format "Error reading nnml marks file %s (%s).  Continuing will use marks from .newsrc.eld.  Continue? " file err))
+					(error "Cannot read nnml marks file %s (%s)" file err)))))
+	;; User didn't have a .marks file.  Probably first time
+	;; user of the .marks stuff.  Bootstrap it from .newsrc.eld.
+	(let ((info (gnus-get-info
+		     (gnus-group-prefixed-name
+		      group
+		      (gnus-server-to-method (format "nnml:%s" server))))))
+	  (nnheader-message 6 "Boostrapping nnml marks...")
+	  (setq nnml-marks (gnus-info-marks info))
+	  (push (cons 'read (gnus-info-read info)) nnml-marks)
+	  (nnml-save-marks group server))))))
 
 (provide 'nnml)
 

Re: self-contained nnml

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Todo: Port to nnfolder etc.  It should be _really_ trivial.  Any
> takers?

Maybe some functions could be moved to nnmail.el, so that they are
easy to reuse.

I'd say: commit it to CVS and have people whine if it doesn't work.
This is development software, after all.

kai
-- 
~/.signature: No such file or directory

Re: self-contained nnml

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>> Todo: Port to nnfolder etc.  It should be _really_ trivial.  Any
>> takers?
>
> Maybe some functions could be moved to nnmail.el, so that they are
> easy to reuse.

Yes, that would be most of it.

> I'd say: commit it to CVS and have people whine if it doesn't work.
> This is development software, after all.

Done.

Re: self-contained nnml

[Daniel Pittman]

On Sat, 18 Aug 2001, Simon Josefsson wrote:
> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
> 
>>> Todo: Port to nnfolder etc.  It should be _really_ trivial.  Any
>>> takers?
>>
>> Maybe some functions could be moved to nnmail.el, so that they are
>> easy to reuse.
> 
> Yes, that would be most of it.
> 
>> I'd say: commit it to CVS and have people whine if it doesn't work.
>> This is development software, after all.
> 
> Done.

Whine. :/

I updated my Gnus to the CVS version at:
Built at Sun Aug 19 15:53:16 EST 2001

Recompile, yada, yada, then we boot XEmacs and it starts Gnus...

...the bootstrapping of the NNML marks works just fine.

Then I note that the summary buffer is ... odd. The majority of my
groups seem to have three articles in them now, which is odd, given the
standard set is > 2000 for most of them.

So, this is odd. To continue testing, I try entering the group where I
file the output of my log monitors and such. It's only got one message
in it, which is nice.

It, of course, reports one unread out of three articles. Trying to enter
the group results in the single message being marked read but *no*
indication of a summary buffer. Ho, hum.

So, I make the last test and 'g' in the groups buffer, which restores
the system message to life. Ugh.

So, back out the changes to the day before your commit of the
stand-alone NNML stuff and it's all good again.

*sigh*

So, Simon, suggestions on how I might go about debugging this?
        Daniel

-- 
If I knew for a certainty that some man was coming to my house with the
conscious intention of doing me good, I would run for my life.
        -- Henry David Thoreau

Re: self-contained nnml

[Simon Josefsson]

Daniel Pittman <daniel@rimspace.net> writes:

> Recompile, yada, yada, then we boot XEmacs and it starts Gnus...
>
> ...the bootstrapping of the NNML marks works just fine.

Does the .marks file contain correct marks for the nnml groups?

(If so, backup them now :-))

> Then I note that the summary buffer is ... odd. The majority of my
> groups seem to have three articles in them now, which is odd, given the
> standard set is > 2000 for most of them.

What does `G E' on the group show?  Does `g' or `M-g' change anything?

Does the .marks file contain exactly the same marks as `G E'?

> So, this is odd. To continue testing, I try entering the group where I
> file the output of my log monitors and such. It's only got one message
> in it, which is nice.
>
> It, of course, reports one unread out of three articles. Trying to enter
> the group results in the single message being marked read but *no*
> indication of a summary buffer. Ho, hum.

So, to get this right, it says 1/3 in the group buffer, you press RET
and you don't get a summary buffer and now it says 0/3.  Did I
understand you correctly?

What was the error message you got when trying to enter the group?

> So, I make the last test and 'g' in the groups buffer, which restores
> the system message to life. Ugh.

It says 1/3 in the group buffer again?  Entering the summary buffer
still doesn't work?

Do you have more than one nnml server, btw?

Could you edebug `nnml-request-update-info'?  Especially compare the
`info' on input with how it looks when the function is about to
terminate.

I committed some fixes, one of them may make a difference so please
try latest CVS first.

Weird, I can't trigger this at all.  It just works.  But I only have
one test nnml server.  Maybe there's a proble if you have more than
one nnml server.

Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Daniel Pittman]

On Sun, 19 Aug 2001, Simon Josefsson wrote:
> Daniel Pittman <daniel@rimspace.net> writes:
> 
>> Recompile, yada, yada, then we boot XEmacs and it starts Gnus...
>>
>> ...the bootstrapping of the NNML marks works just fine.
> 
> Does the .marks file contain correct marks for the nnml groups?
> (If so, backup them now :-))

Yeah, it's got the right values and all. So does my .newsrc.eld

What isn't right is that .newsrc.eld has the wrong number in the first
field of the record for each group, such that:

("Inbox.stats.mail" 3 ((1 . 34)) nil nil ((timestamp 15232 18635)))

The 3 there just after the group name seems to be what's causing the
problems; that's the number of items shown for each group in the
*Groups* buffer...

>> Then I note that the summary buffer is ... odd. The majority of my
>> groups seem to have three articles in them now, which is odd, given
>> the standard set is > 2000 for most of them.
> 
> What does `G E' on the group show?  Does `g' or `M-g' change anything?

G E shows what I expect, the correct information, save for the 3 as
shown in the record above.

`g' will bring all the groups back into sync with what .newsrc.eld shows
for total number of messages, 3. Every NNML shows this result when I hit
`g', as it applies the same operation to them all.

`M-g' makes the individual groups work correctly. They show the correct
number of unread messages, as they did before, as well as showing the
correct total number.

The groups also work after `M-g' is hit, but don't when `g' is hit. That
is, I can enter the group, modify it, etc, and it works correctly.

> Does the .marks file contain exactly the same marks as `G E'?

Yes, with the exception of adding `read' to the start of the read field,
which isn't present in .newsrc.eld, and *not* featuring the magic number
3 anywhere in the record.

I presume both these differences are by design, though, but include the
record for the group above for comparison:

((read (1 . 3)))

[...]

> So, to get this right, it says 1/3 in the group buffer, you press RET
> and you don't get a summary buffer and now it says 0/3.  Did I
> understand you correctly?

Yes. That's exactly the observed behavior.

> What was the error message you got when trying to enter the group?

Nothing. Not a single error, not a log message, not a tittle or jot.
I have `gnus-verbose' set to 7.

>> So, I make the last test and 'g' in the groups buffer, which restores
>> the system message to life. Ugh.
> 
> It says 1/3 in the group buffer again?  Entering the summary buffer
> still doesn't work?

Yes, that's correct.

> Do you have more than one nnml server, btw?

No.

> Could you edebug `nnml-request-update-info'?  Especially compare the
> `info' on input with how it looks when the function is about to
> terminate.

That's all correct. The issue is that when using `g', rather than `M-g',
this function gets 3 passed in as the, er, second value of the INFO
argument.

This is identical when it's called through the path of `g' or `M-g'. The
marks are correct in either case.

> I committed some fixes, one of them may make a difference so please
> try latest CVS first.

This is identical with a CVS update from this morning. :/

> Weird, I can't trigger this at all.  It just works.  But I only have
> one test nnml server.  Maybe there's a proble if you have more than
> one nnml server.

*sigh*  No, only the one server.

I think that my .newsrc.eld got corrupted somewhere along the way but,
due to some glitch in the system previously, I never saw that result.

This would work if something has changed recently so that NNML no longer
takes identical paths for `g' and `M-g' to find the total number of
articles...

Suggestions on how to fix this, anyone?

        Daniel

-- 
When the state is most corrupt, then the laws are most multiplied.
        -- Tacitus

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Daniel Pittman]

On Mon, 20 Aug 2001, Daniel Pittman wrote:
> On Sun, 19 Aug 2001, Simon Josefsson wrote:
>> Daniel Pittman <daniel@rimspace.net> writes:

[...]

>>> Then I note that the summary buffer is ... odd. The majority of my
>>> groups seem to have three articles in them now, which is odd, given
>>> the standard set is > 2000 for most of them.

Right. I did some more investigating using the tool "cvs update -D" :)

The problem started with the check-in of your NNML marks code, Simon. I
can't say for sure that it's the *cause* of it, but it's certainly the
change that causes it to be exhibited.

For what it's worth, using CVS at "2001-08-19 02:00:00" works, while
"2001-08-19 04:00:00" fails. I had to manually pull the nnimap file into
nnml to get it to work, because of a function not being bound otherwise,
but it showed the problem I describe.

The chance in question has this changelog:

,----
|  2001-08-18  Simon Josefsson  <jas@extundo.com>
|  
| 	Make nnml groups self-contained as far as marks are concerned.
| 
| 	* nnml.el (nnml-request-delete-group): Delete marks file.
| 	(nnml-request-rename-group): Move marks file.
| 	(nnml-marks-file-name, nnml-marks-is-evil, nnml-marks): New server
| 	variables.
| 	(nnml-request-set-mark, nnml-request-update-info): New server
| 	functions.
| 	(nnml-save-marks, nnml-open-marks): New functions.
`----

It looks like adding the `request-set-mark' or `request-update-info'
paths to the NNML code shows up the b0rken .newsrc.eld data, while the
path that does not use them gets it right some other way...

So, I hope that there is something that someone can suggest as a
solution for this. If it's manually hacking .newsrc.eld, I don't mind
that much...

...but it does occur to me that, given the .marks files are created
correctly, I might be able to blow away the .eld content and have it
recreated correctly, yes?

        Daniel

-- 
Waste no more time arguing what a good man should be.  Be one.
        -- Marcus Aurelius

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Simon Josefsson]

On Mon, 20 Aug 2001, Daniel Pittman wrote:

> On Sun, 19 Aug 2001, Simon Josefsson wrote:
> > Daniel Pittman <daniel@rimspace.net> writes:
> >
> >> Recompile, yada, yada, then we boot XEmacs and it starts Gnus...
> >>
> >> ...the bootstrapping of the NNML marks works just fine.
> >
> > Does the .marks file contain correct marks for the nnml groups?
> > (If so, backup them now :-))
>
> Yeah, it's got the right values and all. So does my .newsrc.eld
>
> What isn't right is that .newsrc.eld has the wrong number in the first
> field of the record for each group, such that:
>
> ("Inbox.stats.mail" 3 ((1 . 34)) nil nil ((timestamp 15232 18635)))
>
> The 3 there just after the group name seems to be what's causing the
> problems; that's the number of items shown for each group in the
> *Groups* buffer...

No, it's the group level.  It looks ok.

It looks as if your nnml server is your primary select method?  If so,
that may explain things, the bootstrap hack is ugly. Perhaps you could
evaluate the following two expressions:

(gnus-server-to-method "nnml:")
(gnus-group-prefixed-name "Inbox.stats.mail"
                          (gnus-server-to-method "nnml:"))
(gnus-get-info (gnus-group-prefixed-name "Inbox.stats.mail"
                                         (gnus-server-to-method "nnml:")))

If the third expression evaluate to nil, I believe we've found the bug.

> > Does the .marks file contain exactly the same marks as `G E'?
>
> Yes, with the exception of adding `read' to the start of the read field,
> which isn't present in .newsrc.eld, and *not* featuring the magic number
> 3 anywhere in the record.
>
> I presume both these differences are by design, though

Yup.

Hm.  Maybe it should copy the group level and group parameters as well.

> > Could you edebug `nnml-request-update-info'?  Especially compare the
> > `info' on input with how it looks when the function is about to
> > terminate.
>
> That's all correct. The issue is that when using `g', rather than `M-g',
> this function gets 3 passed in as the, er, second value of the INFO
> argument.

Err, the second value of info should probably be 3, it's the group level.
Could you post how they look both with `g' and `M-g'?

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Daniel Pittman]

On Mon, 20 Aug 2001, Simon Josefsson wrote:
> On Mon, 20 Aug 2001, Daniel Pittman wrote:
>> On Sun, 19 Aug 2001, Simon Josefsson wrote:
>> > Daniel Pittman <daniel@rimspace.net> writes:
>> >
>> >> Recompile, yada, yada, then we boot XEmacs and it starts Gnus...
>> >>
>> >> ...the bootstrapping of the NNML marks works just fine.
>> >
>> > Does the .marks file contain correct marks for the nnml groups?
>> > (If so, backup them now :-))
>>
>> Yeah, it's got the right values and all. So does my .newsrc.eld
>>
>> What isn't right is that .newsrc.eld has the wrong number in the
>> first field of the record for each group, such that:
>>
>> ("Inbox.stats.mail" 3 ((1 . 34)) nil nil ((timestamp 15232 18635)))
>>
>> The 3 there just after the group name seems to be what's causing the
>> problems; that's the number of items shown for each group in the
>> *Groups* buffer...
> 
> No, it's the group level.  It looks ok.

Ah. Here was I feeling all clever-like, too. :)

> It looks as if your nnml server is your primary select method?  

Yes, it is. :)

> If so, that may explain things, the bootstrap hack is ugly. Perhaps
> you could evaluate the following two expressions:
> 
> (gnus-server-to-method "nnml:")
> (gnus-group-prefixed-name "Inbox.stats.mail"
>                           (gnus-server-to-method "nnml:"))
> (gnus-get-info (gnus-group-prefixed-name "Inbox.stats.mail"
>                                          (gnus-server-to-method
>                                          "nnml:")))

Er, I went and updated my Gnus to the latest CVS version...

...and it works now. *sigh*

Anyway, those three evaluate as:

,----
| (gnus-server-to-method "nnml:")
| => (nnml "")
| 
| (gnus-group-prefixed-name "Inbox.stats.mail"
|                           (gnus-server-to-method "nnml:"))
| => "Inbox.stats.mail"
| 
| (gnus-get-info (gnus-group-prefixed-name "Inbox.stats.mail"
|                                          (gnus-server-to-method
| 					  "nnml:")))
| => ("Inbox.stats.mail" 3 ((1 . 125)) ((seen (35 . 52) (57 . 125))) nil ((timestamp 15233 60165)))
`----

>> > Does the .marks file contain exactly the same marks as `G E'?
>>
>> Yes, with the exception of adding `read' to the start of the read
>> field, which isn't present in .newsrc.eld, and *not* featuring the
>> magic number 3 anywhere in the record.
>>
>> I presume both these differences are by design, though
> 
> Yup.
> 
> Hm. Maybe it should copy the group level and group parameters as well.

Well, group parameters such as the `to-list' and `to-address', I would
expect to transfer and back-up correctly. If I stopped backing up the
.eld file and they got lost, I would go find a corner and cry or
something. :)

>> > Could you edebug `nnml-request-update-info'? Especially compare the
>> > `info' on input with how it looks when the function is about to
>> > terminate.
>>
>> That's all correct. The issue is that when using `g', rather than
>> `M-g', this function gets 3 passed in as the, er, second value of the
>> INFO argument.
> 
> Err, the second value of info should probably be 3, it's the group
> level. Could you post how they look both with `g' and `M-g'?

Now I know. Anyway, it works again. I can go back to the broken version
if you want, but that's not high on my priority list...

        Daniel

-- 
The fundamental delusion of humanity is to suppose
that I am here and you are there.
        -- Yasutani Roshi

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Simon Josefsson]

On Tue, 21 Aug 2001, Daniel Pittman wrote:

> Er, I went and updated my Gnus to the latest CVS version...
>
> ...and it works now. *sigh*

Oh.  Good.

> | (gnus-get-info (gnus-group-prefixed-name "Inbox.stats.mail"
> |                                          (gnus-server-to-method
> | 					  "nnml:")))
> | => ("Inbox.stats.mail" 3 ((1 . 125)) ((seen (35 . 52) (57 . 125))) nil ((timestamp 15233 60165)))

Fine, there is no problem with primary nnml groups then.

> Well, group parameters such as the `to-list' and `to-address', I would
> expect to transfer and back-up correctly. If I stopped backing up the
> .eld file and they got lost, I would go find a corner and cry or
> something. :)

Well, one point of the `.marks' file is that you don't need to backup
.newsrc.eld at all, so I think it would be good to put more backend
state in it.

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> On Tue, 21 Aug 2001, Daniel Pittman wrote:
>> Well, group parameters such as the `to-list' and `to-address', I would
>> expect to transfer and back-up correctly. If I stopped backing up the
>> .eld file and they got lost, I would go find a corner and cry or
>> something. :)
> 
> Well, one point of the `.marks' file is that you don't need to backup
> .newsrc.eld at all, so I think it would be good to put more backend
> state in it.

The problem here is that there is no analogue of -request-set-mark for
group parameters.  If Gnus changes marks, then the backend finds out
via -request-set-mark.  So the next time Gnus asks for marks with
-request-update-info, the backend knows that its marks are at least as
accurate as Gnus's current marks, and it can assume any differences
are an inaccuracy in Gnus's marks and correct them.  But with group
parameters, the backend has no way to decide whether Gnus's current
parameters or the backend's stored parameters are more accurate.
-request-update-info is the only way the backend ever gets parameters
from Gnus, but also the only way it has to change Gnus's parameters.


paul

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

>> Well, one point of the `.marks' file is that you don't need to backup
>> .newsrc.eld at all, so I think it would be good to put more backend
>> state in it.
>
> The problem here is that there is no analogue of -request-set-mark for
> group parameters.  If Gnus changes marks, then the backend finds out
> via -request-set-mark.  So the next time Gnus asks for marks with
> -request-update-info, the backend knows that its marks are at least as
> accurate as Gnus's current marks, and it can assume any differences
> are an inaccuracy in Gnus's marks and correct them.  But with group
> parameters, the backend has no way to decide whether Gnus's current
> parameters or the backend's stored parameters are more accurate.
> -request-update-info is the only way the backend ever gets parameters
> from Gnus, but also the only way it has to change Gnus's parameters.

Right.  Should this change?  It wouldn't be difficult to add a backend
interface to update group parameters, group levels etc.

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Daniel Pittman]

On Tue, 21 Aug 2001, Simon Josefsson wrote:
> On Tue, 21 Aug 2001, Daniel Pittman wrote:
> 
>> Er, I went and updated my Gnus to the latest CVS version...
>>
>> ...and it works now. *sigh*
> 
> Oh.  Good.

Er, yes. Sorry. I should have been more positive about it, but I always
get nervous when a bug just stops without an obvious cause. Anyway,
thanks for your help with this, it /is/ much appreciated.

[...]

>> Well, group parameters such as the `to-list' and `to-address', I
>> would expect to transfer and back-up correctly. If I stopped backing
>> up the .eld file and they got lost, I would go find a corner and cry
>> or something. :)
> 
> Well, one point of the `.marks' file is that you don't need to backup
> .newsrc.eld at all, so I think it would be good to put more backend
> state in it.

Quite. Ah, easy to back up. Now all I need do is get the 1.2GB of it
backed up. :)

Anyway, this is a good thing. Thanks for the effort.

        Daniel

-- 
We propose to burn the academic libraries, because Theology 
is only fanaticism, History is lies, Philosophy is dreams, 
and Science is unnecessary.
        -- The Commune of Marseilles

Re: Problems with .newsrc.eld (aws: Re: self-contained nnml)

[Daniel Pittman]

On Tue, 21 Aug 2001, Simon Josefsson wrote:

[...]

> Right.  Should this change?  It wouldn't be difficult to add a backend
> interface to update group parameters, group levels etc.

Yes. Imagine the ability to connect from a Gnus on any machine in the
world to your IMAP host... and have it remember all your settings. :)

More importantly to me, though, the ability to have a self contained
NNML setup would be good -- and the group parameters are part of that
setup.

        Daniel

-- 
> What should I look for in a good bird bath?
And in response, thus spake the Oracle:
} In a good bird bath? I'd expect to find birds.
} In a bad bird bath, tarantulas.

Re: self-contained nnml

[Rob Browning]

Simon Josefsson <jas@extundo.com> writes:

> Once the .marks file has been (automatically) created you should be
> able to tar up a nnml server, move it to some other machine or back
> it on tape up or whatever, and be able to plunk it back into a Gnus
> and the marks should be discovered.  You could also be able to share
> a nnml server between users, and have flags shared as well.

So does this also apply for a group within a server?  i.e. could I "B
m" a bunch of articles from nnml+rlb:ding-gnus to
nnml+rlb:ding-gnus-2000 and then tar up and remove the ding-gnus-2000
directory without seriously confusing gnus?  Similarly could I restore
the group by dropping it back into place in the filesystem?

BTW, this is something I've wanted for a *long* time, so thanks for
the work.  I'm was just in the process of trying to see if I could
switch to nnmaildir for this very reason. so now I guess I'll have to
decide between two backends :>

One nice thing about nnmaildir is that it's supposed to automatically
see groups that appear/disappear from the filesystem without getting
confused.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

Re: self-contained nnml

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> So does this also apply for a group within a server?  i.e. could I "B
> m" a bunch of articles from nnml+rlb:ding-gnus to
> nnml+rlb:ding-gnus-2000 and then tar up and remove the ding-gnus-2000
> directory without seriously confusing gnus?  Similarly could I restore
> the group by dropping it back into place in the filesystem?

OK.  After looking at what happens (now that I've upgraded to Oort), I
can see that the .marks are per-group.  It also mentions that it's
per-group in GNUS-NEWS.  Now I'll just have to see how happy Gnus is
with removals and restores.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

Re: self-contained nnml

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> Now I'll just have to see how happy Gnus is with removals and
> restores.

OK, removal was easy; restore took a little hand-editing.

I tarred up a group directory, and then killed it via "C-u G DEL".
That got rid of the group quite nicely.

However I had a little trouble figuring out how to get it back.  Just
untarring the group in the right place didn't work because it still
wasn't in the active file.  Hand-editing the active file fixed that,
but is there a better way?

Thanks

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

Re: self-contained nnml

[ShengHuo ZHU]

Rob Browning <rlb@defaultvalue.org> writes:

> Rob Browning <rlb@defaultvalue.org> writes:
>
>> Now I'll just have to see how happy Gnus is with removals and
>> restores.
>
> OK, removal was easy; restore took a little hand-editing.
>
> I tarred up a group directory, and then killed it via "C-u G DEL".
> That got rid of the group quite nicely.
>
> However I had a little trouble figuring out how to get it back.  Just
> untarring the group in the right place didn't work because it still
> wasn't in the active file.  Hand-editing the active file fixed that,
> but is there a better way?

How about `G m ding-gnus RET nnml+rlb: RET'

ShengHuo

Re: self-contained nnml

[Simon Josefsson]

Rob Browning <rlb@defaultvalue.org> writes:

>> So does this also apply for a group within a server?  i.e. could I "B
>> m" a bunch of articles from nnml+rlb:ding-gnus to
>> nnml+rlb:ding-gnus-2000 and then tar up and remove the ding-gnus-2000
>> directory without seriously confusing gnus?  Similarly could I restore
>> the group by dropping it back into place in the filesystem?
>
> OK.  After looking at what happens (now that I've upgraded to Oort), I
> can see that the .marks are per-group.  It also mentions that it's
> per-group in GNUS-NEWS.  Now I'll just have to see how happy Gnus is
> with removals and restores.

Remember that the active file is per server. I think nnml
automatically regenerates active info for new groups though.  Just
don't untar it in directory that already have a active entry.  Please
do test this, I'm not aware of anyone that actually _used_ it.

The per-server active file could go into a per-group .active file, but
we'd might as well remove the active file completely. Nnml could store
this info in a lisp variable.  To populate it when Gnus boots it could
look at first/last line of .overview at startup.  I don't think the
active file contains any information not efficiently available by
other methods.  Then we would have self-contained nnml groups.

Group level and group parameters are not store stored in the nnml
directory though.

Re: self-contained nnml

[Rob Browning]

Simon Josefsson <jas@extundo.com> writes:

> Remember that the active file is per server. I think nnml
> automatically regenerates active info for new groups though.

Well, it doesn't seem to notice new directories.  So just untarring
the directory isn't sufficient.  I'm going to try ShengHuo's
suggestion of using "G m", but I'm not sure if that will work since if
I do "G m" before I untar the directory, I'll get an empty group with
an incorrect active listing, but if I do it after, "G m" might be
confused by the pre-existing directory.  What I really need is
something like a nnml-scan-for-new-groups, or an
nnml-notice-new-group.

Actually, what would be more appropriate, and perhaps useful to
others, would be gnus-group-archive-group and gnus-group-restore-group
commands.  Each one would prompt you for the destination, and the
group contents would be saved or restored as appropriate.  The format
could be a compressed tarfile containing the messages and the
metadata.

> Group level and group parameters are not store stored in the nnml
> directory though.

If we wanted to have truly stand-alone nnml groups, then I suppose
they should be.  However, if they were handled properly by the
archive/restore functions, then that would probably be good enough for
most purposes.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

Re: self-contained nnml

[Simon Josefsson]

Rob Browning <rlb@defaultvalue.org> writes:

>> Remember that the active file is per server. I think nnml
>> automatically regenerates active info for new groups though.
>
> Well, it doesn't seem to notice new directories.  So just untarring
> the directory isn't sufficient.  I'm going to try ShengHuo's
> suggestion of using "G m", but I'm not sure if that will work since if
> I do "G m" before I untar the directory, I'll get an empty group with
> an incorrect active listing, but if I do it after, "G m" might be
> confused by the pre-existing directory.

No, `G m' is fine.  It's just a matter of making Gnus notice the new
directory.

> Actually, what would be more appropriate, and perhaps useful to
> others, would be gnus-group-archive-group and gnus-group-restore-group
> commands.  Each one would prompt you for the destination, and the
> group contents would be saved or restored as appropriate.  The format
> could be a compressed tarfile containing the messages and the
> metadata.

Having it happen automatically and without starting Gnus is easier, I
think, and it also allows people to share marks in nnml/nnfolder's
with each other.  So the gnus-group-archive-group always happen, and
the gnus-group-restore-group is `G m'.

Re: self-contained nnml

[Rob Browning]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> How about `G m ding-gnus RET nnml+rlb: RET'

Actually it was nnml:rlb, but other than that, it worked great.
Thanks.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD

Re: self-contained nnml

[Rob Browning]

Simon Josefsson <jas@extundo.com> writes:

> No, `G m' is fine.  It's just a matter of making Gnus notice the new
> directory.

Seemed to work well.  Thanks.

> Having it happen automatically and without starting Gnus is easier,
> I think, and it also allows people to share marks in nnml/nnfolder's
> with each other.  So the gnus-group-archive-group always happen, and
> the gnus-group-restore-group is `G m'.

OK, though for this to be a "complete" archive, the group parameters
would need to be moved to the group directories too.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD