marks in gnus / uw imapd: what marks / how to set?

marks in gnus / uw imapd: what marks / how to set?

[John Owens]

I'd like to migrate my mail from another mailer (VM) (which stores its
files in mbox format) and maintain as much information from VM's state
as possible, including "marks" (read, new, forwarded, replied, etc.).
Currently I am thinking I need to write a perl script to convert VM
mail to gnus-nnimap mail so that I can maintain these marks. (Other
suggestions gratefully accepted.)

VM seems to store all marks in a special header (X-VM-v5-data). VM
uses the following flags, which I'd like to move to gnus.

vm-new-flag vm-unread-flag vm-deleted-flag vm-filed-flag 
vm-replied-flag vm-written-flag vm-forwarded-flag vm-edited-flag 
vm-redistributed-flag 

I'm trying to figure out how gnus in conjunction with UW imapd stores
the same marks.

As far as I can tell, "Read" and "Old" flags are kept in Status and
"Deleted", "Flagged", "Answered" and "Draft" are in X-Status. So 

Status: RO
X-Status: DFAT

is the maximum amount of flags stored here?

Gnus seems to keep other keywords. Among them is a forwarded flag,
which is kept in X-Keywords as "gnus-forward". I haven't got anything
else to appear in this header yet. But there are other gnus flags:

gnus-replied mark - I presume this corresponds to Answered.

gnus-recent mark - I presume this corresponds to not-Old, but I don't
know how this works with gnus-unseen-mark.

gnus-cached-mark - No idea.

gnus-saved-mark - No idea. (Hopefully it is triggered when I copy a
message from a folder to another folder.)

Other flags? Distinction between "new" and "unread"? "edited"?
"redistributed"? 

Any suggestions as to how to move these flags over would be
appreciated. I figure perl is my best bet? Suggestions of a perl
module that lets me parse mail a message at a time from mbox format,
and mess with headers too, would be great.

JDO

Re: marks in gnus / uw imapd: what marks / how to set?

[Simon Josefsson]

John Owens <jowens@ece.ucdavis.edu> writes:

> On another note, then, let's say I want to do a one-to-one conversion
> of VM folders to nnimap folders. What would be the procedure for doing
> this if I don't want to do anything other than the one-to-one mapping
> (i.e. I don't want to run through new splitting rules)? Any way to do
> this for each of ~200 folders efficiently?

I recall someone else asking this recently, but I don't recall an
efficient solution.

With the hacked nndoc.el, you can G f each file, M-g it, enter it and
process mark all articles and copy them into a nnimap group.
Disabling threading helps preserve article order.  It is probably
possible to set up a keyboard macro or write some elisp to automate
it some more.

Re: marks in gnus / uw imapd: what marks / how to set?

[Simon Josefsson]

NB! Re-added CC to ding.

John Owens <jowens@ece.ucdavis.edu> writes:

> And here comes the folder (attached below). The summary follows:
>
> ->  1    R  Kyle Jones        Nov 18   12/412   ""Invalid search bound (wrong side of point)" VM error on MIME send?"
>     2    R  Kyle Jones        Nov 18   23/720   "Re: "Invalid search bound (wrong side of point)" VM error on MIME send?"
>     3    Z  Yahoo! Groups     Jan 24   47/1695  "Unable to deliver your message"
>     4       Matthew Eldridge  Feb  7   14/439   "good VM hack"
>     5   F E Matthew Eldridge  Jun 26   27/721   "Re: widths of thesis figures"
>     6       Simon Josefsson   Oct 15   24/1039  "Re: [jowens@graphics.stanford.edu: Re: starttls: windows binary?]"
>     7       Kai Großjohann    Dec 11   14/358   "Re: remote emacs"
>     8    R  Hans-Peter Binder Feb 18   22/689   "Re: still trying ..."
>     9    B  Hans-Peter Binder Feb 18   43/1582  "Re: still trying ..."

With the attached patch, after "G f simon RET", M-g, up-arrow, RET, I
get the following summary buffer in Gnus on the same group:

 A [  13: Kyle Jones             ] "Invalid search bound (wrong side of point)" VM error on MIME send?
 A     <  24: Kyle Jones             > 
 . [  15: Matthew Eldridge       ] good VM hack
 . [  28: Matthew Eldridge       ] Re: widths of thesis figures
 . [  25: -> John Owens          ] Re: [jowens@graphics.stanford.edu: Re: starttls: windows binary?]
 F [  48: Yahoo! Groups          ] Unable to deliver your message
 . [  15: Kai Großjohann         ] Re: remote emacs
 A [  23: Hans-Peter Binder      ] Re: still trying ...
 .     <  44: Hans-Peter Binder      > 

As you can see, the patch only handles read, reply and forward marks,
but I wasn't sure how to map the others.

(The M-g part shouldn't be necessary, will fix that.)

Does it work for you?

--- nndoc.el.~6.21.~	Sun May 11 22:41:06 2003
+++ nndoc.el	Mon Aug  4 22:51:08 2003
@@ -77,7 +77,8 @@
      (body-end-function . nndoc-rnews-body-end))
     (mbox
      (article-begin-function . nndoc-mbox-article-begin)
-     (body-end-function . nndoc-mbox-body-end))
+     (body-end-function . nndoc-mbox-body-end)
+     (header-hook . nndoc-mbox-header-hook))
     (babyl
      (article-begin . "\^_\^L *\n")
      (body-end . "\^_")
@@ -170,6 +171,7 @@
 (defvoo nndoc-body-begin-function nil)
 (defvoo nndoc-head-begin-function nil)
 (defvoo nndoc-body-end nil)
+(defvoo nndoc-header-hook nil)
 ;; nndoc-dissection-alist is a list of sublists.  Each sublist holds the
 ;; following items.  ARTICLE acts as the association key and is an ordinal
 ;; starting at 1.  HEAD-BEGIN [0], HEAD-END [1], BODY-BEGIN [2] and BODY-END
@@ -185,6 +187,8 @@
 (defvoo nndoc-article-begin-function nil)
 (defvoo nndoc-generate-article-function nil)
 (defvoo nndoc-dissection-function nil)
+(defvoo nndoc-update-info-read-data nil)
+(defvoo nndoc-update-info-mark-data nil)
 
 (defvoo nndoc-status-string "")
 (defvoo nndoc-group-alist nil)
@@ -269,6 +273,12 @@
      (t
       (nnheader-insert "211 %d %d %d %s\n" number 1 number group)))))
 
+(deffoo nndoc-request-update-info (group info &optional server)
+  (when (nndoc-possibly-change-buffer group server)
+    (gnus-info-set-read info nndoc-update-info-read-data)
+    (gnus-info-set-marks info nndoc-update-info-mark-data)
+    info))
+
 (deffoo nndoc-request-type (group &optional article)
   (cond ((not article) 'unknown)
 	(nndoc-post-type nndoc-post-type)
@@ -362,7 +372,7 @@
 		nndoc-generate-head-function nndoc-body-begin-function
 		nndoc-head-begin-function
 		nndoc-generate-article-function
-		nndoc-dissection-function)))
+		nndoc-dissection-function nndoc-header-hook)))
     (while vars
       (set (pop vars) nil)))
   (let (defs)
@@ -427,6 +437,46 @@
 		      (looking-at message-unix-mail-delimiter)))))
       (goto-char len))))
 
+(defun nndoc-mbox-header-hook (article start end)
+  (save-excursion
+    (let (data attributes new-flag unread-flag deleted-flag
+	       filed-flag replied-flag written-flag
+	       forwarded-flag edited-flag redistributed-flag)
+      (goto-char start)
+      (when (and (re-search-forward (concat "^X-VM-v5-Data: ") end t)
+		 (goto-char (match-end 0))
+		 (setq data (read (current-buffer)))
+		 (setq attributes (car-safe data))
+		 (arrayp attributes)
+		 (>= (length attributes) 8))
+	(setq new-flag (aref attributes 0)
+	      unread-flag (aref attributes 1)
+	      deleted-flag (aref attributes 2)
+	      filed-flag (aref attributes 3)
+	      replied-flag (aref attributes 4)
+	      written-flag (aref attributes 5)
+	      forwarded-flag (aref attributes 6)
+	      edited-flag (aref attributes 7)
+	      redistributed-flag (aref attributes 8))
+	(if (not unread-flag)
+	    (gnus-add-to-range nndoc-update-info-read-data (list article)))
+	(if replied-flag
+	    (setq nndoc-update-info-mark-data
+		  (gnus-update-alist-soft
+		   'reply
+		   (gnus-add-to-range
+		    (cdr (assoc 'reply nndoc-update-info-mark-data))
+		    (list article))
+		   nndoc-update-info-mark-data)))
+	(if forwarded-flag
+	    (setq nndoc-update-info-mark-data
+		  (gnus-update-alist-soft
+		   'forward
+		   (gnus-add-to-range
+		    (cdr (assoc 'forward nndoc-update-info-mark-data))
+		    (list article))
+		   nndoc-update-info-mark-data)))))))
+
 (defun nndoc-mmdf-type-p ()
   (when (looking-at "\^A\^A\^A\^A$")
     t))
@@ -801,6 +851,8 @@
 	    (setq head-begin (point))
 	    (nndoc-search (or nndoc-head-end "^$"))
 	    (setq head-end (point))
+	    (if nndoc-header-hook
+		(funcall nndoc-header-hook (1+ i) head-begin head-end))
 	    (if nndoc-body-begin-function
 		(funcall nndoc-body-begin-function)
 	      (nndoc-search (or nndoc-body-begin "^\n")))

Re: marks in gnus / uw imapd: what marks / how to set?

[John Owens]

Simon Josefsson writes:
    John Owens <jowens@ece.ucdavis.edu> writes:
    
    > Simon Josefsson writes:
    >     The Gnusish way to do this would be to make Gnus read VM mailboxes
    >     properly.  I wonder if this is difficult.  Is the VM mailbox format
    >     described anywhere?  Maybe it is just babyl and some external marks
    >     stuff?  I'm not really familiar with VM.
    >
    > It's not a hard format. My elisp fu is weak, but I would certainly be
    > willing to help with the transition. It's a simple format: every
    > message has the following header:
    >
    > X-VM-v5-Data: ([nil nil nil nil t nil nil nil nil]
    > 	["2009" "Monday" "4" "August" "2003" "09:23:04" "+0200" "Simon Josefsson" "jas@extundo.com" "<iluu18xrinb.fsf@latte.josefsson.org>" "58" "Re: marks in gnus / uw imapd: what marks / how to set?" "^From:" nil nil "8" nil nil (number " " mark "  R  Simon Josefsson   Aug  4   58/2009  " thread-indent "\"Re: marks in gnus / uw imapd: what marks / how to set?\"\n") nil nil nil nil nil nil]
    > 	nil)
    
    What format is the file?  mbox, babyl, etc?  It sounds pretty simple
    to add a nnvm, if the files is in a already supported format.  Simply
    copy nnmbox or nnbabyl, remove some cruft and add a
    nnvm-request-update-info that frob Gnus's flags using the headers.
    
It's just mbox, with this one magic header per message (X-VM-v5-Data)
that contains the marks, and a bunch of other magic headers on only
the first message in the folder (X-VM-Message-Order,
X-VM-Summary-Format, X-VM-Labels, X-VM-VHeader, X-VM-Last-Modified,
X-VM-IMAP-Retrieved, X-VM-POP-Retrieved, and X-VM-Bookmark), none of
which make any difference for this purpose and can be safely ignored
(and probably should be deleted).

    Hm, if you can create a VM mailbox with say 5 messages with some
    different flags and make it available, I can attempt to create a nnvm.
    
I'll send you a file separately, and if anyone else is interested in
looking at such a file, please let me know and I'll send it to you
too. I'm sure it's a simple (< 1 hour) task for someone familiar with
creating a nn.

On another note, then, let's say I want to do a one-to-one conversion
of VM folders to nnimap folders. What would be the procedure for doing
this if I don't want to do anything other than the one-to-one mapping
(i.e. I don't want to run through new splitting rules)? Any way to do
this for each of ~200 folders efficiently?

JDO

Re: marks in gnus / uw imapd: what marks / how to set?

[Simon Josefsson]

John Owens <jowens@ece.ucdavis.edu> writes:

> Simon Josefsson writes:
>     The Gnusish way to do this would be to make Gnus read VM mailboxes
>     properly.  I wonder if this is difficult.  Is the VM mailbox format
>     described anywhere?  Maybe it is just babyl and some external marks
>     stuff?  I'm not really familiar with VM.
>
> It's not a hard format. My elisp fu is weak, but I would certainly be
> willing to help with the transition. It's a simple format: every
> message has the following header:
>
> X-VM-v5-Data: ([nil nil nil nil t nil nil nil nil]
> 	["2009" "Monday" "4" "August" "2003" "09:23:04" "+0200" "Simon Josefsson" "jas@extundo.com" "<iluu18xrinb.fsf@latte.josefsson.org>" "58" "Re: marks in gnus / uw imapd: what marks / how to set?" "^From:" nil nil "8" nil nil (number " " mark "  R  Simon Josefsson   Aug  4   58/2009  " thread-indent "\"Re: marks in gnus / uw imapd: what marks / how to set?\"\n") nil nil nil nil nil nil]
> 	nil)

What format is the file?  mbox, babyl, etc?  It sounds pretty simple
to add a nnvm, if the files is in a already supported format.  Simply
copy nnmbox or nnbabyl, remove some cruft and add a
nnvm-request-update-info that frob Gnus's flags using the headers.

Hm, if you can create a VM mailbox with say 5 messages with some
different flags and make it available, I can attempt to create a nnvm.

If someone knows a reason this hasn't been done before, I'm all ears.

Re: marks in gnus / uw imapd: what marks / how to set?

[John Owens]

Simon Josefsson writes:
    The Gnusish way to do this would be to make Gnus read VM mailboxes
    properly.  I wonder if this is difficult.  Is the VM mailbox format
    described anywhere?  Maybe it is just babyl and some external marks
    stuff?  I'm not really familiar with VM.

It's not a hard format. My elisp fu is weak, but I would certainly be
willing to help with the transition. It's a simple format: every
message has the following header:

X-VM-v5-Data: ([nil nil nil nil t nil nil nil nil]
	["2009" "Monday" "4" "August" "2003" "09:23:04" "+0200" "Simon Josefsson" "jas@extundo.com" "<iluu18xrinb.fsf@latte.josefsson.org>" "58" "Re: marks in gnus / uw imapd: what marks / how to set?" "^From:" nil nil "8" nil nil (number " " mark "  R  Simon Josefsson   Aug  4   58/2009  " thread-indent "\"Re: marks in gnus / uw imapd: what marks / how to set?\"\n") nil nil nil nil nil nil]
	nil)

which are then read by the following elisp code:

(defsubst vm-attributes-of (message) (aref message 2))
(defsubst vm-new-flag (message) (aref (aref message 2) 0))
(defsubst vm-unread-flag (message) (aref (aref message 2) 1))
(defsubst vm-deleted-flag (message) (aref (aref message 2) 2))
(defsubst vm-filed-flag (message) (aref (aref message 2) 3))
(defsubst vm-replied-flag (message) (aref (aref message 2) 4))
(defsubst vm-written-flag (message) (aref (aref message 2) 5))
(defsubst vm-forwarded-flag (message) (aref (aref message 2) 6))
(defsubst vm-edited-flag (message) (aref (aref message 2) 7))
(defsubst vm-redistributed-flag (message) (aref (aref message 2) 8))

So in the header above, the 5th 't' corresponds to vm-replied-flag.

Really, I'd be very happy if this capability was added, and am happy
to take direction as to how I could help.

JDO

Re: marks in gnus / uw imapd: what marks / how to set?

[Simon Josefsson]

John Owens <jowens@ece.ucdavis.edu> writes:

> Simon Josefsson writes:
>     John Owens <jowens@ece.ucdavis.edu> writes:
>     
>     > Status: RO
>     > X-Status: DFAT
>     >
>     > is the maximum amount of flags stored here?
>     
>     That I do not know.
>
> But that is correct? (Or how it works with UW imapd?)

I dunno, the UW imapd FAQ or source code might tell.

>     > Gnus seems to keep other keywords. Among them is a forwarded flag,
>     > which is kept in X-Keywords as "gnus-forward". I haven't got anything
>     > else to appear in this header yet. But there are other gnus flags:
>
> Real quick - what happens if there's more than one mark?
>
> Keywords: gnus-forwarded gnus-whatever etc.
>
> or
>
> Keywords: gnus-forwarded, gnus-whatever, etc.
>
> ?

Same here, this is a UW imapd internal thing, it isn't part of IMAP or
any other standard.  But the reasonable thing is that multiple marks
are allowed, otherwise the UW imapd mbox store would be very limited.

The Gnusish way to do this would be to make Gnus read VM mailboxes
properly.  I wonder if this is difficult.  Is the VM mailbox format
described anywhere?  Maybe it is just babyl and some external marks
stuff?  I'm not really familiar with VM.

Re: marks in gnus / uw imapd: what marks / how to set?

[John Owens]

Simon Josefsson writes:
    John Owens <jowens@ece.ucdavis.edu> writes:
    
    > Status: RO
    > X-Status: DFAT
    >
    > is the maximum amount of flags stored here?
    
    That I do not know.

But that is correct? (Or how it works with UW imapd?)
    
    > Gnus seems to keep other keywords. Among them is a forwarded flag,
    > which is kept in X-Keywords as "gnus-forward". I haven't got anything
    > else to appear in this header yet. But there are other gnus flags:

Real quick - what happens if there's more than one mark?

Keywords: gnus-forwarded gnus-whatever etc.

or

Keywords: gnus-forwarded, gnus-whatever, etc.

?

    Maybe VM can export messages to some format that Gnus support, that
    supports marks?  Maildir?

I'll take a look. Thanks.

JDO

Re: marks in gnus / uw imapd: what marks / how to set?

[Simon Josefsson]

John Owens <jowens@ece.ucdavis.edu> writes:

> I'm trying to figure out how gnus in conjunction with UW imapd stores
> the same marks.
>
> As far as I can tell, "Read" and "Old" flags are kept in Status and
> "Deleted", "Flagged", "Answered" and "Draft" are in X-Status. So 
>
> Status: RO
> X-Status: DFAT
>
> is the maximum amount of flags stored here?

That I do not know.

> Gnus seems to keep other keywords. Among them is a forwarded flag,
> which is kept in X-Keywords as "gnus-forward". I haven't got anything
> else to appear in this header yet. But there are other gnus flags:
>
> gnus-replied mark - I presume this corresponds to Answered.

Yes.

> gnus-recent mark - I presume this corresponds to not-Old, but I don't
> know how this works with gnus-unseen-mark.

I think you can ignore both recent and unseen, it is just for newly
delivered mail.

> gnus-cached-mark - No idea.

Don't frob this one, it is maintained internally by Gnus, when you
add/remove an article to the Gnus cache.

> gnus-saved-mark - No idea. (Hopefully it is triggered when I copy a
> message from a folder to another folder.)

Almost, I think it is set when you save a message using e.g. O f or
something.

> Other flags? Distinction between "new" and "unread"? "edited"?
> "redistributed"? 

Do you have lots of VM-"new" messages?  Isn't it just a transient mark
used by vm to identity new messages?  If so, it doesn't look like it
is something worth maintaining (all your messages are non-new).

"unread" is implied if you don't set a read mark in Gnus (can a
message in VM be both "read" and "unread"?).  I'm not sure if there is
something similar to edited or redistributed.

> Any suggestions as to how to move these flags over would be
> appreciated. I figure perl is my best bet? Suggestions of a perl
> module that lets me parse mail a message at a time from mbox format,
> and mess with headers too, would be great.

Maybe VM can export messages to some format that Gnus support, that
supports marks?  Maildir?

marks in gnus / uw imapd: what marks / how to set?

[John Owens]

(Apologies if this gets accidentally posted twice.)

I'd like to migrate my mail from another mailer (VM) (which stores its
files in mbox format) and maintain as much information from VM's state
as possible, including "marks" (read, new, forwarded, replied, etc.).
Currently I am thinking I need to write a perl script to convert VM
mail to gnus-nnimap mail so that I can maintain these marks. (Other
suggestions gratefully accepted.)

VM seems to store all marks in a special header (X-VM-v5-data). VM
uses the following flags, which I'd like to move to gnus.

vm-new-flag vm-unread-flag vm-deleted-flag vm-filed-flag 
vm-replied-flag vm-written-flag vm-forwarded-flag vm-edited-flag 
vm-redistributed-flag 

I'm trying to figure out how gnus in conjunction with UW imapd stores
the same marks.

As far as I can tell, "Read" and "Old" flags are kept in Status and
"Deleted", "Flagged", "Answered" and "Draft" are in X-Status. So 

Status: RO
X-Status: DFAT

is the maximum amount of flags stored here?

Gnus seems to keep other keywords. Among them is a forwarded flag,
which is kept in X-Keywords as "gnus-forward". I haven't got anything
else to appear in this header yet. But there are other gnus flags:

gnus-replied mark - I presume this corresponds to Answered.

gnus-recent mark - I presume this corresponds to not-Old, but I don't
know how this works with gnus-unseen-mark.

gnus-cached-mark - No idea.

gnus-saved-mark - No idea. (Hopefully it is triggered when I copy a
message from a folder to another folder.)

Other flags? Distinction between "new" and "unread"? "edited"?
"redistributed"? 

Any suggestions as to how to move these flags over would be
appreciated. I figure perl is my best bet? Suggestions of a perl
module that lets me parse mail a message at a time from mbox format,
and mess with headers too, would be great.

JDO