Extend nnimap to request Gmail labels with message headers

Extend nnimap to request Gmail labels with message headers

[Trevor Murphy]

This is a proof-of-concept / RFC series of patches.  It may not make sense to
treat these labels as message headers, I just find it suits my workflow pretty
well.

I had to significantly rewrite `nnimap-header-parameters' to incorporate the
option, so please pay special attention to the first patch.  Note that the it
shouldn't change the functionality, it's just a rewrite that makes the later
patches possible.

After that first patch the next two just parse Google's response.  I haven't
extensively tested Google's API, so even though this code works for me and
looks simple, I don't know when it will break.

The net result is that you get X-GM-LABELS exposed as a message header.  The
content is always a parenthesized (possibly empty) list.  Full docs here:
https://developers.google.com/gmail/imap_extensions#access_to_gmail_labels_x-gm-labels

I add this to `gnus-extra-headers' and use it all over my Gmail summary
buffers.

For instance, you can display the labels in summary buffer lines with a user
format function and narrow the buffer to a label with limit-to-extra.

Thoughts?

-- 
Trevor Murphy, ASA, MAAA
GnuPG Key: 0x83881C0A

[RFC 1/3] Rewrite `nnimap-header-parameters'

[Trevor Murphy]

This change should not affect the behavior of the function.  We build
a list then format it with the "%s" specification, as opposed to
writing the list structure directly into format's string argument.
---
 lisp/gnus/nnimap.el | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index ad48d47..390ccd5 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -166,14 +166,18 @@ textual parts.")
   (nnimap-find-process-buffer nntp-server-buffer))
 
 (defun nnimap-header-parameters ()
-  (format "(UID RFC822.SIZE BODYSTRUCTURE %s)"
-	  (format
+  (let (l)
+    (push "UID" l)
+    (push "RFC822.SIZE" l)
+    (push "BODYSTRUCTURE" l)
+    (push (format
 	   (if (nnimap-ver4-p)
 	       "BODY.PEEK[HEADER.FIELDS %s]"
 	     "RFC822.HEADER.LINES %s")
 	   (append '(Subject From Date Message-Id
 			     References In-Reply-To Xref)
-		   nnmail-extra-headers))))
+		   nnmail-extra-headers)) l)
+    (format "%s" (nreverse l))))
 
 (deffoo nnimap-retrieve-headers (articles &optional group server fetch-old)
   (when group
-- 
2.1.3

[RFC 2/3] Request Gmail labels in IMAP headers

[Trevor Murphy]

---
 lisp/gnus/nnimap.el | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 390ccd5..5eb812a 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -169,6 +169,8 @@ textual parts.")
   (let (l)
     (push "UID" l)
     (push "RFC822.SIZE" l)
+    (when (nnimap-capability "X-GM-EXT-1")
+      (push "X-GM-LABELS" l))
     (push "BODYSTRUCTURE" l)
     (push (format
 	   (if (nnimap-ver4-p)
-- 
2.1.3

[RFC 3/3] Parse Gmail labels as IMAP headers

[Trevor Murphy]

---
 lisp/gnus/nnimap.el | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lisp/gnus/nnimap.el b/lisp/gnus/nnimap.el
index 5eb812a..f9b92eb 100644
--- a/lisp/gnus/nnimap.el
+++ b/lisp/gnus/nnimap.el
@@ -203,7 +203,7 @@ textual parts.")
 
 (defun nnimap-transform-headers ()
   (goto-char (point-min))
-  (let (article lines size string)
+  (let (article lines size string labels)
     (block nil
       (while (not (eobp))
 	(while (not (looking-at "\\* [0-9]+ FETCH"))
@@ -238,6 +238,9 @@ textual parts.")
 				      t)
 		   (match-string 1)))
 	(beginning-of-line)
+	(when (search-forward "X-GM-LABELS" (line-end-position) t)
+	  (setq labels (ignore-errors (read (current-buffer)))))
+	(beginning-of-line)
 	(when (search-forward "BODYSTRUCTURE" (line-end-position) t)
 	  (let ((structure (ignore-errors
 			     (read (current-buffer)))))
@@ -257,6 +260,8 @@ textual parts.")
 	  (insert (format "Chars: %s\n" size)))
 	(when lines
 	  (insert (format "Lines: %s\n" lines)))
+	(when labels
+	  (insert (format "X-GM-LABELS: %s\n" labels)))
 	;; Most servers have a blank line after the headers, but
 	;; Davmail doesn't.
 	(unless (re-search-forward "^\r$\\|^)\r?$" nil t)
-- 
2.1.3

Re: Extend nnimap to request Gmail labels with message headers

[Ted Zlatanov]

On Mon, 10 Nov 2014 20:09:20 -0500 Trevor Murphy <trevor.m.murphy@gmail.com> wrote: 

TM> This is a proof-of-concept / RFC series of patches.  It may not make sense to
TM> treat these labels as message headers, I just find it suits my workflow pretty
TM> well.

Very cool.  Lars should review, and write should be supported too, so he
may want to use the patch but integrate it deeper.

Ted

Re: Extend nnimap to request Gmail labels with message headers

[Eric Abrahamsen]

Trevor Murphy <trevor.m.murphy@gmail.com> writes:

> This is a proof-of-concept / RFC series of patches.  It may not make sense to
> treat these labels as message headers, I just find it suits my workflow pretty
> well.
>
> I had to significantly rewrite `nnimap-header-parameters' to incorporate the
> option, so please pay special attention to the first patch.  Note that the it
> shouldn't change the functionality, it's just a rewrite that makes the later
> patches possible.
>
> After that first patch the next two just parse Google's response.  I haven't
> extensively tested Google's API, so even though this code works for me and
> looks simple, I don't know when it will break.
>
> The net result is that you get X-GM-LABELS exposed as a message header.  The
> content is always a parenthesized (possibly empty) list.  Full docs here:
> https://developers.google.com/gmail/imap_extensions#access_to_gmail_labels_x-gm-labels
>
> I add this to `gnus-extra-headers' and use it all over my Gmail summary
> buffers.
>
> For instance, you can display the labels in summary buffer lines with a user
> format function and narrow the buffer to a label with limit-to-extra.
>
> Thoughts?

I'm curious how this interacts with Gmail labels' current behavior:
treating them like imap groups. If a message has multiple labels, it
appears copied to multiple groups, is that right? So won't all the
messages in a group just have the same label? (Unless a message happens
to have multiple labels.)

Just curious what this will look like...

Eric

Re: Extend nnimap to request Gmail labels with message headers

[Trevor Murphy]

On Mon, Nov 10 2014, Eric Abrahamsen wrote: 
 
> I'm curious how this interacts with Gmail labels' current 
> behavior: treating them like imap groups. If a message has 
> multiple labels, it appears copied to multiple groups, is that 
> right? So won't all the messages in a group just have the same 
> label? (Unless a message happens to have multiple labels.) 
 
Google's actually pretty clever about that.  They change up the 
way they give you the X-GM-LABELS info depending on which folder 
you're reading from.  So if you throw it into a format function 
you don't get the redundant group name, only the other labels if 
there are any.

At least, I think that's Google's doing.  I know I didn't write 
code to make it happen.

Google does send down some labels that I never want to see in my 
summary lines, though.  "Important", "Starred", "Sent", and some 
old labels that I keep around but find boring.  Since the 
X-GM-LABELS info always comes as a parenthesized list, it's easy 
to go:

(defun my-gmail-label-format-function (header)
  (let (labels (ignore-errors (read (gnus-extra-header 
  'X-GM-LABELS header))))
   ...))

Now `labels' is a (possibly empty) list of symbols and I just go 
to town with delq and what not.  I strip out the ones I don't care 
about and mapconcat the symbol-names of the rest.

-- 
Trevor Murphy, ASA, MAAA
GnuPG Key: 0x83881C0A

Re: Extend nnimap to request Gmail labels with message headers

[Eric Abrahamsen]

Trevor Murphy <trevor.m.murphy@gmail.com> writes:

> On Mon, Nov 10 2014, Eric Abrahamsen wrote: 
>
>> I'm curious how this interacts with Gmail labels' current behavior:
>> treating them like imap groups. If a message has multiple labels, it
>> appears copied to multiple groups, is that right? So won't all the
>> messages in a group just have the same label? (Unless a message
>> happens to have multiple labels.) 
>
> Google's actually pretty clever about that.  They change up the way
> they give you the X-GM-LABELS info depending on which folder you're
> reading from.  So if you throw it into a format function you don't get
> the redundant group name, only the other labels if there are any.
>
> At least, I think that's Google's doing.  I know I didn't write code
> to make it happen.
>
> Google does send down some labels that I never want to see in my
> summary lines, though.  "Important", "Starred", "Sent", and some old
> labels that I keep around but find boring.  Since the X-GM-LABELS info
> always comes as a parenthesized list, it's easy to go:
>
> (defun my-gmail-label-format-function (header)
>  (let (labels (ignore-errors (read (gnus-extra-header 'X-GM-LABELS
> header))))
>   ...))
>
> Now `labels' is a (possibly empty) list of symbols and I just go to
> town with delq and what not.  I strip out the ones I don't care about
> and mapconcat the symbol-names of the rest.

Interesting -- thanks for the rundown! I guess I rarely have messages
with more than one label (I don't use the webmail interface at all, and
do my best to pretend I'm not using gmail), but I can certainly see
where this would be useful for those who do.

Thanks,
Eric

Re: Extend nnimap to request Gmail labels with message headers

[Trevor Murphy]

On Mon, Nov 10 2014, Trevor Murphy wrote: 

> This is a proof-of-concept / RFC series of patches.  It may not 
> make sense to treat these labels as message headers, I just find 
> it suits my workflow pretty well.
> 
> [snip]
> 
> Thoughts?

Bumping this thread with a cc: to Lars.  I've already signed the 
FSF copyright paperwork, and would love some feedback whether or 
not these patches are worth applying.
-- 
Trevor Murphy, ASA, MAAA
GnuPG Key: 0x83881C0A

Re: Extend nnimap to request Gmail labels with message headers

[Lars Ingebrigtsen]

Trevor Murphy <trevor.m.murphy@gmail.com> writes:

> This is a proof-of-concept / RFC series of patches.  It may not make sense to
> treat these labels as message headers, I just find it suits my workflow pretty
> well.

[...]

> I add this to `gnus-extra-headers' and use it all over my Gmail summary
> buffers.
>
> For instance, you can display the labels in summary buffer lines with a user
> format function and narrow the buffer to a label with limit-to-extra.
>
> Thoughts?

Looks good to me.  Do you have Emacs copyright assignment paperwork on
file?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Extend nnimap to request Gmail labels with message headers

[Trevor Murphy]

On Sun, Jan 25 2015, Lars Ingebrigtsen wrote: 

> Looks good to me.  Do you have Emacs copyright assignment 
> paperwork on file?

Yes.  Back in October I submitted some patches to Org Mode.  I was 
case #939344.
-- 
Trevor Murphy, ASA, MAAA
GnuPG Key: 0x83881C0A

Re: Extend nnimap to request Gmail labels with message headers

[Lars Ingebrigtsen]

Trevor Murphy <trevor.m.murphy@gmail.com> writes:

> On Sun, Jan 25 2015, Lars Ingebrigtsen wrote: 
>
>> Looks good to me.  Do you have Emacs copyright assignment paperwork
>> on file?
>
> Yes.  Back in October I submitted some patches to Org Mode.  I was
> case #939344.

Great.  I'll apply your patches now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Extend nnimap to request Gmail labels with message headers

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Trevor Murphy <trevor.m.murphy@gmail.com> writes:
>
>> On Sun, Jan 25 2015, Lars Ingebrigtsen wrote: 
>>
>>> Looks good to me.  Do you have Emacs copyright assignment paperwork
>>> on file?
>>
>> Yes.  Back in October I submitted some patches to Org Mode.  I was
>> case #939344.
>
> Great.  I'll apply your patches now.

This should probably have some kind of documentation in the Gnus manual,
too.  Can you suggest some text to go with this?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Extend nnimap to request Gmail labels with message headers

[Trevor Murphy]

On Mon, Jan 26 2015, Lars Ingebrigtsen wrote: 

> This should probably have some kind of documentation in the Gnus 
> manual, too.  Can you suggest some text to go with this?

Sure can!  I went looking for existing documentation of suggested 
extra headers or Gmail integration, but didn't see anything.  So I 
didn't write this to fit in with anything in particular.  If you 
let me know where you're planning to put this, I'll rewrite to 
suit.

Support for IMAP Extensions:

If you're using Google's Gmail, you may want to see your Gmail 
labels when reading your mail.  Gnus can give you this information 
if you ask for X-GM-LABELS in the variable `gnus-extra-headers'. 
For example:

    (setq gnus-extra-headers
          '(To Newsgroups X-GM-LABELS))

This will result in Gnus storing your labels in message header 
structures for later use.  The content is always a parenthesized 
(possible empty) list.

-- 
Trevor Murphy, ASA, MAAA
GnuPG Key: 0x83881C0A

Re: Extend nnimap to request Gmail labels with message headers

[Ted Zlatanov]

On Thu, 29 Jan 2015 13:13:42 -0500 Trevor Murphy <trevor.m.murphy@gmail.com> wrote: 

TM> If you're using Google's Gmail, you may want to see your Gmail labels
TM> when reading your mail.  Gnus can give you this information if you ask
TM> for X-GM-LABELS in the variable `gnus-extra-headers'. For example:

TM>    (setq gnus-extra-headers
TM>          '(To Newsgroups X-GM-LABELS))

TM> This will result in Gnus storing your labels in message header
TM> structures for later use.  The content is always a parenthesized
TM> (possible empty) list.

Is there a downside to making this the default?  Other than paying
tribute to our Google Overlords?

Ted

Re: Extend nnimap to request Gmail labels with message headers

[Lars Ingebrigtsen]

Trevor Murphy <trevor.m.murphy@gmail.com> writes:

> Sure can!  I went looking for existing documentation of suggested
> extra headers or Gmail integration, but didn't see anything.  So I
> didn't write this to fit in with anything in particular.  If you let
> me know where you're planning to put this, I'll rewrite to suit.
>
> Support for IMAP Extensions:

I put it into its own node under the nnimap node in git Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Extend nnimap to request Gmail labels with message headers

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Is there a downside to making this the default?  Other than paying
> tribute to our Google Overlords?

There's this loop:

	    ;; Extra.
	    (when gnus-extra-headers
	      (let ((extra gnus-extra-headers)
		    out)
		(while extra
		  (goto-char p)
		  (when (search-forward
			 (concat "\n" (symbol-name (car extra)) ":") nil t)
		    (push (cons (car extra) (nnheader-header-value))
			  out))
		  (pop extra))
		out))))

So the longer the list is, the more it'll search.  But Gmail is so
popular that it might make sense just to put it in there, anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Extend nnimap to request Gmail labels with message headers

[Ted Zlatanov]

On Thu, 05 Feb 2015 13:38:41 +1100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 

LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Is there a downside to making this the default?  Other than paying
>> tribute to our Google Overlords?

...
LI> So the longer the list is, the more it'll search.  But Gmail is so
LI> popular that it might make sense just to put it in there, anyway.

OK, please count me in favor.

Ted

Re: Extend nnimap to request Gmail labels with message headers

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 05 Feb 2015 13:38:41 +1100 Lars Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LI> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> Is there a downside to making this the default?  Other than paying
>>> tribute to our Google Overlords?
>
> ...
> LI> So the longer the list is, the more it'll search.  But Gmail is so
> LI> popular that it might make sense just to put it in there, anyway.
>
> OK, please count me in favor.

Ok; added.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/