gmail and X-GM-EXT1 extensions

gmail and X-GM-EXT1 extensions

[Leonidas Tsampros]

Hello,

long time no see :)

TLDR: My Gmail inbox is full even after splitting mail. Here is why.

Due to some not so recent events, I eventually had to move 95% of my
email setup to gmail.

My current setup is that all email arrives at gmail in a specific folder
and with the help of nnimap-inbox, I fancy-split all the mails that
arrive there. The series of commands that I see in my *imap log* buffer
is the following:

12206 LIST "" "*"
12207 SELECT "ltsampros@upnet.gr"
12208 UID FETCH 1:* FLAGS
12209 UID FETCH 30246:30254,30256:30259 (UID BODY.PEEK[HEADER])
12210 UID COPY 30247,30250:30251,30253,30256,30258 "mail.gnu.emacs.devel"
12211 UID COPY 30246,30248:30249,30252,30254,30257,30259 "mail.gnu.emacs.orgmode"
12212 UID STORE 30246:30254,30256:30259 +FLAGS.SILENT (\Deleted)
12213 UID EXPUNGE 30246:30254,30256:30259

which is working.

However, we all know that Gmail IMAP is a weird monster. My problem is
that all these messages are appearing in the "Inbox" folder on Gmail's
interface ,even after the COPY and EXPUNGE, which means now that my
inbox is filled with very very very many emails and that *irritates* me.

So, I started digging by selecting a random mailbox created by
nnimap-split-incoming:

UID SELECT mail.gnu.emacs.devel
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen gnus-save gnus-forward $has_cal)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen gnus-save gnus-forward $has_cal \*)] Flags permitted.
* OK [UIDVALIDITY 129] UIDs valid.
* 3032 EXISTS
* 0 RECENT
* OK [UIDNEXT 11812] Predicted next UID.
UID OK [READ-WRITE] mail.gnu.emacs.devel selected. (Success)

I think this is pretty much normal. But let's fetch flags of the last
email in this folder:

UID FETCH 3032 FLAGS
* 3032 FETCH (FLAGS ())

Again it does seem normal, however this mail in Gmail's web interface is
STILL listed with the following labels "Inbox" and
"mail.gnu.emacs.devel".

After an enlightening moment of google-funess, I arrived at this URL:

https://developers.google.com/google-apps/gmail/imap_extensions?hl=el#access_to_gmail_labels_x-gm-labels

Which pretty much explains the situation:

UID FETCH 3032 (FLAGS X-GM-LABELS)
* 3032 FETCH (X-GM-LABELS ("\\Inbox") FLAGS ())

Again, gmail confuses me a bit since I would expect this email to have two
labels ("\\inbox" and "mail.gnu.emacs.devel") since these two are
displayed on the web interface but let's accept that the gmail interface
interprets the folder created by nnimap-split-incomings COPY as a
label.

So, how do we remove this fraking \\Inbox X-GM-LABEL which messes up and
clatters our web interface?

Simple:

UID STORE 3032 -X-GM-LABELS ("\\Inbox")
* 3032 FETCH (X-GM-LABELS ())
UID OK Success

And now the million dollar question:

Is there an easy way to run this command before the expunge during
nnimap-split-incoming?

Best Regards,
Leonidas Tsampros

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Leonidas Tsampros]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Leonidas Tsampros <ltsampros@upnet.gr> writes:
> Hello,
>
> long time no see :)
>
> TLDR: My Gmail inbox is full even after splitting mail. Here is why.
>

<snip>

That was easier than I thought. Please take a look at the patch attached
and send me your comments.

The changes can be described as such:

1. Add nnimap server user defined variable that defines whether the
server supports x-gm-ext-1 extension. Ideally this should come
from CAPABILITY string, but I wanted an explicit user knob for that.

2. Add nnimap-rfc822-date-to-imap4date that takes an rfc822 date value
and returns an imap4 datetime string (took the format from rfc3501).

3. Improve nnmail-split-incoming-mail so that if nnimap-x-gm-ext-1 is
defined for the current server, remove that fraking \\Inbox label from
the to be splitted mail. (Should this behaviour be configurable?)

4. Improve nnimap-request-accept-article so that it sets the
INTERNALDATE of the APPENDed message to the value of the message's date
header. (that's what I needed 2.)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap x-gm-ext-1 improvement --]
[-- Type: text/x-patch, Size: 1816 bytes --]

=== modified file 'lisp/gnus/nnimap.el'
--- lisp/gnus/nnimap.el	2013-01-01 09:11:05 +0000
+++ lisp/gnus/nnimap.el	2013-04-21 23:04:35 +0000
@@ -66,6 +66,10 @@
 If nnimap-stream is `ssl', this will default to `imaps'.  If not,
 it will default to `imap'.")
 
+(defvoo nnimap-x-gm-ext-1 nil
+  "Defines whether this server supports the Gmail extensions
+  X-GM-EXT-1. Defaults to nil.")
+
 (defvoo nnimap-stream 'undecided
   "How nnimap talks to the IMAP server.
 The value should be either `undecided', `ssl' or `tls',
@@ -1060,6 +1064,7 @@
   (when (nnimap-possibly-change-group nil server)
     (nnmail-check-syntax)
     (let ((message-id (message-field-value "message-id"))
+	  (dt (message-field-value "date"))
 	  sequence message)
       (nnimap-add-cr)
       (setq message (buffer-substring-no-properties (point-min) (point-max)))
@@ -1072,7 +1077,8 @@
 	    (nnimap-unselect-group))
 	  (erase-buffer)
 	  (setq sequence (nnimap-send-command
-			  "APPEND %S {%d}" (utf7-encode group t)
+			  "APPEND %S %S {%d}" (utf7-encode group t)
+			  (nnimap-rfc822-date-to-imap4date dt)
 			  (length message)))
 	  (unless nnimap-streaming
 	    (nnimap-wait-for-connection "^[+]"))
@@ -1907,6 +1913,8 @@
 	(nnimap-fetch-inbox new-articles)
 	(nnimap-transform-split-mail)
 	(nnheader-ms-strip-cr)
+	(when nnimap-x-gm1-ext
+	  (nnimap-wait-for-response (nnimap-send-command "UID STORE %s -X-GM-LABELS (\\Inbox)" (nnimap-article-ranges new-articles))))
 	(nnmail-cache-open)
 	(nnmail-split-incoming (current-buffer)
 			       #'nnimap-save-mail-spec
@@ -2044,6 +2052,8 @@
 		   "(OR (OR HEADER Message-Id %S HEADER REFERENCES %S) %s)"
 		   refid refid value)))))
 
+(defun nnimap-rfc822-date-to-imap4date (datetime)
+  (format-time-string "%d-%b-%Y %H:%M:%S %z" (date-to-time datetime)))
 
 (provide 'nnimap)
 

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Ted Zlatanov]

On Mon, 22 Apr 2013 02:06:19 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 

LT> That was easier than I thought. Please take a look at the patch attached
LT> and send me your comments.

Looks OK to me.  Can the behavior be made optional?

Ted

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Leonidas Tsampros]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Mon, 22 Apr 2013 02:06:19 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 
>
> LT> That was easier than I thought. Please take a look at the patch attached
> LT> and send me your comments.
>
> Looks OK to me.  Can the behavior be made optional?
>
> Ted

Yes,  I have added a per server nnimap switch nnimap-x-gm-ext-1 with the
default value set to nil. For example a gmail server configuration would
look like this:

(setq gnus-secondary-select-methods '((nnimap "gmail"
					      (nnimap-address "imap.gmail.com")
					      (nnimap-server-port 993)
					      (nnimap-stream ssl)
					      (nnimap-x-gm-ext-1 t))))

So, nnmail-split-incoming-mail behaviour is togglable by the
nnimap-x-gm-ext1 switch. However, I don't have a setup with multiple
imap servers to test it.

Do we need point 4 (INTERNALDATE of APPENDed messages) to be an optional
behaviour as well?

--
Leonidas

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Ted Zlatanov]

On Sun, 09 Jun 2013 21:53:49 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 

LT> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Mon, 22 Apr 2013 02:06:19 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 
>> 
LT> That was easier than I thought. Please take a look at the patch attached
LT> and send me your comments.
>> 
>> Looks OK to me.  Can the behavior be made optional?

LT> Yes,  I have added a per server nnimap switch nnimap-x-gm-ext-1 with the
LT> default value set to nil. For example a gmail server configuration would
LT> look like this:

(setq gnus-secondary-select-methods '((nnimap "gmail"
					      (nnimap-address "imap.gmail.com")
					      (nnimap-server-port 993)
					      (nnimap-stream ssl)
					      (nnimap-x-gm-ext-1 t))))

LT> So, nnmail-split-incoming-mail behaviour is togglable by the
LT> nnimap-x-gm-ext1 switch. However, I don't have a setup with multiple
LT> imap servers to test it.

OK.  It feels like a pretty special one-off so maybe it could be

(nnimap-ext x-gm-ext-1)

instead, to accomodate other weird future extensions and to avoid
special defvoos?

LT> Do we need point 4 (INTERNALDATE of APPENDed messages) to be an optional
LT> behaviour as well?

Erm.... yes? :)  Can you fit it into the nnimap-ext style I suggested?

Ted

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Leonidas Tsampros]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Sun, 09 Jun 2013 21:53:49 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 
>
> LT> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> On Mon, 22 Apr 2013 02:06:19 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 
>>> 
> LT> That was easier than I thought. Please take a look at the patch attached
> LT> and send me your comments.
>>> 
>>> Looks OK to me.  Can the behavior be made optional?
>
> LT> Yes,  I have added a per server nnimap switch nnimap-x-gm-ext-1 with the
> LT> default value set to nil. For example a gmail server configuration would
> LT> look like this:
>
> (setq gnus-secondary-select-methods '((nnimap "gmail"
> 					      (nnimap-address "imap.gmail.com")
> 					      (nnimap-server-port 993)
> 					      (nnimap-stream ssl)
> 					      (nnimap-x-gm-ext-1 t))))
>
> LT> So, nnmail-split-incoming-mail behaviour is togglable by the
> LT> nnimap-x-gm-ext1 switch. However, I don't have a setup with multiple
> LT> imap servers to test it.
>
> OK.  It feels like a pretty special one-off so maybe it could be
>
> (nnimap-ext x-gm-ext-1)
>
> instead, to accomodate other weird future extensions and to avoid
> special defvoos?

Ok. Sounds rational.

> LT> Do we need point 4 (INTERNALDATE of APPENDed messages) to be an optional
> LT> behaviour as well?
>
> Erm.... yes? :)  Can you fit it into the nnimap-ext style I suggested?

I'll do that, but technically it's not an extension.

According to RFC3501 (2.3.3 and 6.3.11) INTERNALDATE is optional during
APPEND. My interpretation is that is legal to set INTERNALDATE using the
value of the 'date' header and we could do it because:

a) there is no other way to manipulate the INTERNALDATE of a
message.

b) servers MUST use the current date/time if INTERNALDATE is
not defined during APPEND.

Thunderbird got this fixed in 2006

https://bugzilla.mozilla.org/show_bug.cgi?id=332626#c31

and they had a discussion here as well:

http://forums.mozillazine.org/viewtopic.php?p=2757812#2757812

Is there really a use case were we don't want the INTERNALDATE to be set
during APPEND?

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Ted Zlatanov]

On Mon, 10 Jun 2013 08:46:26 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 

LT> Ted Zlatanov <tzz@lifelogs.com> writes:
>> On Sun, 09 Jun 2013 21:53:49 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 

LT> Do we need point 4 (INTERNALDATE of APPENDed messages) to be an optional
LT> behaviour as well?
>> 
>> Erm.... yes? :)  Can you fit it into the nnimap-ext style I suggested?

LT> I'll do that, but technically it's not an extension.

OK, don't do it then.  I am really OK either way as long as it works for you.

LT> Is there really a use case were we don't want the INTERNALDATE to be set
LT> during APPEND?

I don't think so.  I thought you might, since you asked.

Ted

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Leonidas Tsampros]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Mon, 10 Jun 2013 08:46:26 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 
>
> LT> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> On Sun, 09 Jun 2013 21:53:49 +0300 Leonidas Tsampros <ltsampros@upnet.gr> wrote: 
>
> LT> Do we need point 4 (INTERNALDATE of APPENDed messages) to be an optional
> LT> behaviour as well?
>>> 
>>> Erm.... yes? :)  Can you fit it into the nnimap-ext style I suggested?
>
> LT> I'll do that, but technically it's not an extension.
>
> OK, don't do it then.  I am really OK either way as long as it works for you.
>
> LT> Is there really a use case were we don't want the INTERNALDATE to be set
> LT> during APPEND?
>
> I don't think so.  I thought you might, since you asked.
>
> Ted

Attached a new version as discussed last time.

Recap:
- new server option nnimap-ext. Current offering is x-gm-ext-1. When
`x-gm-ext-1' is enabled, the \\Inbox gmail label is removed from fancy
splitted mail.

- we now set the INTERNALDATE of messages when doing APPEND (fixes the
  incorrect date's in gmail web ui when importing mail in gmail).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap-patch --]
[-- Type: text/x-patch, Size: 1865 bytes --]

=== modified file 'lisp/gnus/nnimap.el'
--- lisp/gnus/nnimap.el	2013-01-01 09:11:05 +0000
+++ lisp/gnus/nnimap.el	2013-07-18 03:18:12 +0000
@@ -66,6 +66,11 @@
 If nnimap-stream is `ssl', this will default to `imaps'.  If not,
 it will default to `imap'.")
 
+(defvoo nnimap-ext nil
+  "Defines which extensions are supported by the
+  server. Currently `x-gm-ext-1' is offered for gmail imap
+  server.  Defaults to nil.")
+
 (defvoo nnimap-stream 'undecided
   "How nnimap talks to the IMAP server.
 The value should be either `undecided', `ssl' or `tls',
@@ -1060,6 +1065,7 @@
   (when (nnimap-possibly-change-group nil server)
     (nnmail-check-syntax)
     (let ((message-id (message-field-value "message-id"))
+	  (dt (message-field-value "date"))
 	  sequence message)
       (nnimap-add-cr)
       (setq message (buffer-substring-no-properties (point-min) (point-max)))
@@ -1072,7 +1078,8 @@
 	    (nnimap-unselect-group))
 	  (erase-buffer)
 	  (setq sequence (nnimap-send-command
-			  "APPEND %S {%d}" (utf7-encode group t)
+			  "APPEND %S %S {%d}" (utf7-encode group t)
+			  (nnimap-rfc822-date-to-imap4date dt)
 			  (length message)))
 	  (unless nnimap-streaming
 	    (nnimap-wait-for-connection "^[+]"))
@@ -1907,6 +1914,8 @@
 	(nnimap-fetch-inbox new-articles)
 	(nnimap-transform-split-mail)
 	(nnheader-ms-strip-cr)
+	(when (eq nnimap-ext 'x-gm-ext-1)
+	  (nnimap-wait-for-response (nnimap-send-command "UID STORE %s -X-GM-LABELS (\\Inbox)" (nnimap-article-ranges new-articles))))
 	(nnmail-cache-open)
 	(nnmail-split-incoming (current-buffer)
 			       #'nnimap-save-mail-spec
@@ -2044,6 +2053,8 @@
 		   "(OR (OR HEADER Message-Id %S HEADER REFERENCES %S) %s)"
 		   refid refid value)))))
 
+(defun nnimap-rfc822-date-to-imap4date (datetime)
+  (format-time-string "%d-%b-%Y %H:%M:%S %z" (date-to-time datetime)))
 
 (provide 'nnimap)
 

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Lars Magne Ingebrigtsen]

Leonidas Tsampros <ltsampros@upnet.gr> writes:

> - new server option nnimap-ext. Current offering is x-gm-ext-1. When
> `x-gm-ext-1' is enabled, the \\Inbox gmail label is removed from fancy
> splitted mail.
>
> - we now set the INTERNALDATE of messages when doing APPEND (fixes the
>   incorrect date's in gmail web ui when importing mail in gmail).

Looks good.  Do you have Emacs copyright assignment papers on file?

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Leonidas Tsampros]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Leonidas Tsampros <ltsampros@upnet.gr> writes:
>
>> - new server option nnimap-ext. Current offering is x-gm-ext-1. When
>> `x-gm-ext-1' is enabled, the \\Inbox gmail label is removed from fancy
>> splitted mail.
>>
>> - we now set the INTERNALDATE of messages when doing APPEND (fixes the
>>   incorrect date's in gmail web ui when importing mail in gmail).
>
> Looks good.  Do you have Emacs copyright assignment papers on file?

No I don't, but I'm willing to sign them up. How do we accomplish that?

Re: [PATCH] gmail and X-GM-EXT1 extensions

[Lars Magne Ingebrigtsen]

Leonidas Tsampros <ltsampros@upnet.gr> writes:

> No I don't, but I'm willing to sign them up. How do we accomplish that?

Send an email to copyright-clerk@fsf.org saying you want to assign
copyright for code to Emacs, and they'll send you the required
paperwork.

-- 
(domestic pets only, the antidote for overdose, milk.)
  No Gnus T-Shirt for sale: http://ingebrigtsen.no/no.php