That newfangled IMAP thing...

That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

I've been trolling the gnus-bug group a bit, and it seems like the
thing people complain about most is the IMAP support.

So perhaps I should have a peek at it.

But I have no idea how it works, really.  You point Gnus at an IMAP
server, and then what?  How do you tell it how to split your email and
stuff?

I mean, I could probably find out from the manual, but it seems to
assume that you already kinda know, er, something about IMAP.  At least
how to spell I-M-P-A.  Oops.

What's like the minimal work to do if you have a huge nnmail-split-fancy
rule to go over to being an IMAPper? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> But I have no idea how it works, really.  You point Gnus at an IMAP
> server, and then what?

Depends on what you want to do.  You can go to the server buffer, open
the server and explore its folders, then subscribe to them as groups.

Or you can add foreign groups blindly with `G m'.

I've done both.

> How do you tell it how to split your email and stuff?

It doesn't.  You create one nnimap group for each folder found on the
server, pretty much like nntp groups on an nntp server.

Or at least: that's the way I've always done it.

Split incoming mail with procmail, or sieve, or the mail splitting tool
of your choice, directly into the imap server's folders.

You _could_ use IMAP (the protocol) as a smarter, leaner POP3 without
the ugly extensions.

But that's not how most nnimap users use it, I think.

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've been trolling the gnus-bug group a bit, and it seems like the thing
> people complain about most is the IMAP support.  So perhaps I should have a
> peek at it.
>
> But I have no idea how it works, really.  You point Gnus at an IMAP
> server, and then what?  How do you tell it how to split your email and
> stuff?

Like Steinar says, you don't.  The main attraction of IMAP is that you can
access it from multiple places, and usually with multiple clients.

(For me, my phone, a web interface, and Gnus.  I like Gnus, but I can't always
 have access to it everywhere I go, so other paths might be needed.)

[...]

> What's like the minimal work to do if you have a huge nnmail-split-fancy
> rule to go over to being an IMAPper?

It totally depends on your IMAP server, but generally:

1. Transform nnmail-split-fancy into whatever mail filtering language your
   server supports.

2. Look and see if there is Gnus support for that.  (eg: the Sieve generation
   stuff that with group parameters.)

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Tibor Simko]

On Sun, 05 Sep 2010, Lars Magne Ingebrigtsen wrote:
> What's like the minimal work to do if you have a huge
> nnmail-split-fancy rule to go over to being an IMAPper?

There is nnimap-split-fancy; I use something like:

(setq nnimap-crosspost nil)
(setq nnimap-split-inbox '("INBOX"))
(setq nnimap-split-predicate "UNDELETED")
(setq nnimap-split-rule 'nnimap-split-fancy)
(setq nnimap-split-fancy
   '(|  ;; to boot, I don't like duplicates:
        ("Gnus-Warning" "This is a duplicate" "mail.spam.duplicates")
        ;; secondly let us detect mailing lists I am subscribed to:
        ("List-ID" "ding\\.gnus\\.org" "mail.app.emacs.gnus")
        ;; etc...
        ;; default:
        "mail.personal.misc"))

Best regards
--
Tibor Simko

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sun, 05 Sep 2010 03:04:26 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> But I have no idea how it works, really.  You point Gnus at an IMAP
LMI> server, and then what?  How do you tell it how to split your email and
LMI> stuff?

LMI> What's like the minimal work to do if you have a huge nnmail-split-fancy
LMI> rule to go over to being an IMAPper? 

IMAP is a weird protocol with lots of extensions.  It's stateful: you
select a mailbox and then you do things in it.  You can store or
retrieve many types of flags (there's a standard subset).  The
UIDVALIDITY is a really annoying wrinkle, but explaining it would take a
while.  One major issue with IMAP is you can't edit articles, only
append or delete (which is why `e' doesn't work in nnimap groups).

Mailboxes are prefixed and you can find out the server's prefix and path
separator.  For example Courier uses "INBOX" and "." respectively, so
the Maildir path ROOT/a/b/c translates into "INBOX.a.b.c"; I'll use my
Courier setup for the rest of this.

Splitting from INBOX (usually the incoming mail goes there) is pretty
easy as long as you have the server as a primary or secondary.  Here's
my setup:

(defun nnimap-courier-lifelogs-split-fancy ()
  (let ((nnimap-split-fancy nnimap-courier-lifelogs-split-fancy)
        (spam-split-group "INBOX.spam"))
    (nnimap-split-fancy)))

(setq
 nnimap-split-rule '(
 ; ...several servers skipped...
                     ("lifelogs" ("INBOX" nnimap-courier-lifelogs-split-fancy)))
 nnmail-split-methods 'nnmail-split-fancy
 nnimap-split-inbox "INBOX"
 nnimap-courier-lifelogs-split-fancy '(|
                      (: gnus-registry-split-fancy-with-parent)
                      (: spam-split 'spam-use-regex-headers)
; ...
                      (any "cfengine"   "INBOX.cfengine")
; ...
                      (: spam-split)
                      ;; default mailbox
                      "INBOX.mail"))

I may be forgetting something, but basically you need 

1) the nnimap-split-rule for your server
2) the split-inbox "INBOX"
3) the actual split rules

and the rest is automatic.  I prefer this setup to server-side splitting
because of my gnus-registry and spam-split needs.

If you need a Courier account I can set you up (but with a Maildir
backend it's really trivial to set it up yourself).  IMO dovecot is
better but I haven't gotten around to switching in years now.  GMail's
IMAP support is weird: their mailboxes are tags and a message can be in
more than one mailbox.

HTH
Ted

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> ...  I prefer this setup to server-side splitting because of my
> gnus-registry and spam-split needs.

Right.  I tried it, and dropped it, back in... whenever... (1999-2000 or
thereabouts) because I found it too slow.  It was better to have the
email presplit when I wanted to read it.

Since I came to nnimap from split-fancy into nnml meant was losing some
functionality (I liked split fancy a lot more than procmail).  But I
learned to live with it.

When I went from UoW imapd to cyrus I had to use sieve.  But I never
got the hang of that, so it was actually a relief to go back to
procmail, when I ditched cyrus for dovecot for my IMAP server.  Dovecot
has sieve support, but I've never looked at it.

Lars: if you want to install an imap server to test against, I recommend
dovecot.   Really easy to set up.  Uses maildir or mbox (as well as a
couple other formats) in home directory subdirectories ( ̃/Maildir and
̃/Mail I think...).  I use maildir and deliver to the mail folders with
procmail and procmail native maildir support.

Re: That newfangled IMAP thing...

[Bjørn Mork]

[-- Attachment #1: Type: text/plain, Size: 628 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've been trolling the gnus-bug group a bit, and it seems like the
> thing people complain about most is the IMAP support.
>
> So perhaps I should have a peek at it.

Please do.

And while you are at it, please consider something like the attached
patch.  I've been using it since ... forever, and find it absolutely
necessary to even consider doing expiry of nnimap groups with a target
group.  Without this, all expired articles are downloaded and then
uploaded even if the target is on the same IMAP server.  That can be
quite annoying on a low bandwidth link...


Bjørn


[-- Attachment #2: 0001-nnimap.el.patch --]
[-- Type: text/x-diff, Size: 2154 bytes --]

From b957eaf442b63ea24eb3bee0159bf14b0f3f73d8 Mon Sep 17 00:00:00 2001
From: =?utf-8?q?Bj=C3=B8rn=20Mork?= <bjorn@mork.no>
Date: Tue, 20 Apr 2010 17:37:23 +0200
Subject: [PATCH] nnimap.el:
  - accelerate the common server-internal expiry-target case
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Signed-off-by: Bjørn Mork <bjorn@mork.no>
---
 lisp/nnimap.el |   25 +++++++++++++++++--------
 1 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index c76169c..3a06adb 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -1553,14 +1553,23 @@ function is generally only called when Gnus is shutting down."
 
 (defun nnimap-expiry-target (arts group server)
   (unless (eq nnmail-expiry-target 'delete)
-    (with-temp-buffer
-      (dolist (art arts)
-	(nnimap-request-article art group server (current-buffer))
-	;; hints for optimization in `nnimap-request-accept-article'
-	(let ((nnimap-current-move-article art)
-	      (nnimap-current-move-group group)
-	      (nnimap-current-move-server server))
-	  (nnmail-expiry-target-group nnmail-expiry-target group))))
+    (if (and (not (functionp nnmail-expiry-target)) 
+	     (gnus-server-equal (gnus-group-method nnmail-expiry-target) 
+				(gnus-server-to-method 
+				 (format "nnimap:%s" server))))
+	;; moving article within same server, speed it up...
+	(and (nnimap-possibly-change-group group)
+	     (imap-message-copy (imap-range-to-message-set arts)
+				(gnus-group-short-name nnmail-expiry-target) 
+				'dontcreate nil nnimap-server-buffer))
+      (with-temp-buffer
+	(dolist (art arts)
+	  (nnimap-request-article art group server (current-buffer))
+	  ;; hints for optimization in `nnimap-request-accept-article'
+	  (let ((nnimap-current-move-article art)
+		(nnimap-current-move-group group)
+		(nnimap-current-move-server server))
+	    (nnmail-expiry-target-group nnmail-expiry-target group)))))
     ;; It is not clear if `nnmail-expiry-target' somehow cause the
     ;; current group to be changed or not, so we make sure here.
     (nnimap-possibly-change-group group server)))
-- 
1.5.6.5

Re: That newfangled IMAP thing...

[Bjørn Mork]

Steinar Bang <sb@dod.no> writes:
>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
>
>> ...  I prefer this setup to server-side splitting because of my
>> gnus-registry and spam-split needs.
>
> Right.  I tried it, and dropped it, back in... whenever... (1999-2000 or
> thereabouts) because I found it too slow.  It was better to have the
> email presplit when I wanted to read it.

I do both, depending on the server capabilities.  One of the IMAP
servers I use doesn't support any fancy server side splitting.

But server-side splitting is preferred when available.


Bjørn

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Depends on what you want to do.  You can go to the server buffer, open
> the server and explore its folders, then subscribe to them as groups.

But there are no folders on a server before somebody has created them,
I'd have thought?

> Split incoming mail with procmail, or sieve, or the mail splitting tool
> of your choice, directly into the imap server's folders.

Yuck.  On the mail server?  Do people set up procmail rules still, in
this day and age?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I may be forgetting something, but basically you need 
>
> 1) the nnimap-split-rule for your server
> 2) the split-inbox "INBOX"
> 3) the actual split rules
>
> and the rest is automatic.  I prefer this setup to server-side splitting
> because of my gnus-registry and spam-split needs.

So nnimap pulls down the all the messages from the INBOX folder, applies
the splitting rules, and then moves the messages to the other folders on
the IMAP server, creating the folders if they don't exist?  That sounds
like something that can be made faster with streaming.  :-)

> If you need a Courier account I can set you up (but with a Maildir
> backend it's really trivial to set it up yourself).  IMO dovecot is
> better but I haven't gotten around to switching in years now.  GMail's
> IMAP support is weird: their mailboxes are tags and a message can be in
> more than one mailbox.

The MTA we're using is Dovecot, so I thought I'd just set up an IMAP
backend there.

But I could also create a Gmail account, just to try...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Sebastian Krause]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> Steinar Bang <sb@dod.no> writes:
>> Split incoming mail with procmail, or sieve, or the mail splitting tool
>> of your choice, directly into the imap server's folders.
>
> Yuck.  On the mail server?  Do people set up procmail rules still, in
> this day and age?

Some do. For others it's simply configuring up your Gmail filters
through the web page. They let you access your mails through IMAP
after all...

Re: That newfangled IMAP thing...

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>> Depends on what you want to do.  You can go to the server buffer, open
>> the server and explore its folders, then subscribe to them as groups.
>
> But there are no folders on a server before somebody has created them,
> I'd have thought?

You normally have just a folder Inbox created by default and create new
ones as needed.

>> Split incoming mail with procmail, or sieve, or the mail splitting tool
>> of your choice, directly into the imap server's folders.
>
> Yuck.  On the mail server?  Do people set up procmail rules still, in
> this day and age?

In my case: Yes.

Nothing prevents you from using normal mail splitting capabilities of
Gnus where you split mails from nnimap:Inbox into nnimap:Foo.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: That newfangled IMAP thing...

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]


  But I have no idea how it works, really.  You point Gnus at an IMAP
  server, and then what?  How do you tell it how to split your email and
  stuff?

Others have replied, but I thought I'd explain what I do.

I have split mail with procmail for a long time, and used to used MH-E
to read it.  Then I got a handheld (palm TX) that did IMAP, and
converted to IMAP for storage of my mail.  This caused me to switch to
gnus, because it was on the short list of sane readers with imap
support.

Now, I use 3 devices to read mail:

  gnus on the same computer as the imap server

  iphone (just inbox)

  mac and Mail.app (just inbox, occasionally slightly more)

Most imap folders (== groups in gnus) I read only with gnus; I'd go
crazy without scoring and 'c y'.

Given the above setup, it's critical to have something
pre-final-delivery do spliting.  A lot of the point is to avoid putting
anything in my INBOX unless it's timely and personal - so i can not have
the phone pull it down and not have my brain have to deal with mail with
a slow UI unless it's important - and I'd like that to happen without
gnus running anywhere.

I'm not a raving fan of procmail, but it's good enough and basically I
add a new rule for every mailinglist.  I have 1200 lines (wc -l) between
two accounts, which is more or less 300 commented rules.

From the gnus/imap point of view, what I'd like to see is:

  any caching/agent support to explicitly get that other imap clients
  can change state of the mailbox

  perhaps faster polling by using imap features more smartly

  accurate unread counts by asking the server for unread, not computing
  on sequence numbers

Features I wouldn't necessarily use, but might be cool

  storing gnus split rules as sieve rules on the server.  The basic
  problem is that with imap you usually want multiple clients to deal.

I have really been pretty happy with gnus' imap support over the last
4.5 years.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Greg Troxel <gdt@work.lexort.com> writes:

> I'm not a raving fan of procmail, but it's good enough and basically I
> add a new rule for every mailinglist.  I have 1200 lines (wc -l) between
> two accounts, which is more or less 300 commented rules.

Wow.  :-)

But I can't imagine that's how most non-techie users do it...  On the
other hand, most Gnus users are pretty techie, so...

>   accurate unread counts by asking the server for unread, not computing
>   on sequence numbers

I think that's being worked on.

> Features I wouldn't necessarily use, but might be cool
>
>   storing gnus split rules as sieve rules on the server.  The basic
>   problem is that with imap you usually want multiple clients to deal.

I haven't looked into it, but Gnus does have sieve support, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> But there are no folders on a server before somebody has created them,
> I'd have thought?

There's at least one folder, usually called INBOX (or Inbox, or inbox.
That particular folder name is case-insensitive.  Whether other folder
names are case insensitive or sensitive is implementation-dependent).
	http://tools.ietf.org/html/rfc3501#section-5.1

> Yuck.  On the mail server?  Do people set up procmail rules still, in
> this day and age?

It worked for me in 1995, it works for me now.  Why learn something new
if the old stuff still works?

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Well, that was kinda painful...

So I created a Gmail account, and went to the server buffer and hit
`RET' on the nnmap method.  It queried me for user name and password,
even though that's already in the .authinfo file, and then said:

imap: Connecting to imap.gmail.com...
imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
Waiting for response from imap.gmail.com...done
imap: Plaintext authentication...
imap: Login successful...
imap: Could not UTF7 encode `*', using it unencoded...
imap: Could not UTF7 decode `[Gmail]', using it undecoded...
imap: Could not UTF7 decode `INBOX', using it undecoded...

etc etc etc for quite a while.  Each of these took several seconds, so I
just grew impatient and stopped the thing.

Is that normal?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> So nnimap pulls down the all the messages from the INBOX folder,
> applies the splitting rules, and then moves the messages to the other
> folders on the IMAP server, creating the folders if they don't exist?
> That sounds like something that can be made faster with streaming.
> :-)

There are other things to consider: IMAP allows you to pull down the
header information, and even the IMAP structure of a message, and then
move the messages _on_ the server, without downloading them, or
downloading all of them.

Actually, I think `B m' moves messages between IMAP folders, without
actually downloading them.  I'm not sure, but I think so.

Having partial downloads of messages in nnimap has been a request for a
long time.

> The MTA we're using is Dovecot, so I thought I'd just set up an IMAP
> backend there.

Dovecot is an easy IMAP server to deal with, and very robust (I wouldn't
have called it an MTA though.  Exim4 is my MTA, Dovecot is my IMAP
server).

> But I could also create a Gmail account, just to try...

Yup, it's good to try more than one server at the same time, and several
kinds of server.

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> There are other things to consider: IMAP allows you to pull down the
> header information, and even the IMAP structure of a message, and then
> move the messages _on_ the server, without downloading them, or
> downloading all of them.

Hm.  Is this generally supported across different (popular) server
types, or does it vary a lot?

> Dovecot is an easy IMAP server to deal with, and very robust (I wouldn't
> have called it an MTA though.  Exim4 is my MTA, Dovecot is my IMAP
> server).

Er, yes.  We currently just use it for pop3.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> I haven't looked into it, but Gnus does have sieve support, I think.

Yes, and no.  There is a sieve mode, and it's supposed to be connected
to a server.  But I was never able to get it to work right with cyrus.
I had to upload the rules outside of emacs.

And I've never tried it with dovecot.

I just resurrected my procmail file from subversion, and modified the
folder delivery from mbox (used with UoW imapd) to maildir.

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> And while you are at it, please consider something like the attached
> patch.  I've been using it since ... forever, and find it absolutely
> necessary to even consider doing expiry of nnimap groups with a target
> group.  Without this, all expired articles are downloaded and then
> uploaded even if the target is on the same IMAP server.  That can be
> quite annoying on a low bandwidth link...

That sounds useful.  Do you have copyright assignment papers on file
with the FSF for Emacs patches?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Bjørn Mork <bjorn@mork.no>:

> But server-side splitting is preferred when available.

True.

Now... if one could get gnus fancy-splitting on the server, that would
be neat.

How hard would it be to write an emacs-plugin to dovecot, I wonder...? :-)

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Is that normal?

Could someone explain to me what all the utf-7 stuff around the IMAP
code is?  Is IMAP defined as being in utf-7?  That's like "er, what"
just there...

Anyway, the problems I'm having is that this takes forever:

(encode-coding-string "hei" 'utf-7)

Literally.  It doesn't terminate.

(encode-coding-string "hei" 'utf-8)
=> "hei"

works fine.  Anybody seen this kind of behaviour before?  This is with
Emacs 24.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> (encode-coding-string "hei" 'utf-7)
>
> Literally.  It doesn't terminate.

If I try it in a fresh Emacs 24, it says:

Error during redisplay: (error Lisp nesting exceeds `max-lisp-eval-depth')
if: Lisp nesting exceeds `max-lisp-eval-depth'

It works fine in Emacs 23.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:


> Steinar Bang <sb@dod.no> writes:
>> There are other things to consider: IMAP allows you to pull down the
>> header information, and even the IMAP structure of a message, and then
>> move the messages _on_ the server, without downloading them, or
>> downloading all of them.

> Hm.  Is this generally supported across different (popular) server
> types, or does it vary a lot?

I think it's mandated by the RFC...?  Let me see...

It lists a lot of message attributes,
	http://tools.ietf.org/html/rfc3501#section-2.3
with body structure and one of them
	http://tools.ietf.org/html/rfc3501#section-2.3.6

At a quick overview, none of the message attributes I looked at were
listed at optional, so I guess you can pull them down and look at to use
for splitting.

Moves aren't actually implemented in IMAP.  You have to do a COPY, and
then set the delete flag, followed by an EXPUNGE of the original.
	http://tools.ietf.org/html/rfc3501#section-6.4.7

The COPY command works on the server.  You don't have to download a
message for that.

>> Dovecot is an easy IMAP server to deal with, and very robust (I wouldn't
>> have called it an MTA though.  Exim4 is my MTA, Dovecot is my IMAP
>> server).

> Er, yes.  We currently just use it for pop3.

Actually I haven't even enabled pop3 on mine.  If someone just wants
POP-like behaviour, IMAP is actually a lot nicer than POP3, and needs no
extensions to do what modern POP3 clients would like to do (leave on
server, delete specific messages, etc.).

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> If I try it in a fresh Emacs 24, it says:
>
> Error during redisplay: (error Lisp nesting exceeds `max-lisp-eval-depth')
> if: Lisp nesting exceeds `max-lisp-eval-depth'

Must just be driver error or something.  I did a "bzr update; make
clean; make", and now this works in Emacs 24.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ok.  I can see why nnimap is slow on `g'.

I've included below the *imap-log* on my fresh, almost-empty Gmail
account from this command:

(nnimap-request-list "imap.gmail.com")

As we can see, it first issues a LIST command, and then issues an
EXAMINE for each folder, totally synchronous.  It takes 1.5s for me, and
there's no there there.

I'm guessint that EXAMINE is the equivalent of NNTP GROUP -- it selects
the group, and then it does a "FETCH 1,* UID" for the groups that says
that is has "EXISTS"...

Perhaps I should just read the IMAP RFC instead of guessing.  :-)

Anyway, this has to be painfully slow if you have lots of folders and
high latency.

257 UNSELECT
257 OK Returned to authenticated state. (Success)
258 LIST "" "*%"
* LIST (\HasNoChildren) "/" "INBOX"
* LIST (\HasNoChildren) "/" "Personal"
* LIST (\HasNoChildren) "/" "Receipts"
* LIST (\HasNoChildren) "/" "Travel"
* LIST (\HasNoChildren) "/" "Work"
* LIST (\Noselect \HasChildren) "/" "[Gmail]"
* LIST (\HasNoChildren) "/" "[Gmail]/All Mail"
* LIST (\HasNoChildren) "/" "[Gmail]/Drafts"
* LIST (\HasChildren \HasNoChildren) "/" "[Gmail]/Sent Mail"
* LIST (\HasChildren \HasNoChildren) "/" "[Gmail]/Spam"
* LIST (\HasNoChildren) "/" "[Gmail]/Starred"
* LIST (\HasNoChildren) "/" "[Gmail]/Trash"
258 OK Success
259 EXAMINE "[Gmail]"
259 NO Unknown Mailbox: [Gmail] (Failure)
260 EXAMINE "INBOX"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845937]
* 3 EXISTS
* 0 RECENT
* OK [UIDNEXT 4]
260 OK [READ-ONLY] INBOX selected. (Success)
261 FETCH 1,* UID
* 1 FETCH (UID 1)
* 3 FETCH (UID 3)
261 OK Success
262 EXAMINE "Work"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845935]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
262 OK [READ-ONLY] Work selected. (Success)
263 EXAMINE "[Gmail]/Trash"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845940]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
263 OK [READ-ONLY] [Gmail]/Trash selected. (Success)
264 EXAMINE "[Gmail]/All Mail"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845948]
* 3 EXISTS
* 0 RECENT
* OK [UIDNEXT 4]
264 OK [READ-ONLY] [Gmail]/All Mail selected. (Success)
265 FETCH 1,* UID
* 1 FETCH (UID 1)
* 3 FETCH (UID 3)
265 OK Success
266 EXAMINE "[Gmail]/Sent Mail"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845944]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
266 OK [READ-ONLY] [Gmail]/Sent Mail selected. (Success)
267 EXAMINE "Travel"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845934]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
267 OK [READ-ONLY] Travel selected. (Success)
268 EXAMINE "Personal"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845933]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
268 OK [READ-ONLY] Personal selected. (Success)
269 EXAMINE "[Gmail]/Spam"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845938]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
269 OK [READ-ONLY] [Gmail]/Spam selected. (Success)
270 EXAMINE "Receipts"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845936]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
270 OK [READ-ONLY] Receipts selected. (Success)
271 EXAMINE "[Gmail]/Drafts"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845942]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
271 OK [READ-ONLY] [Gmail]/Drafts selected. (Success)
272 EXAMINE "[Gmail]/Starred"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS ()]
* OK [UIDVALIDITY 641845939]
* 0 EXISTS
* 0 RECENT
* OK [UIDNEXT 1]
272 OK [READ-ONLY] [Gmail]/Starred selected. (Success)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

I have to say that I find the nnimap/imap code slightly opaque...
things just seem to...  happen.

But I think the nnimap-request-list can be made almost totally
streaming, which should help.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Tom Rauchenwald]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> If I try it in a fresh Emacs 24, it says:
>>
>> Error during redisplay: (error Lisp nesting exceeds `max-lisp-eval-depth')
>> if: Lisp nesting exceeds `max-lisp-eval-depth'
>
> Must just be driver error or something.  I did a "bzr update; make
> clean; make", and now this works in Emacs 24.

I guess it was a problem in the trunk, I had the same issue and it is
fixed now.

-tom

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> But I think the nnimap-request-list can be made almost totally
> streaming, which should help.

Here are some relevant bits from the RFC:
	http://tools.ietf.org/html/rfc3501#section-5.5

Re: That newfangled IMAP thing...

[Mark Plaksin]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ok.  I can see why nnimap is slow on `g'.

FWIW when my nnimap groups are 'foreign' it takes a long time to check
each folder.  When they are not foreign it's pretty fast.  (I haven't
updated Gnus since you started working on speeding things up.)

Re: That newfangled IMAP thing...

[David Engster]

Lars Magne Ingebrigtsen writes:
> Ok.  I can see why nnimap is slow on `g'.
>
> I've included below the *imap-log* on my fresh, almost-empty Gmail
> account from this command:
>
> (nnimap-request-list "imap.gmail.com")
>
> As we can see, it first issues a LIST command, and then issues an
> EXAMINE for each folder, totally synchronous.  It takes 1.5s for me, and
> there's no there there.
>
> I'm guessint that EXAMINE is the equivalent of NNTP GROUP -- it selects
> the group, and then it does a "FETCH 1,* UID" for the groups that says
> that is has "EXISTS"...
>
> Perhaps I should just read the IMAP RFC instead of guessing.  :-)
>
> Anyway, this has to be painfully slow if you have lots of folders and
> high latency.

When I press 'g', nnimap does a 'STATUS' on all mailboxes immediately
and then asynchronously collects the responses (see also
`nnimap-retrieve-groups-asynchronous'). People repeatedly have reported
that it does an EXAMINE instead. I forgot why that happened; maybe it's
because you have the server as a foreign one?

-David

Re: That newfangled IMAP thing...

[David Engster]

David Engster writes:
> Lars Magne Ingebrigtsen writes:
>> Ok.  I can see why nnimap is slow on `g'.
>>
>> I've included below the *imap-log* on my fresh, almost-empty Gmail
>> account from this command:
>>
>> (nnimap-request-list "imap.gmail.com")

Sorry, I didn't read that one right. That one is slow, indeed. However,
it shouldn't be called when just checking for new mails in subscribed
groups.

-David

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> When I press 'g', nnimap does a 'STATUS' on all mailboxes immediately
> and then asynchronously collects the responses (see also
> `nnimap-retrieve-groups-asynchronous'). People repeatedly have reported
> that it does an EXAMINE instead. I forgot why that happened; maybe it's
> because you have the server as a foreign one?

There's no longer any difference in how foreign and other groups are
retrieved, but you're right, nnimap does have retrieve-groups, so
request-list isn't used for that.  But it's used to find new groups and
the like, and it'd be nice if it were faster, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I have to say that I find the nnimap/imap code slightly opaque...
> things just seem to...  happen.

It's really complicated code, especially the things that deal with the
network connection and filters and stuff, which just seems kinda
unnecessary.  I mean, the protocol is dead simple.  You issue a
(numbered) command, and you get a (numbered) response.

At this point I feel I would have gotten further if I had just started
from scratch and written a new set of primitives.

This may not be totally compatible with the old version, but does
anybody mind?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> At this point I feel I would have gotten further if I had just started
> from scratch and written a new set of primitives.

But I'm not going to do that today.  My fingers are starting to ache...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Bjørn Mork <bjorn@mork.no> writes:
>
>> And while you are at it, please consider something like the attached
>> patch.  I've been using it since ... forever, and find it absolutely
>> necessary to even consider doing expiry of nnimap groups with a target
>> group.  Without this, all expired articles are downloaded and then
>> uploaded even if the target is on the same IMAP server.  That can be
>> quite annoying on a low bandwidth link...
>
> That sounds useful.  Do you have copyright assignment papers on file
> with the FSF for Emacs patches?

No, unfortunately not.  I guess your question indicates that I should
have.  Any pointers to the current procedure?


Bjørn

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> No, unfortunately not.  I guess your question indicates that I should
> have.  Any pointers to the current procedure?

Good question.  It's been so long that I've forgotten what the procedure
is now.  Anybody?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Greg Troxel <gdt@work.lexort.com> writes:
>
>> I'm not a raving fan of procmail, but it's good enough and basically I
>> add a new rule for every mailinglist.  I have 1200 lines (wc -l) between
>> two accounts, which is more or less 300 commented rules.
>
> Wow.  :-)
>
> But I can't imagine that's how most non-techie users do it...  On the
> other hand, most Gnus users are pretty techie, so...

Non-Techie users use some web front-end. In my university people use
Squirrelmail to define mail splitting rules. With gmail, you use the
Gmail filters etc.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: That newfangled IMAP thing...

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> I may be forgetting something, but basically you need 
>>
>> 1) the nnimap-split-rule for your server
>> 2) the split-inbox "INBOX"
>> 3) the actual split rules
>>
>> and the rest is automatic.  I prefer this setup to server-side splitting
>> because of my gnus-registry and spam-split needs.
>
> So nnimap pulls down the all the messages from the INBOX folder, applies
> the splitting rules, and then moves the messages to the other folders on
> the IMAP server, creating the folders if they don't exist?  That sounds
> like something that can be made faster with streaming.  :-)

Hmm, looks like the nnimap splitting disappeared completely with the
latest git pull.  Probably faster, but still...


Bjørn

Re: That newfangled IMAP thing...

[Bjørn Mork]

Bjørn Mork <bjorn@mork.no> writes:

> Hmm, looks like the nnimap splitting disappeared completely with the
> latest git pull.  Probably faster, but still...

Nevermind, it was already fixed.  Just needed another pull.


Bjørn

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> Hmm, looks like the nnimap splitting disappeared completely with the
> latest git pull.  Probably faster, but still...

Oh, really?  There shouldn't have been any functional changes in the
imap/nnimap changes I committed, just cleanups.

But that's what they all say.

So what happens?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Memnon Anon]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Bjørn Mork <bjorn@mork.no> writes:
>
>> No, unfortunately not.  I guess your question indicates that I should
>> have.  Any pointers to the current procedure?
>
> Good question.  It's been so long that I've forgotten what the procedure
> is now.  Anybody?

,----[ http://orgmode.org/worg/org-contribute.php#sec-2 ]
| Copyright issues when contributing to Emacs org-mode
| 
| Org is made of many files. Most of them are also distributed as part of
| GNU Emacs. These files are called the Org core, and they are all
| copyrighted by the Free Software Foundation, Inc. If you consider
| contributing to these files, your first need to grant the right to
| include your works in GNU Emacs to the FSF. For this you need to
| complete this form, send it to assign@gnu.org, and tell the Org-mode
| maintainer when this process is complete. Some people consider this a
| hassle. I don't want to discuss this in detail here - there are some
| good reasons for getting the copyright registered, an example is
| discussed in this FLOSS weekly podcast. Furthermore, by playing
| according to the Emacs rules, we gain the fantastic advantage that every
| version of Emacs ships with Org-mode already fully built in. So please
| consider doing this - it makes our work as maintainers so much easier,
| because we can then take your patches without any additional work.
| 
| If you want to learn more about why copyright assignments are collected,
| read this: Why the FSF gets copyright assignments from contributors?
`----

If you want the links in this text, go to the url :)

Memnon

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> This may not be totally compatible with the old version, but does
> anybody mind?

Not as long as you get it working again...:-)

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> It's really complicated code, especially the things that deal with the
> network connection and filters and stuff, which just seems kinda
> unnecessary.  I mean, the protocol is dead simple.  You issue a
> (numbered) command, and you get a (numbered) response.

I can't remember but I think that some of the complexity came from
trying to shoehorn IMAP into the backend API at the time, and the
assumptions made by a backend intended for NNTP.

But perhaps the backend API has been changed to be a better fit since
then...?

Re: That newfangled IMAP thing...

[Steinar Bang]

One thing that has happened after the latest pull: the username/password
of authinfo was no longer used, for my private IMAP server.

The server is in gnus-secondary-select-methods, and not added in the
*Server* buffer.  I don't know if that makes a difference?

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> One thing that has happened after the latest pull: the username/password
> of authinfo was no longer used, for my private IMAP server.

Oh, that was a thing I did change, because I thought the current
behavior was a bug.  But perhaps it wasn't?

I've now reverted, sorta.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Oh, that was a thing I did change, because I thought the current
> behavior was a bug.  But perhaps it wasn't?

I always thought it was the intended behaviour.  Not neccessary logical,
but that was the way it was (looking for the imaps password as imap).

> I've now reverted, sorta.

Perhaps you shouldn't?  It feels right to have it the way you reverted
(as long as I know what the cause is, it's trivial to fix).  On the
other hand, it is a change in behaviour so there is bound to be a lot of
"my .authinfo is no longer used" questions...

Re: That newfangled IMAP thing...

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> Yuck.  On the mail server?  Do people set up procmail rules still, in
LMI> this day and age?

Done right, the mail (delivery) server is local.  And even with its
warts procmail still works better than anything else.  (Splitting and
delivery should be in the background; Emacs/gnus should not waste
forground time doing menial chores like that. ☺)

Postfix calls procmail; procmail decides which box; dbmail delivers to
that box, and gnus/nnimap read it in that box.

And all the filtering and splitting occurs in the background.
Where it belongs.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> I always thought it was the intended behaviour.  Not neccessary logical,
> but that was the way it was (looking for the imaps password as imap).

imaps would be cleaner, but let's keep the old behaviour.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

On Sun, Sep 05 2010, Lars Magne Ingebrigtsen wrote: 

> Oh, that was a thing I did change, because I thought the current 
> behavior was a bug.  But perhaps it wasn't? 

If that's the look for imaps in the .authinfo, FWIW, I've been 
bitten the other way around when I've started using Gnus: I though 
it'd use the imaps entry, but it did not.

The change kinda had sense to me. Maybe you should not revert it, 
but had a fallback to imap or something like that.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> The change kinda had sense to me. Maybe you should not revert it, but
> had a fallback to imap or something like that.

Yeah.  When looking for the imaps password it would make sense to fall
back to the imap password, if it can't fint the imaps password.  But not
the other way around, for security reasons.

Remind me if I forget to implement that when I reimplement nnimap.  :-)
(Perhaps next weekend-ish.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Is that normal?
>
> Could someone explain to me what all the utf-7 stuff around the IMAP
> code is?  Is IMAP defined as being in utf-7?

Yes.  Originally, it wasn't particularly Unicode, so UTF-7 encoding was added
to transport the mailbox names safely.

> That's like "er, what" just there...
> Anyway, the problems I'm having is that this takes forever:
> (encode-coding-string "hei" 'utf-7)

Works for me on 23.2.1, I fear.
        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Steinar Bang <sb@dod.no> writes:
>
>> Split incoming mail with procmail, or sieve, or the mail splitting tool
>> of your choice, directly into the imap server's folders.
>
> Yuck.  On the mail server?  Do people set up procmail rules still, in
> this day and age?

Sure.  Why not?  It has a well documented, simple and stable user
interface.  You can't beat a stable interface by creating a new one.


Bjørn

Re: That newfangled IMAP thing...

[Bjørn Mork]

Memnon Anon <gegendosenfleisch@googlemail.com> writes:

> ,----[ http://orgmode.org/worg/org-contribute.php#sec-2 ]

Thanks.  This helped me find
http://www.gnu.org/prep/maintain/html_node/Legal-Matters.html
as well.


Bjørn

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

On Sun, Sep 05 2010, Lars Magne Ingebrigtsen wrote: 

> Anybody seen this kind of behaviour before?  This is with Emacs 
> 24.

There probably had been a problem in emacs 24 at some point.

I've restarted Gnus on emacs-snapshot version 20100903 (Debian 
package from Romain Francoise) and got bitten by the same problem.

For now I came back to the snapshot before, waiting for the next one. :-)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Bjørn Mork]

Bjørn Mork <bjorn@mork.no> writes:
> Bjørn Mork <bjorn@mork.no> writes:
>
>> Hmm, looks like the nnimap splitting disappeared completely with the
>> latest git pull.  Probably faster, but still...
>
> Nevermind, it was already fixed.  Just needed another pull.

Nope, that was not it.  Seems to work on initial connect, but then start
failing when doing g.  However, M-g on the INBOX runs the splitting.

There's something fishy here.


Bjørn

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sun, 05 Sep 2010 18:59:25 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> I have to say that I find the nnimap/imap code slightly opaque...
>> things just seem to...  happen.

LMI> It's really complicated code, especially the things that deal with the
LMI> network connection and filters and stuff, which just seems kinda
LMI> unnecessary.  I mean, the protocol is dead simple.  You issue a
LMI> (numbered) command, and you get a (numbered) response.

LMI> At this point I feel I would have gotten further if I had just started
LMI> from scratch and written a new set of primitives.

LMI> This may not be totally compatible with the old version, but does
LMI> anybody mind?

I think it's a good direction.  The body structure and encapsulation
commands will drive you insane, though.  That portion of the RFC was
designed to inflict mental pain.

Ted

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sun, 05 Sep 2010 21:58:44 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Steinar Bang <sb@dod.no> writes:
>> I always thought it was the intended behaviour.  Not neccessary logical,
>> but that was the way it was (looking for the imaps password as imap).

LMI> imaps would be cleaner, but let's keep the old behaviour.

IMO nnimap.el should just check auth-source-user-or-password and drop
the direct call to the netrc parses.  Then the user can configure
auth-sources appropriately.

In general I just suggest dropping the specific protocol in authinfo
unless there's a specific need to have it.  auth-source-user-or-password
will pick the general entry if there's no overriding specific one.
That's better than trying to cleverly do "imaps first, then fall back to
imap, then maybe pop3, and eventually nntp" as these rules tend to
evolve.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> IMO nnimap.el should just check auth-source-user-or-password and drop
> the direct call to the netrc parses.  Then the user can configure
> auth-sources appropriately.

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Mon, 06 Sep 2010 15:01:38 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> IMO nnimap.el should just check auth-source-user-or-password and drop
>> the direct call to the netrc parses.  Then the user can configure
>> auth-sources appropriately.

LMI> Yup.

...so if and when you do the nnimap rewrite, just change the let from

	   (auth-user (nth 0 auth-info))
	   (auth-passwd (nth 1 auth-info))
	   (user (or
		  auth-user ; this is preferred to netrc-*
		  (netrc-machine-user-or-password
		   "login"
		   list
		   (list server
			 (or nnimap-server-address
			     nnimap-address))
		   (list port)
		   (list "imap" "imaps" "143" "993"))))
	   (passwd (or
		    auth-passwd ; this is preferred to netrc-*
		    (netrc-machine-user-or-password
		     "password"
		     list
		     (list server
			   (or nnimap-server-address
			       nnimap-address))
		     (list port)
		     (list "imap" "imaps" "143" "993")))))

to

	   (user (nth 0 auth-info))
	   (passwd (nth 1 auth-info))

and let me handle the "omg Gnus is b0rken" e-mails.

Ted

Re: That newfangled IMAP thing...

[Romain Francoise]

Julien Danjou <julien@danjou.info> writes:

> I've restarted Gnus on emacs-snapshot version 20100903 (Debian
> package from Romain Francoise) and got bitten by the same problem.

> For now I came back to the snapshot before, waiting for the next
> one. :-)

If emacs-snapshot is too broken to be usable I can usually unbreak
it right away, but someone needs to let me know... :-)

In this case doing (encode-coding-string "hei" 'utf-7) appears to be
working fine for me and I had no other report of breakage. Is some
special environment required to reproduce the bug?

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

On Mon, Sep 06 2010, Romain Francoise wrote: 

> If emacs-snapshot is too broken to be usable I can usually 
> unbreak it right away, but someone needs to let me know... :-) 
> In this case doing (encode-coding-string "hei" 'utf-7) appears 
> to be working fine for me and I had no other report of breakage. 
> Is some special environment required to reproduce the bug? 

I don't know but this is what I got at startup:
Error during redisplay: (error Lisp nesting exceeds 
`max-lisp-eval-depth') imap: Could not UTF7 encode 
`lists.guile-devel', using it unencoded...  Error during 
redisplay: (error Lisp nesting exceeds `max-lisp-eval-depth') 
imap: Could not UTF7 encode `lists.frnog', using it unencoded...

etc.

(I have to press C-g to to stop it blocking on each folder, but in 
the end Gnus does not start).

    (encode-coding-string "hei" 'utf-7)

Runs for a couple of seconds and then:
     when: Lisp nesting exceeds `max-lisp-eval-depth'

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> Nope, that was not it.  Seems to work on initial connect, but then start
> failing when doing g.  However, M-g on the INBOX runs the splitting.
>
> There's something fishy here.

I think I see why this is happening.  The imap splitting is done from
`nnimap-request-scan', which is called with the group name as the
parameter.  `g' no longer does this, but I guess it should.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> At this point I feel I would have gotten further if I had just started
> from scratch and written a new set of primitives.
>
> This may not be totally compatible with the old version, but does
> anybody mind?

I would guess that for new users, IMAP is the most common protocol they
will use, since it is offered by most large organizations, is
well-documented, has good features for distributed access, etc.  So
excellent IMAP support would be extremely good for Gnus.

If you really intend to tackle this, I have a few comments:

- Be sure to test with multiple IMAP servers.  Gmail has an IMAP
  interface grafted on as an afterthought, and it has various oddities.  
  I would use dovecot or another common server as the main platform
  for testing.

- The backend should know that things can change behind it's back.

- It'd be great to get accurate unread counts.

- Article editing isn't directly possible, but it should be possible
  to replace an article with a new one (e.g. to strip attachments or
  do other edits).  I've posted patches that do this, but they don't
  correctly update the summary buffer or transfer the marks over to
  the new article.  Here is the thread:

    http://thread.gmane.org/gmane.emacs.gnus.general/69094

- I haven't updated my Gnus in a few months, but I believe there is a 
  bug in nnimap when moving an article to a group on the same server:
  if the marks on the article have changed, but haven't been saved,
  then the moved article still gets the old marks.

- I wonder if nnimap should have a variable which makes changes to
  marks, etc be saved as you read the group, rather than when you exit
  it?  Since other devices/programs may monitor the group (e.g. a new
  mail notification daemon, a cell phone, etc) having this information
  updated continuously could be useful.

Dan

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I think I see why this is happening.  The imap splitting is done from
> `nnimap-request-scan', which is called with the group name as the
> parameter.  `g' no longer does this, but I guess it should.

This is now fixed and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Gijs Hillenius]

On  6 Sep 2010, Romain Francoise wrote:


[...]

> If emacs-snapshot is too broken to be usable I can usually unbreak
> it right away, but someone needs to let me know... :-)
>
> In this case doing (encode-coding-string "hei" 'utf-7) appears to be
> working fine for me and I had no other report of breakage. Is some
> special environment required to reproduce the bug?


adding (encode-coding-string "hei" 'utf-7) to .emacs does not help me
here with the emacs-snapshot from 2010-09-03.

Gnus takes forever to get the mail. I can send debug if that is helpful?

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Gijs Hillenius <gijs@hillenius.net> writes:

> adding (encode-coding-string "hei" 'utf-7) to .emacs does not help me
> here with the emacs-snapshot from 2010-09-03.

It's not supposed to.  :-)  If your Emacs bugs out when eval'ing that
statement, then your Emacs is buggy.  If it doesn't, it's not.

> Gnus takes forever to get the mail. I can send debug if that is helpful?

If the statement works, but Gnus is slow, it's a Gnus bug.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> - Be sure to test with multiple IMAP servers.  Gmail has an IMAP
>   interface grafted on as an afterthought, and it has various oddities.  
>   I would use dovecot or another common server as the main platform
>   for testing.

Yes, we have dovecot at work, so I'll be testing with that and Gmail.
Are there other common implementations that are worth thinking about? 

> - It'd be great to get accurate unread counts.

This is allegedly possible, so I'll be keeping that in mind.

(Hey!  I was wondering why my writing was error-free all of a sudden,
and then I noticed that the *Messages* buffer said

Starting new Ispell process [british] ...
Enabling Flyspell mode gave an error

Which explains that.)

> - Article editing isn't directly possible, but it should be possible
>   to replace an article with a new one (e.g. to strip attachments or
>   do other edits).  I've posted patches that do this, but they don't
>   correctly update the summary buffer or transfer the marks over to
>   the new article.  Here is the thread:
>
>     http://thread.gmane.org/gmane.emacs.gnus.general/69094

Looks reasonable.

> - I wonder if nnimap should have a variable which makes changes to
>   marks, etc be saved as you read the group, rather than when you exit
>   it?  Since other devices/programs may monitor the group (e.g. a new
>   mail notification daemon, a cell phone, etc) having this information
>   updated continuously could be useful.

I don't see why not.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Dehennin]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

Julien Danjou <julien@danjou.info> writes:


[...]

> I don't know but this is what I got at startup:
> Error during redisplay: (error Lisp nesting exceeds
> max-lisp-eval-depth') imap: Could not UTF7 encode `lists.guile-devel',
> using it unencoded...  Error during redisplay: (error Lisp nesting
> exceeds `max-lisp-eval-depth') imap: Could not UTF7 encode
> lists.frnog', using it unencoded...
>
> etc.
>
> (I have to press C-g to to stop it blocking on each folder, but in the
> end Gnus does not start).

I encounter the same problem with an old Gnus snapshot (20100607) on
orebokech debian emacs-snapshot (20100903).

I have the problem when replying to this mail, if I start a new emacs
and then Gnus, there is no problem, as soon as I try to follow up, it
gives me:

imap: Could not UTF7 encode `INBOX', using it unencoded... [3 times]
imap: Could not UTF7 encode `gnu.emacs.gnus.ding', using it
unencoded... [3 times]

And I must stop it with C-g.

Regards.
-- 
Daniel Dehennin
Récupérer ma clef GPG:
gpg --keyserver pgp.mit.edu --recv-keys 0x6A2540D1

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Yes, we have dovecot at work, so I'll be testing with that and Gmail.
> Are there other common implementations that are worth thinking about? 

Exchange.

It's an important one for many gnus users, to read their work email
from.

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Daniel Dehennin <daniel.dehennin@baby-gnu.org> writes:

> I encounter the same problem with an old Gnus snapshot (20100607) on
> orebokech debian emacs-snapshot (20100903).

Romain has confirmed that the Emacs snapshot is buggy, and he's building
a new one.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Exchange.
>
> It's an important one for many gnus users, to read their work email
> from.

Right.  I don't have access to any Exchange IMAP servers, though...
could someone set one up, put some messages on it and give me access?
:-) 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> Exchange.
>> 
>> It's an important one for many gnus users, to read their work email
>> from.

> Right.  I don't have access to any Exchange IMAP servers, though...
> could someone set one up, put some messages on it and give me access?
> :-)

If you get access to one, you might be interested in trying this one as
well:  http://davmail.sourceforge.net/

It's used to get access to those Exchange servers with admins that won't
open for IMAP (oh yes, they _do_ exist!).  I know it is used by at least
one gnus user.

Of course the way you're going now, you'll probably just write an
nnwebdav backend and be done with it...?

(WebDav access is always open because that's used by Outlook in some
cases, and the MS email program for the Mac (can't remember what its
called), and the Exchange ActiveX client for IE)

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Of course the way you're going now, you'll probably just write an
> nnwebdav backend and be done with it...?

Well, I'm not very interested in writing interfaces to Microsoft
stuff... 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Well, I'm not very interested in writing interfaces to Microsoft
> stuff... 

Well, it was intended as a joke.

I wouldn't really want to distract you from an improved/rewritten
nnimap.

Re: That newfangled IMAP thing...

[Daniel Pittman]

Steinar Bang <sb@dod.no> writes:
>>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:
>
>> Yes, we have dovecot at work, so I'll be testing with that and Gmail.
>> Are there other common implementations that are worth thinking about?
>
> Exchange.  It's an important one for many gnus users, to read their work
> email from.

I would also encourage you to add Cyrus IMAP, which is probably easy because
it is nice about many things, and the Zimbra IMAP server.

The last would be valuable: it uses non-monotonous IMAP UID values, so while
they meet the spec they *don't* do it the same way that other servers do.

(eg: a three item mailbox may have UIDs 1, 12, and 47 without anything but
 three emails being delivered in.)

Because of this it returns much larger and more sparse UID ranges than other
IMAP servers.  This caused me *serious* performance problems with the current
NNIMAP because every `gnus-expand-range' call would result in several hundred
MB of memory allocation...


So, yeah: a good test, and highly standards compliant with different
implementation decisions.  They have a FOSS version relatively easily
installable, although it likes to take over the system it is running on.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> I would also encourage you to add Cyrus IMAP, which is probably easy because
> it is nice about many things, and the Zimbra IMAP server.
>
> The last would be valuable: it uses non-monotonous IMAP UID values, so while
> they meet the spec they *don't* do it the same way that other servers do.
>
> (eg: a three item mailbox may have UIDs 1, 12, and 47 without anything but
>  three emails being delivered in.)

Sounds interesting.

Anybody have accounts on Zimbra and Cyrus for me to test with?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> I would also encourage you to add Cyrus IMAP, which is probably easy
>> because it is nice about many things, and the Zimbra IMAP server.
>>
>> The last would be valuable: it uses non-monotonous IMAP UID values, so
>> while they meet the spec they *don't* do it the same way that other servers
>> do.
>>
>> (eg: a three item mailbox may have UIDs 1, 12, and 47 without anything but
>>  three emails being delivered in.)
>
> Sounds interesting.  Anybody have accounts on Zimbra and Cyrus for me to
> test with?  :-)

I can give you an account on a Zimbra IMAP server, if you like; contact me
privately for the details.  It should be good for at least the next couple of
months — possibly more, depending on if these overseas job offers come through
or not.

It is my personal production mail server, so please be nice to it, and it
currently sits at the end of an ADSL line with ~ 1Mbit upstream, but it should
be sufficient for the purpose.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Gijs Hillenius]

On  7 Sep 2010, Daniel Pittman wrote:


[...]

>> Sounds interesting.  Anybody have accounts on Zimbra and Cyrus for me to
>> test with?  :-)
>
> I can give you an account on a Zimbra IMAP server, if you like;
> contact me privately for the details.  It should be good for at least
> the next couple of months — possibly more, depending on if these
> overseas job offers come through or not.
>
> It is my personal production mail server, so please be nice to it, and
> it currently sits at the end of an ADSL line with ~ 1Mbit upstream,
> but it should be sufficient for the purpose.

Ditto all this re Cyrus

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Anybody have accounts on Zimbra and Cyrus for me to test with?  :-)

I had a cyrus imapd as my private server, but I swapped it for dovecot
and never looked back.

But if you have a spare debian box, it's apt-get installable there.

Re: That newfangled IMAP thing...

[Wes Hardaker]

>>>>> On Sun, 05 Sep 2010 13:20:27 +0200, Lars Magne Ingebrigtsen <larsi@gnus.org> said:

>> Split incoming mail with procmail, or sieve, or the mail splitting tool
>> of your choice, directly into the imap server's folders.

LMI> Yuck.  On the mail server?  Do people set up procmail rules still, in
LMI> this day and age?

Procmail sucks (and needs to be rewritten).

Getting mail without procmail would suck far worse.


[Sieve looks to be a rewritten procmail, but I wouldn't call the
language more user-friendly and there aren't really any great gui
editors for it yet to make it easy for the masses.  At least that I know
of, which doesn't mean much.  Forget I mentioned it] 
-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: That newfangled IMAP thing...

[Richard Riley]

Wes Hardaker <wes@hardakers.net> writes:

>>>>>> On Sun, 05 Sep 2010 13:20:27 +0200, Lars Magne Ingebrigtsen <larsi@gnus.org> said:
>
>>> Split incoming mail with procmail, or sieve, or the mail splitting tool
>>> of your choice, directly into the imap server's folders.
>
> LMI> Yuck.  On the mail server?  Do people set up procmail rules still, in
> LMI> this day and age?
>
> Procmail sucks (and needs to be rewritten).

What sucks about procmail? It's used throughout the world and heartily
recommended by many. Certainly my (admittedly minor) use of it suggested
it was very competent, reliable and easy to use.

Re: That newfangled IMAP thing...

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yes, we have dovecot at work, so I'll be testing with that and Gmail.
> Are there other common implementations that are worth thinking about? 

Courier


Bjørn

Re: That newfangled IMAP thing...

[Wes Hardaker]

>>>>> On Tue, 07 Sep 2010 16:52:37 +0200, Richard Riley <rileyrg@googlemail.com> said:

RR> What sucks about procmail? It's used throughout the world and heartily
RR> recommended by many. Certainly my (admittedly minor) use of it suggested
RR> it was very competent, reliable and easy to use.

Don't get me wrong, I use it:

  wc -l .procmailrc .procmailrc.imap 
    245 .procmailrc
    675 .procmailrc.imap
    920 total

But if we were designing it from scratch again I suspect we'd change the
language format quite a bit based on lessons learned.

EG, the ":0" backwards-compatibility solution/problem would probably go away.
The sad thing is that we probably could go back to ":" at this point
since I'm 99.999% sure that no one is running the older version it was
written to allow backwards-compatibility with.

(I also still like sendmail, but I've still switched to postfix)

-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: That newfangled IMAP thing...

[James Cloos]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> Yes, we have dovecot at work, so I'll be testing with that and Gmail.
LMI> Are there other common implementations that are worth thinking about? 

dbmail.  Upstream uses sparse UIDs, but I run it patched to use dense UIDs.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> James Cloos <cloos@jhcloos.com>:

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
LMI> Yes, we have dovecot at work, so I'll be testing with that and Gmail.
LMI> Are there other common implementations that are worth thinking about? 

> dbmail.  Upstream uses sparse UIDs, but I run it patched to use dense UIDs.

Archiveopteryx http://archiveopteryx.org/

Re: That newfangled IMAP thing...

[Vegard Vesterheim]

On Tue, 07 Sep 2010 20:16:23 +0200 Steinar Bang <sb@dod.no> wrote:

>>>>>> James Cloos <cloos@jhcloos.com>:
>
>>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> LMI> Yes, we have dovecot at work, so I'll be testing with that and Gmail.
> LMI> Are there other common implementations that are worth thinking about? 
>
>> dbmail.  Upstream uses sparse UIDs, but I run it patched to use dense UIDs.
>
> Archiveopteryx http://archiveopteryx.org/

Yes, both dbmail and Archiveopteryx are interesting solutions for
IMAP, so having good Gnus support for these could be nice. 

There are several different dimensions along which you could organise
email (sender, receiver, subject, date, content, manual sorting, etc),
and you often need to combine them: "Show me all email with subject
'Gnus' sent from Lars the last month". If you have presorted the email
into folders, you need to figure out which folders you need to search
to find these messages. I find this concept of presorting email into
actual folders cumbersome, one email message may fit into several
different folders. Imagine being able to just stuff all your email
into one large folder, and then having a good search engine to access
this. Then you could define a set of "virtual folders" as needed, a
virtual folder being no more than a predefined IMAP query. Many email
clients support something like this, but I would like to see a good
server side solution, so that you have access to the same set of
folders on different clients. I think both dbmail and archiveopteryx
are promising candidates for something like this.

I know that most of is does not directly affect the IMAP client, but I
would just like to bring this concept of "server-side virtual folders"
into consideration. I guess there are several thorny issues here, how
to make this effective/fast enough, how to administer such virtual
folders via the client, etc. But I find this concept very interesting.

 - Vegard V -

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Vegard Vesterheim <vegard.vesterheim@uninett.no>:

> ... I find this concept of presorting email into actual folders
> cumbersome, one email message may fit into several different
> folders. Imagine being able to just stuff all your email into one
> large folder, and then having a good search engine to access this.

FWIW opera mail has done this, and google/gmail has done this, and I
never got comfortable with either.

The presorting I do is split mail for mailing lists.  Crossposted emails
end up in the first matching rule and gets followups only for that list.

I don't try to manually sort email later, but I move/expire mail into
archive folders (I back all high-traffic folders with archive folders),
and I would have liked search that spans both the folder and its archive
folder.

Re: That newfangled IMAP thing...

[Vegard Vesterheim]

On Wed, 08 Sep 2010 11:06:49 +0200 Steinar Bang <sb@dod.no> wrote:

>>>>>> Vegard Vesterheim <vegard.vesterheim@uninett.no>:
>
>> ... I find this concept of presorting email into actual folders
>> cumbersome, one email message may fit into several different
>> folders. Imagine being able to just stuff all your email into one
>> large folder, and then having a good search engine to access this.
>
> FWIW opera mail has done this, and google/gmail has done this, and I
> never got comfortable with either.

Yes, I think the later versions of thunderbird also does some
aggressive indexing of email in order to do advanced and effective
searching. Some people seem to like this.

You can of course keep the old folder structure, but you would just
define them as virtual folders instead. Your email-client doesn't
necessarily need to be aware of this as long as the search/retrieval
is done effectively.

> The presorting I do is split mail for mailing lists.  Crossposted emails
> end up in the first matching rule and gets followups only for that list.
>
> I don't try to manually sort email later, but I move/expire mail into
> archive folders (I back all high-traffic folders with archive folders),
> and I would have liked search that spans both the folder and its archive
> folder.

Reminds me of another problem with separate folders. I tend to archive
my replies in the same folder as the original message to keep the
thread intact. My regular outgoing email is archived in the "Sent"
Folder. Whenever I want to search for my own messages, I often find it
hard to decide which folders to search in.

 - Vegard V -

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

On Wed, Sep 08 2010, Vegard Vesterheim wrote:

> Yes, I think the later versions of thunderbird also does some
> aggressive indexing of email in order to do advanced and effective
> searching. Some people seem to like this.

Since we're talking about searching, Dovecot IMAP server has nice
IMAP-implemented-features making search on the server possible with
SEARCH BODY, and even fast fulltext:

         http://wiki2.dovecot.org/Plugins/FTS/Squat

I don't think nnimap support IMAP SEARCH commands. It would be nice in
the future.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Vegard Vesterheim <vegard.vesterheim@uninett.no>:

> Reminds me of another problem with separate folders. I tend to archive
> my replies in the same folder as the original message to keep the
> thread intact.

AOL.

> My regular outgoing email is archived in the "Sent" Folder. Whenever I
> want to search for my own messages, I often find it hard to decide
> which folders to search in.

I use BCC and let the same mail filtering as handles incoming email do
the splitting.  With nnimap groups holding the messages from/to a
mailing list, I let the mailing list send me a copy, and skip the BCC
(because if I get my own messages back from the list, I know that the
list has received it).

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Julien Danjou <julien@danjou.info>:

> I don't think nnimap support IMAP SEARCH commands. It would be nice in
> the future.

It does.  Or rather: nnir does.  Here's the secondary-select-methods
entry for my private dovecot server:

      '((nnimap "privat"
		(nnimap-address "myhost.mydomain.no")
                (nnir-search-engine imap)
                (nnimap-stream ssl))

Position the cursor over an nnimap group, and do `G G' and type in a
search string.  This will create a temporary group with the matches.

I would have liked to:
 - Make the search span both that particular group, as well as its
   archive group (which I normally have hidden)
 - Make it possible to navigate from a match to the match inside the
   actual group, so that 
     - It is possible to tick it, if it is unticked
     - I will get the usual group parameters and gnus-posting-styles for
       the group (I would also have liked to have those match the actual
       group, even if the article is found in the archive group), so
       that I can respond correctly to the article

The reason I use an archive group is that (the current) nnimap has a
problem with groups with a large number of articles.  There's also a
general gnus problem with groups with a large number of ticked
articles... well there used to be a problem... eg. back in 2001 on emacs
20 on machines with 24MB of memory.  Not neccessarily today with emacs
22 or 23 and 1GB or more of memory.

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Wed, Sep 08 2010, Steinar Bang wrote:

>>>>>> Julien Danjou <julien@danjou.info>:
>
> It does.  Or rather: nnir does.  Here's the secondary-select-methods
> entry for my private dovecot server:
>
>       '((nnimap "privat"
> 		(nnimap-address "myhost.mydomain.no")
>                 (nnir-search-engine imap)
>                 (nnimap-stream ssl))

Thanks for the tip!

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Vegard Vesterheim]

On Wed, 08 Sep 2010 14:06:22 +0200 Steinar Bang <sb@dod.no> wrote:

> Position the cursor over an nnimap group, and do `G G' and type in a
> search string.  This will create a temporary group with the matches.

Yes, I use this all the time. Unfortunately it is a bit slow,
especially if there are many hits. I find that adding a 'SINCE <date>'
clause in the IMAP SEARCH helps speed up things when I want to find
recent articles. In fact I have defined a simple function which does
this for me, and I have defined the keybinding as 'G g' for this.

>
> I would have liked to:
>  - Make the search span both that particular group, as well as its
>    archive group (which I normally have hidden)

You can mark several groups and issue a nnir/imap search for those.

>  - Make it possible to navigate from a match to the match inside the
>    actual group, so that 

I guess that is what gnus-summary-nnir-goto-thread ('G T') is supposed
to do, it is documented like this:

;; When looking at the retrieval result (in the Summary buffer) you
;; can type `G T' (aka M-x gnus-summary-nnir-goto-thread RET) on an
;; article.  You will be teleported into the group this article came
;; from, showing the thread this article is part of.  (See below for
;; restrictions.)

 - Vegard V -

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Vegard Vesterheim <vegard.vesterheim@uninett.no>:

> Yes, I use this all the time. Unfortunately it is a bit slow,
> especially if there are many hits. I find that adding a 'SINCE <date>'
> clause in the IMAP SEARCH helps speed up things when I want to find
> recent articles. In fact I have defined a simple function which does
> this for me, and I have defined the keybinding as 'G g' for this.

Hm, ok.  Thanx for the tip.  Unfortunately when I need to search for
something I just have the vaguest idea of when the message I'm searching
for happened.

> You can mark several groups and issue a nnir/imap search for those.

Yes, but I don't really want to see the archive group.  I just want to
search in the frontend group and include the archive group.

> I guess that is what gnus-summary-nnir-goto-thread ('G T') is supposed
> to do, it is documented like this:

Yes, but I've never gotten it to work.  It never completes, and when I
eventually interrupt it's in an ephemeral summary buffer and not in the
real group summary buffer.

Re: That newfangled IMAP thing...

[Wes Hardaker]

>>>>> On Wed, 08 Sep 2010 13:50:02 +0200, Steinar Bang <sb@dod.no> said:

>>>>> Vegard Vesterheim <vegard.vesterheim@uninett.no>:
>> Reminds me of another problem with separate folders. I tend to archive
>> my replies in the same folder as the original message to keep the
>> thread intact.

SB> AOL.

I've meant to (but haven't yet) written the small bit of elisp needed to
sit above my non-archive folder and mark both it and the archive one
(which uses a derived name) and then perform the search so you get both
folders...

One of these days...
-- 
Wes Hardaker                                     
My Pictures:  http://capturedonearth.com/
My Thoughts:  http://pontifications.hardakers.net/

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I think it's a good direction.  The body structure and encapsulation
> commands will drive you insane, though.  That portion of the RFC was
> designed to inflict mental pain.

Speaking of the body structure part of IMAP...

One thing that has been a desired feature since nnimap's first days:
partial download of messages.  In particular: to be able to read the
text parts of a message and delay the download of big attachments.

This was very important 10 years ago, but is probably less so today.
Lines are generally faster, and people are less likely to send big
attachments.  Instead they send a URL.

But since this is a capability of the protocol, I think it's worth
considering.  The nnimap backend leaning on the message structure
knowledge of the server may have other benefits than partial
downloads...? 

Previous discussions of the subject:
 http://thread.gmane.org/gmane.emacs.gnus.general/52154/focus=52411
 http://thread.gmane.org/gmane.emacs.gnus.general/45839
 http://thread.gmane.org/gmane.emacs.gnus.general/68821/focus=68861

Re: That newfangled IMAP thing...

[Bjørn Mork]

Steinar Bang <sb@dod.no> writes:

> This was very important 10 years ago, but is probably less so today.
> Lines are generally faster, and people are less likely to send big
> attachments.  Instead they send a URL.

You should get more Exchange exposure.  

Not only do people attach large documents containing mostly nonsense
(like a PowerPoint meeting agenda with a couple pictures), but
Outlook/Exchange also help them ensuring that the attachments are copied
to every followup, like when the schedule or venue is updated.


Bjørn

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> One thing that has been a desired feature since nnimap's first days:
> partial download of messages.  In particular: to be able to read the
> text parts of a message and delay the download of big attachments.

If doing client-side mail splitting, I think one would want to skip
downloading attachments, at least...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Steinar Bang <sb@dod.no> writes:
>
>> One thing that has been a desired feature since nnimap's first days:
>> partial download of messages.  In particular: to be able to read the
>> text parts of a message and delay the download of big attachments.
>
> If doing client-side mail splitting, I think one would want to skip
> downloading attachments, at least...

When I used Gnus attached to a GPRS data connection, from Thailand, or when I
use it that way from a bench in Melbourne over a nasty, terrible 3G connection
that makes that GPRS system look fast I would also love it.

Being able to delay loading of non-inline body parts would be *great* for a
bunch of reasons, and a non-trivial number of those are about more mobile
devices that can and frequently do run Emacs.

        Daniel

Well, OK, maybe just /my/ mobile devices frequently run Emacs. ;)
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Mon, 06 Sep 2010 17:28:39 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> I think I see why this is happening.  The imap splitting is done from
>> `nnimap-request-scan', which is called with the group name as the
>> parameter.  `g' no longer does this, but I guess it should.

LMI> This is now fixed and pushed.

Running with the latest, my splitting rules are not applied to INBOX
with nnimap (I have multiple IMAP servers).  My setup is the same as
I posted previously as an example of splitting rules.  Should I start
digging or is it something obvious?

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Running with the latest, my splitting rules are not applied to INBOX
> with nnimap (I have multiple IMAP servers).  My setup is the same as
> I posted previously as an example of splitting rules.  Should I start
> digging or is it something obvious?

I've looked over the code, and I can't see which of my changes caused
this problem.  If you could have a look at it, it'd be great.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Thu, 09 Sep 2010 16:58:33 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Running with the latest, my splitting rules are not applied to INBOX
>> with nnimap (I have multiple IMAP servers).  My setup is the same as
>> I posted previously as an example of splitting rules.  Should I start
>> digging or is it something obvious?

LMI> I've looked over the code, and I can't see which of my changes caused
LMI> this problem.  If you could have a look at it, it'd be great.

I'll take a look when I can.  

More trouble: after using Courier IMAP a bit, my nnmaildir primary
backend didn't believe it had any groups anymore, saying
"gnus-select-newsgroup: Couldn't request group X: No such group: X".

I `q'uit Gnus and started it again without restarting Emacs and
everything worked again.  I'll debug if I see this again but can't now.

Ted

Re: That newfangled IMAP thing...

[Sebastian Krause]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> Steinar Bang <sb@dod.no> writes:
>> One thing that has been a desired feature since nnimap's first days:
>> partial download of messages.  In particular: to be able to read the
>> text parts of a message and delay the download of big attachments.
>
> If doing client-side mail splitting, I think one would want to skip
> downloading attachments, at least...

Now that we're talking about advanced IMAP features, is there any
possibilty of supporting IMAP IDLE? Or is it totally against the
concept of how Gnus works?

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Sebastian Krause <sebastian@realpath.org> writes:

> Now that we're talking about advanced IMAP features, is there any
> possibilty of supporting IMAP IDLE? Or is it totally against the
> concept of how Gnus works?

The RFC doesn't mention IDLE.  Do you have a pointer?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Jason L Tibbitts III]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

LMI> The RFC doesn't mention IDLE.  Do you have a pointer?

It's RFC2177, http://www.faqs.org/rfcs/rfc2177.html

 - J<

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

"Jason L Tibbitts III" <tibbs@math.uh.edu> writes:

> It's RFC2177, http://www.faqs.org/rfcs/rfc2177.html

So you send IDLE after selecting a mailbox, and you're given updates
about new messages (and stuff) happening in that mailbox?  The Gnus
response would be to mark articles as cancelled, and insert new summary
lines.

I don't think this sounds particularly useful.  I almost never sit
inside a single group -- I'm always in the group buffer.  I don't
think it would be very difficult to implement, actually (FSVO
"difficult"), but I think it sounds more neat than essential.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> "Jason L Tibbitts III" <tibbs@math.uh.edu> writes:
>
>> It's RFC2177, http://www.faqs.org/rfcs/rfc2177.html
>
> So you send IDLE after selecting a mailbox, and you're given updates
> about new messages (and stuff) happening in that mailbox?  The Gnus
> response would be to mark articles as cancelled, and insert new summary
> lines.
>
> I don't think this sounds particularly useful.  I almost never sit
> inside a single group -- I'm always in the group buffer.  I don't
> think it would be very difficult to implement, actually (FSVO
> "difficult"), but I think it sounds more neat than essential.

I see your point, but my usage is that I have INBOX, where mail that (in
theory) I want to see quickly (also on phone) arrives, and many other
mailboxes for things that I want to look at with much higher latency.

Right now I have a mail-notification widget in the status tray that
tells me if INBOX has unread messages.  But it would be nice if the
unread count in INBOX automatically reflected what was happening in
IMAP, so that I could see the 1 from the arrived message and just hit
space to enter INBOX without typing g.  I would only want gnus to IDLE
on a tiny number of mailboxes, probably just 1.

But I don't want this badly enought to write it - just trying to explain
why it would make sense.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Greg Troxel <gdt@work.lexort.com> writes:

> Right now I have a mail-notification widget in the status tray that
> tells me if INBOX has unread messages.  But it would be nice if the
> unread count in INBOX automatically reflected what was happening in
> IMAP, so that I could see the 1 from the arrived message and just hit
> space to enter INBOX without typing g.  I would only want gnus to IDLE
> on a tiny number of mailboxes, probably just 1.

Ah, right, you could IDLE on a mailbox without being in the group.  But
you can only IDLE on one mailbox at a time?  (Without having several
connections open, that is.)

But updating the group buffer lines on IDLE would make perfect sense, I
think, and be useful.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Greg Troxel <gdt@work.lexort.com> writes:
>
>> Right now I have a mail-notification widget in the status tray that
>> tells me if INBOX has unread messages.  But it would be nice if the
>> unread count in INBOX automatically reflected what was happening in
>> IMAP, so that I could see the 1 from the arrived message and just hit
>> space to enter INBOX without typing g.  I would only want gnus to IDLE
>> on a tiny number of mailboxes, probably just 1.
>
> Ah, right, you could IDLE on a mailbox without being in the group.  But
> you can only IDLE on one mailbox at a time?  (Without having several
> connections open, that is.)

Yes.  That makes it essentially useless for anyone doing server-side splitting
without additional work, or one connection per folder.

The LEMONADE working group, who are trying to get IETF standards for "mobile
push email" style stuff, are working on improvements, and someone proposed a
"multi-IDLE" that would deal with all mailboxes, but none of those are
standard.  (...or implemented, AFAIK, by any real IMAP servers.)

> But updating the group buffer lines on IDLE would make perfect sense, I
> think, and be useful.

For folks who used Gnus-side splitting it would be valuable that the IMAP
implementation returned to the INBOX and sat in IDLE, so it could also trigger
automatic filtering inside Gnus.

This would be double-plus extra nice if they wanted to use Gnus splitting
*and* be able to use some other client talking to that IMAP server.

My partner, for complicated historical reasons, usually has her computer at
home running Thunderbird when we travel, since she does client-side filtering
with that, and doesn't want that stopped while reading mail through the web
interface only.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sat, 11 Sep 2010 13:42:36 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 

DP> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> But updating the group buffer lines on IDLE would make perfect sense, I
>> think, and be useful.

DP> For folks who used Gnus-side splitting it would be valuable that the IMAP
DP> implementation returned to the INBOX and sat in IDLE, so it could also trigger
DP> automatic filtering inside Gnus.

Yes, if it only supports one folder at a time, let it be INBOX.  But it
shouldn't do anything expensive like splitting by default, only update
visual indicators like the mail icon in Emacs and perhaps call the D-Bus
notification system (see http://www.emacswiki.org/emacs/notify.el, it's
pretty easy[1]).  notify.el would also be nice for end-of-splitting and
you-have-mail events in general, not just for IMAP.

Ted

[1]

(defun notify-via-dbus (title body)
  "Send notification with TITLE, BODY `D-Bus'."
  (dbus-call-method :session "org.freedesktop.Notifications"
		    "/org/freedesktop/Notifications"
		    "org.freedesktop.Notifications" "Notify"
		    (get 'notify-defaults :app)
		    (setq notify-id (+ notify-id 1))
		    (get 'notify-defaults :icon) title body '(:array)
		    '(:array :signature "{sv}") ':int32
		    (get 'notify-defaults :timeout)))

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Sat, Sep 11 2010, Ted Zlatanov wrote:

> Yes, if it only supports one folder at a time, let it be INBOX.  But it
> shouldn't do anything expensive like splitting by default, only update
> visual indicators like the mail icon in Emacs and perhaps call the D-Bus
> notification system (see http://www.emacswiki.org/emacs/notify.el, it's
> pretty easy[1]).  notify.el would also be nice for end-of-splitting and
> you-have-mail events in general, not just for IMAP.

FWIW, Emacs 24 has notifications.el included, that implements the
desktop notification specification properly and entirely.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

I've now gotten the basics up and running, and I have an implementation
of `nnimap-request-list' that works for all the test servers.  (Well --
"works" -- it gives an upper bound on how many articles might exist in
each group, but it's really fast.)

Next I'll do group entry and article selection, and then it's time to
implement the new group info function that will return real ranges of
unread/read/unseen articles directly to Gnus without going via the
*nntpd* buffer.  It might be called...  er...
`nnimap-request-group-data'? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Next I'll do group entry and article selection, and then it's time to
> implement the new group info function that will return real ranges of
> unread/read/unseen articles directly to Gnus without going via the
> *nntpd* buffer.  It might be called...  er...
> `nnimap-request-group-data'? 

Eurhm.  I'm reading the RFC, and I'm wondering whether I'm seriously
missing something.

Is the client supposed to do a

EXAMINE foo
FETCH 1:* (FLAGS)

on each single mailbox to get all the article data?  If you have a
million emails on the server (and who hasn't?), that'll like take
forever.  Fer sure.

What?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

On Sat, Sep 11 2010, Lars Magne Ingebrigtsen wrote:

> on each single mailbox to get all the article data?  If you have a
> million emails on the server (and who hasn't?), that'll like take
> forever.  Fer sure.

Forever?

FWIW it took about one second in a 12K mails mailbox, with Dovecot 1.2
as server. Over the Internet.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eurhm.  I'm reading the RFC, and I'm wondering whether I'm seriously
> missing something.
>
> Is the client supposed to do a
>
> EXAMINE foo
> FETCH 1:* (FLAGS)

I mean, the client has to do that once, at least, but is there anything
in the protocol that the client can use to not do a complete rescan the
next time?  The UIDNEXT response to the EXAMINE call will increase if a
new article has been added, which helps, since the client can ignore
scanning groups where that hasn't changed, but I scanning the RFC, I
can't find anything to help with the question "has any flags changed"?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Next I'll do group entry and article selection,

I've now gotten all the basics seemingly working on the servers I've
tried it with.  I haven't implemented article moving yet, so not all the
servers have articles yet, but I have to say that IMAP is a lot easier
to work with than I imagined, just based on the code size.  imap.el +
nnimap.el were like 4600 lines, and the new nnimap.el is (at this
moment) 406 lines...

It'll triple in size before it's done, but so far, so pleasant. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It'll triple in size before it's done, but so far, so pleasant. 

I now have move/copy/accept going, too, so the next thing to handle is
probably client-side mail splitting.

So I guess you'd have a variable that says which mailbox to use as the
source mailbox, which should just default to "INBOX".  And then you
split according to ...  Hm.  The old nnimap has its private
`nnimap-split-rule'.  Is that really necessary?  Can't we just use the
normal `nnmail-split-methods' and stuff?  Why should nnimap be different
from all the other mail backends?

Anyway, the splitting methodology would be as follows:

1) stream-download all the articles in the INBOX group
2) run the mail split rules over each article
3) notice what new groups should be created, if any, and create them
4) stream out a bunch of "COPY 23242 ding" (etc.) commands, to copy the
   articles over to the correct groups
5) parse the responses
6) mark all the messages that were successfully copied as deleted (this
   would normally be all the articles)
7) do an "EXPUNGE"

It should be a fairly quick process -- the main time spent should be in
the actual streaming download of the articles in 1).

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sat, 11 Sep 2010 13:47:43 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Sat, Sep 11 2010, Ted Zlatanov wrote:
>> Yes, if it only supports one folder at a time, let it be INBOX.  But it
>> shouldn't do anything expensive like splitting by default, only update
>> visual indicators like the mail icon in Emacs and perhaps call the D-Bus
>> notification system (see http://www.emacswiki.org/emacs/notify.el, it's
>> pretty easy[1]).  notify.el would also be nice for end-of-splitting and
>> you-have-mail events in general, not just for IMAP.

JD> FWIW, Emacs 24 has notifications.el included, that implements the
JD> desktop notification specification properly and entirely.

Cool, thanks for letting me know.

If Lars doesn't do it I'll keep it in mind.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If Lars doesn't do it I'll keep it in mind.

So far I've only done a totally synchronous (but streaming) IMAP
interface.  But to support gnus-async stuff (for article pre-loading),
I'll have to add some process filters and stuff, and perhaps it would be
natural to have the async connection return to IDLE-ing on the INBOX
whenever it doesn't have any work to do?  And then add some callbacks to
notify.el.

I'm looking forward to using the built-in TLS stuff you're doing for
Emacs 24, though, because now (with four imaps nnimap servers), Gnus
startup is overwhelmingly dominated by "openssl s_client" startup (via
tls.el).  The built-in stuff will be much faster?  Right?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> It'll triple in size before it's done, but so far, so pleasant.
>
> I now have move/copy/accept going, too, so the next thing to handle is
> probably client-side mail splitting.
>
> So I guess you'd have a variable that says which mailbox to use as the
> source mailbox, which should just default to "INBOX".  And then you
> split according to ...  Hm.  The old nnimap has its private
> `nnimap-split-rule'.  Is that really necessary?  Can't we just use the
> normal `nnmail-split-methods' and stuff?  Why should nnimap be different
> from all the other mail backends?
>
> Anyway, the splitting methodology would be as follows:
>
> 1) stream-download all the articles in the INBOX group
> 2) run the mail split rules over each article
> 3) notice what new groups should be created, if any, and create them
> 4) stream out a bunch of "COPY 23242 ding" (etc.) commands, to copy the
>    articles over to the correct groups
> 5) parse the responses
> 6) mark all the messages that were successfully copied as deleted (this
>    would normally be all the articles)
> 7) do an "EXPUNGE"

Watch out: an unconditional EXPUNGE here might not do what the user expects,
since it will destroy all mail deleted, not just the stuff that Gnus handled.

Unless you have UID EXPUNGE, better to either make that an option in Gnus or
just avoid it.  Er, and see my other response in the thread WRT where UID
EXPUNGE can be found.

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Next I'll do group entry and article selection, and then it's time to
>> implement the new group info function that will return real ranges of
>> unread/read/unseen articles directly to Gnus without going via the *nntpd*
>> buffer.  It might be called...  er...  `nnimap-request-group-data'?
>
> Eurhm.  I'm reading the RFC, and I'm wondering whether I'm seriously
> missing something.
>
> Is the client supposed to do a
>
> EXAMINE foo
> FETCH 1:* (FLAGS)
>
> on each single mailbox to get all the article data?

Kind-of.  Plain-jane IMAP is really bare about these things; what you want to
do is query capabilities from the server at startup, then select the
appropriate mechanisms for action from that.

For example, the EXPUNGE after sync discussed elsewhere probably means you
want to demand (unconditionally, because nothing doesn't implement) UIDPLUS
from the server.

That adds a bunch of UID versions of commands that otherwise need to reflect
the id in the current fetch, making life *much* faster for you.


Anyway, to sync flags you want this:

    https://tools.ietf.org/html/rfc5162#section-5

That would be the official upstream documentation about the QRESYNC variant of
the "flag sync" process, and includes how to handle less capable servers.
Given that "less capable" means "most" — since only Zimbra has QRESYNC that
I deal with regularly...


Anyway, CONDSTORE capability is pretty routine, and most things will have
that; I would suggest you stash away a pointer to the right routine somewhere
in the server thing every login, after doing a capabilities query.

> If you have a million emails on the server (and who hasn't?), that'll like
> take forever.  Fer sure.

Yah.  CONDSTORE makes it much faster.
        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Sun, Sep 12 2010, Daniel Pittman wrote:

> That would be the official upstream documentation about the QRESYNC variant of
> the "flag sync" process, and includes how to handle less capable servers.
> Given that "less capable" means "most" — since only Zimbra has QRESYNC that
> I deal with regularly...

Dovecot seems to have it too, FWIW. :)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Daniel Pittman]

Julien Danjou <julien@danjou.info> writes:
> On Sun, Sep 12 2010, Daniel Pittman wrote:
>
>> That would be the official upstream documentation about the QRESYNC variant of
>> the "flag sync" process, and includes how to handle less capable servers.
>> Given that "less capable" means "most" — since only Zimbra has QRESYNC that
>> I deal with regularly...
>
> Dovecot seems to have it too, FWIW. :)

Ah, cool.  It didn't when I last used it, but it is good to see it getting
more wide-spread support.  It changes the IMAP protocol pretty substantially,
but for the best.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Richard Riley]

I pulled nognus today and it takes an age to start :  I dont use
Agent. IMAP and nntp backends. I am also getting imap process errors and
need to quit gnus and restart it to reconnect the imap processes.

Julien Danjou <julien@danjou.info> writes:

> On Sun, Sep 12 2010, Daniel Pittman wrote:
>
>> That would be the official upstream documentation about the QRESYNC variant of
>> the "flag sync" process, and includes how to handle less capable servers.
>> Given that "less capable" means "most" — since only Zimbra has QRESYNC that
>> I deal with regularly...
>
> Dovecot seems to have it too, FWIW. :)

-- 
☘ http://www.shamrockirishbar.com, http://splash-of-open-sauce.blogspot.com/ http://www.richardriley.net

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> Watch out: an unconditional EXPUNGE here might not do what the user
> expects, since it will destroy all mail deleted, not just the stuff
> that Gnus handled.

Well, just from the INBOX, surely?  If you have defined one group as an,
er, inbox inbox, then you want to delete everything after splitting the
messages to the other groups.  Otherwise it doesn't make sense...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> For example, the EXPUNGE after sync discussed elsewhere probably means you
> want to demand (unconditionally, because nothing doesn't implement) UIDPLUS
> from the server.

There are no servers that don't implement UIDPLUS?

> Anyway, to sync flags you want this:
>
>     https://tools.ietf.org/html/rfc5162#section-5
>
> That would be the official upstream documentation about the QRESYNC variant of
> the "flag sync" process, and includes how to handle less capable servers.
> Given that "less capable" means "most" — since only Zimbra has QRESYNC that
> I deal with regularly...
>
> Anyway, CONDSTORE capability is pretty routine, and most things will have
> that; I would suggest you stash away a pointer to the right routine somewhere
> in the server thing every login, after doing a capabilities query.

CONDSTORE seems to work just like I had imagined that something like
that had to work.  Nice.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

When doing client-side splitting, would it be reasonable to only work on
the headers and the first section of each article?

That would be the following command (modulo capabilities):

FETCH 1:* (UID BODY.PEEK[HEADER] BODY.PEEK[1])

I'm guessing that for mail splitting purposes, it's never interesting to
work on attachments.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> When doing client-side splitting, would it be reasonable to only work on
> the headers and the first section of each article?

I think so.  (though I don't use client side splitting right now).

> I'm guessing that for mail splitting purposes, it's never interesting to
> work on attachments.

No... but what about people that use the INBOX for "leftover" messages?
It's what I did when I used client side splitting (I think... it's been
a while...).

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

>> I'm guessing that for mail splitting purposes, it's never interesting to
>> work on attachments.
>
> No... but what about people that use the INBOX for "leftover" messages?
> It's what I did when I used client side splitting (I think... it's been
> a while...).

I don't quite follow...

If you're splitting from the INBOX, then no messages will be left over,
normally.  (Unless you have a split rule that says that it shouldn't go
anywhere, that is.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>>> I'm guessing that for mail splitting purposes, it's never interesting to
>>> work on attachments.
>>
>> No... but what about people that use the INBOX for "leftover" messages?
>> It's what I did when I used client side splitting (I think... it's been
>> a while...).
>
> I don't quite follow...
>
> If you're splitting from the INBOX, then no messages will be left over,
> normally.  (Unless you have a split rule that says that it shouldn't go
> anywhere, that is.)

This isn't really how splitting for me works. I do something like

"To/CC" is my work address -> folder work
"Mailinglist-Header" is X -> folder mail X
...
all the rest stays in Inbox.

(However, I do server-side splitting)

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Frank Schmitt <ich@frank-schmitt.net> writes:

> This isn't really how splitting for me works. I do something like
>
> "To/CC" is my work address -> folder work
> "Mailinglist-Header" is X -> folder mail X
> ...
> all the rest stays in Inbox.

Right...  so the mail splitter should probably pull down all the flags
first, and skip anything that's been flagged as read or deleted before
doing anything.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Richard Riley]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>>> I'm guessing that for mail splitting purposes, it's never interesting to
>>> work on attachments.
>>
>> No... but what about people that use the INBOX for "leftover" messages?
>> It's what I did when I used client side splitting (I think... it's been
>> a while...).
>
> I don't quite follow...
>
> If you're splitting from the INBOX, then no messages will be left over,
> normally.  (Unless you have a split rule that says that it shouldn't go
> anywhere, that is.)

Most of my email does indeed stay in the inbox. I just use spam-split
currently  : this moves spam tagged to a spam folder. The rest stays
put.

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Right...  so the mail splitter should probably pull down all the flags
> first, and skip anything that's been flagged as read or deleted before
> doing anything.

Which means that it has to be careful about the EXPUNGE as mentioned in
a different posting.

I mean: even if you download them in a special place for doing the
splitting, you will want to delete all of the ones that have been COPY'd
in the server side INBOX.

But you don't want to touch the ones that have simply been deleted.

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Which means that it has to be careful about the EXPUNGE as mentioned in
> a different posting.

Yes, I'm only going to do UID EXPUNGE, for those servers that support
it, and let the users of servers without that expunge on their own.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ok, mail splitting is now implemented, and it's pretty snappy.  (FSVO.)
Now I just have to test it on all the different servers to see whether
it breaks...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Are there any IMAP server that don't have the "UID FETCH" (etc.)
commands? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> Watch out: an unconditional EXPUNGE here might not do what the user
>> expects, since it will destroy all mail deleted, not just the stuff
>> that Gnus handled.
>
> Well, just from the INBOX, surely?

Oh.  Sorry, yes, just from the INBOX mailbox, indeed.

> If you have defined one group as an, er, inbox inbox, then you want to
> delete everything after splitting the messages to the other groups.
> Otherwise it doesn't make sense...

I, like many IMAP users, treat my INBOX as precisely that: it collects things
that are not sent elsewhere.  It is not empty, but instead what is commonly
considered the "Inbox", where everything "else" goes.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> For example, the EXPUNGE after sync discussed elsewhere probably means you
>> want to demand (unconditionally, because nothing doesn't implement) UIDPLUS
>> from the server.
>
> There are no servers that don't implement UIDPLUS?

My understanding is that there are no significant IMAP servers without UIDPLUS
today, yes.

>> Anyway, to sync flags you want this:
>>
>>     https://tools.ietf.org/html/rfc5162#section-5
>>
>> That would be the official upstream documentation about the QRESYNC variant of
>> the "flag sync" process, and includes how to handle less capable servers.
>> Given that "less capable" means "most" — since only Zimbra has QRESYNC that
>> I deal with regularly...
>>
>> Anyway, CONDSTORE capability is pretty routine, and most things will have
>> that; I would suggest you stash away a pointer to the right routine somewhere
>> in the server thing every login, after doing a capabilities query.
>
> CONDSTORE seems to work just like I had imagined that something like that
> had to work.  Nice.  :-)

QRESYNC finishes the job, but CONDSTORE is pretty much the trick to making the
process efficient.  IIRC you still need the inefficient version, though,
against certain classes of CONDSTORE response.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Are there any IMAP server that don't have the "UID FETCH" (etc.) commands?

No, those are core parts of the IMAP4 protocol.  UIDPLUS grows them to cover
additional features, but the basic stuff is sound.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> My understanding is that there are no significant IMAP servers without UIDPLUS
> today, yes.

Our Dovecot has the following CAPABILITY:

("IMAP4rev1" "SASL-IR" "SORT" "THREAD=REFERENCES" "MULTIAPPEND"
"UNSELECT" "LITERAL+" "IDLE" "CHILDREN" "NAMESPACE" "LOGIN-REFERRALS")

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Daniel Pittman <daniel@rimspace.net> writes:

>> My understanding is that there are no significant IMAP servers
>> without UIDPLUS today, yes.

> Our Dovecot has the following CAPABILITY:

> ("IMAP4rev1" "SASL-IR" "SORT" "THREAD=REFERENCES" "MULTIAPPEND"
> "UNSELECT" "LITERAL+" "IDLE" "CHILDREN" "NAMESPACE" "LOGIN-REFERRALS")

What version is your dovecot?  According Timo Siriainen (the main author
of dovecot), it has been supported since version 1.1:
	http://search.gmane.org/?query=uidplus&group=gmane.mail.imap.dovecot

My debian stable ("lenny") dovecot is 1.0.15-2.3 (which doesn't support
UIDPLUS).

The current debian testing dovecot has upstream version 1.2.13.  The
Ubuntu 10.04 "lucid lynx" dovecot has upstream version 1.2.9.

Re: That newfangled IMAP thing...

[David Edmondson]

>>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:
>> Daniel Pittman <daniel@rimspace.net> writes:
>
>>> My understanding is that there are no significant IMAP servers
>>> without UIDPLUS today, yes.
>
>> Our Dovecot has the following CAPABILITY:
>
>> ("IMAP4rev1" "SASL-IR" "SORT" "THREAD=REFERENCES" "MULTIAPPEND"
>> "UNSELECT" "LITERAL+" "IDLE" "CHILDREN" "NAMESPACE" "LOGIN-REFERRALS")

I noticed that the declared list of capabilities changes after
authentication. Are you looking at the pre- or post-authentication list?

dme.
-- 
David Edmondson, http://dme.org

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> My debian stable ("lenny") dovecot is 1.0.15-2.3 (which doesn't support
> UIDPLUS).

Ours is also from Debian Stale.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

On Mon, Sep 13 2010, Lars Magne Ingebrigtsen wrote:

>> My debian stable ("lenny") dovecot is 1.0.15-2.3 (which doesn't support
>> UIDPLUS).
>
> Ours is also from Debian Stale.

FWIW, if you want 1.2 branch, you can get it from backports.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

David Edmondson <dme@dme.org> writes:

> I noticed that the declared list of capabilities changes after
> authentication. Are you looking at the pre- or post-authentication list?

Post.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Julien Danjou <julien@danjou.info>:

> On Mon, Sep 13 2010, Lars Magne Ingebrigtsen wrote:
>>> My debian stable ("lenny") dovecot is 1.0.15-2.3 (which doesn't support
>>> UIDPLUS).

>> Ours is also from Debian Stale.

> FWIW, if you want 1.2 branch, you can get it from backports.

Fairly straightforward instructions (and it's now an official debian
service): http://backports.debian.org/Instructions/

The backports dovecot currently has upstream version 1.2.13 as well.

As I understand it, unless backports has a security update currently in
unstable, the backport packages will be automatically upgraded when
stable rolls over from lenny to squeeze.  FAQ number 3:
	http://backports.debian.org/FAQ/

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sun, 12 Sep 2010 04:17:13 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Anyway, the splitting methodology would be as follows:

LMI> 1) stream-download all the articles in the INBOX group
LMI> 2) run the mail split rules over each article
LMI> 3) notice what new groups should be created, if any, and create them
LMI> 4) stream out a bunch of "COPY 23242 ding" (etc.) commands, to copy the
LMI>    articles over to the correct groups
LMI> 5) parse the responses
LMI> 6) mark all the messages that were successfully copied as deleted (this
LMI>    would normally be all the articles)
LMI> 7) do an "EXPUNGE"

LMI> It should be a fairly quick process -- the main time spent should be in
LMI> the actual streaming download of the articles in 1).

Can you make (3) and (4) also work for qualified group names outside the
nnimap server, so splitting can work across servers and backends?

On Sun, 12 Sep 2010 04:32:02 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> If Lars doesn't do it I'll keep it in mind.

LMI> So far I've only done a totally synchronous (but streaming) IMAP
LMI> interface.  But to support gnus-async stuff (for article pre-loading),
LMI> I'll have to add some process filters and stuff, and perhaps it would be
LMI> natural to have the async connection return to IDLE-ing on the INBOX
LMI> whenever it doesn't have any work to do?  And then add some callbacks to
LMI> notify.el.

s/notify/notifications/ apparently, but yes.  I wouldn't put too much
work into this.  It's less valuable than the functional improvements
you're making.

LMI> I'm looking forward to using the built-in TLS stuff you're doing for
LMI> Emacs 24, though, because now (with four imaps nnimap servers), Gnus
LMI> startup is overwhelmingly dominated by "openssl s_client" startup (via
LMI> tls.el).  The built-in stuff will be much faster?  Right?  :-)

It will be much faster.  It's a lot of work, though.  There's no "open
this connection and leave me alone" option.  You have chains of
certificates, examples full of "it's easy to make this work right,
if you could move to Greenland" and so on.

I plan to be able to open an unverified SSL or TLS connection before
dropping it into the core.  From that point on the internals may change
but I'll try to keep the API consistent.  Meanwhile, suffer :)

Ted

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Sun, 12 Sep 2010 17:12:39 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> When doing client-side splitting, would it be reasonable to only work on
LMI> the headers and the first section of each article?

LMI> That would be the following command (modulo capabilities):

LMI> FETCH 1:* (UID BODY.PEEK[HEADER] BODY.PEEK[1])

LMI> I'm guessing that for mail splitting purposes, it's never interesting to
LMI> work on attachments.

For spam.el (spam-split) and spam-stat.el, the first section is
sufficient IMO.  There are edge cases with HTML attachments or an
enclosed message but I don't think those are worth the effort.

There's a "statistical" concept in spam.el that also distinguishes
between the spam backends that look at the whole article vs. those that
only care about the headers.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Fairly straightforward instructions (and it's now an official debian
> service): http://backports.debian.org/Instructions/

Yes, but I don't want that.  :-)  nnimap should work on IMAP servers
without UIDPLUS, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> LMI> 3) notice what new groups should be created, if any, and create them
> LMI> 4) stream out a bunch of "COPY 23242 ding" (etc.) commands, to copy the
> LMI>    articles over to the correct groups

[...]

> Can you make (3) and (4) also work for qualified group names outside the
> nnimap server, so splitting can work across servers and backends?

The mail splitting works on the assumption that only a single backend
"receives" the email, so that would take a bit of work.  It's actually
worse in the nnimap case than in all the other mail backend cases, since
nnimap doesn't download complete messages before doing the split, so it
would have to re-request each complete message (or download complete
messages in the first place) to move it to a different backend.

> I plan to be able to open an unverified SSL or TLS connection before
> dropping it into the core.

Great!

> From that point on the internals may change but I'll try to keep the
> API consistent.  Meanwhile, suffer :)

:-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Steinar Bang <sb@dod.no> writes:
>
>> Fairly straightforward instructions (and it's now an official debian
>> service): http://backports.debian.org/Instructions/
>
> Yes, but I don't want that.  :-)  nnimap should work on IMAP servers
> without UIDPLUS, too.

Aw, heck.  So they should, and I am slightly sad to learn that is the case.
(Plus, it means that if I did finish my hopefully-not-so-awful IMAP<=>IMAP
 sync tool I would also need to handle non-UIDPLUS servers. :/ )

        Daniel

-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Daniel Pittman <daniel@rimspace.net>:

> Aw, heck.  So they should, and I am slightly sad to learn that is the case.
> (Plus, it means that if I did finish my hopefully-not-so-awful IMAP<=>IMAP
>  sync tool I would also need to handle non-UIDPLUS servers. :/ )

Of course, soon (for some values of "soon"), debian stable will tick
over, and all the dovecots on debian stable will become UIDPLUS capable.

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Mon, 13 Sep 2010 13:09:39 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
LMI> 3) notice what new groups should be created, if any, and create them
LMI> 4) stream out a bunch of "COPY 23242 ding" (etc.) commands, to copy the
LMI> articles over to the correct groups

LMI> [...]

>> Can you make (3) and (4) also work for qualified group names outside the
>> nnimap server, so splitting can work across servers and backends?

LMI> The mail splitting works on the assumption that only a single backend
LMI> "receives" the email, so that would take a bit of work.  It's actually
LMI> worse in the nnimap case than in all the other mail backend cases, since
LMI> nnimap doesn't download complete messages before doing the split, so it
LMI> would have to re-request each complete message (or download complete
LMI> messages in the first place) to move it to a different backend.

At the time (a while ago) I had this near the top of my TODO list, I
planned to set up a temporary group and put articles in it, then move
them out in a post-split hook and delete the group.  That would work
despite the lack of complete messages and should work in every backend
that supports splitting.  It will require a special destination header
on each article, I think.

It's not ideal if the process gets interrupted, but worst-case you have
the temporary group full of articles you can move back to INBOX.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

I was thinking about the various...  levels...  of new-article/flag
scanning to be performed against IMAP servers to get a reasonable `g'
behaviour.

(The assumption is a new interface function that will update all new
articles/flags on subscribed nnimap groups.)

1) If it's a newfangled server with the right extensions, getting all
new articles and new flags for all subscribed articles is fast, so
that's a solved problem.

2) If it's a server without the extensions, but it's a server with
client-side splitting, then we need only check the groups that we know
we have received new articles in on `g'.

3) If it's neither 1) or 2), we issue an EXAMINE per group, and if the
NEXTUID hasn't changed, then we don't bother to examine the flags.

I think this will be reasonable default behaviour, and give acceptable
speed on `g' for most users.  If a user wants total synchronisation
between the IMAP server and the Gnus view on all flags, then there can
be a `nnimap-be-really-slow' flag to say that Gnus should check all
groups in depth in cases 2) and 3) every time.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> 1) If it's a newfangled server with the right extensions, getting all
> new articles and new flags for all subscribed articles is fast, so
> that's a solved problem.

Er.  No.

I had only skimmed the QRESYNC stuff.  Basically, on EXAMINE, you say
what the last sequence number was, and you get new flags.  Fine.  But
there's still no way to say which mailboxes you need to EXAMINE.  It an
EXAMINE takes a while.  On imap.gmail.com, for instance, each EXAMINE
takes about 0.1s (and that's with streaming).  If you have 20 groups, a
`g' will take 2 seconds, as a bare minimum.  That's way too slow to be
fun. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

I now mostly have unoptimised flags/marks sync between Gnus and the IMAP
servers.  That is, if you mark something in a Gnus newsgroup, the marks
gets translated into a flag on the server, and if you set a flag on an
article on the server, `g' will bring that mark to Gnus.

But I need to finish QRESYNC support and partial resync, too, which I'm
not going to finish tonight.  Geeze!  Things take more time than you
think...

I'll probably finish the code tomorrow, and should do thorough testing
on all the servers a bit before unleashing the new nnimap on the world.

And Wednesday and Thursday evenings I'm all busy and stuff, so I'll
probably push this on ... Saturday.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Knut Anders Hatlen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I now mostly have unoptimised flags/marks sync between Gnus and the IMAP
> servers.  That is, if you mark something in a Gnus newsgroup, the marks
> gets translated into a flag on the server, and if you set a flag on an
> article on the server, `g' will bring that mark to Gnus.

Are you also storing Gnus-specific flags, like expire and dormant, on
the server? If so, you may want to look out for servers that don't allow
storing custom flags. Look for the presence of \* in the PERMANENTFLAGS
response. I think the old nnimap stored these flags on the server if
supported, and only locally in .newsrc.eld otherwise.

Looking forward to testing it! :)

-- 
Knut Anders

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Knut Anders Hatlen <kahatlen@gmail.com> writes:

> Are you also storing Gnus-specific flags, like expire and dormant, on
> the server?

It's not doing so now, because I didn't think that would be useful.  But
it would if you were using Gnus to read from several different
locations, wouldn't it?

> If so, you may want to look out for servers that don't allow storing
> custom flags. Look for the presence of \* in the PERMANENTFLAGS
> response. I think the old nnimap stored these flags on the server if
> supported, and only locally in .newsrc.eld otherwise.

Is there some name space that one should use if storing private flags?
Like $MyFlag or something?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Knut Anders Hatlen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>
>> Are you also storing Gnus-specific flags, like expire and dormant, on
>> the server?
>
> It's not doing so now, because I didn't think that would be useful.  But
> it would if you were using Gnus to read from several different
> locations, wouldn't it?

Yes, it would. The IMAP server I use the most doesn't support it, and
that's a bit annoying when I switch to a different machine. But as long
as the flags are stored locally and Gnus doesn't fail when attempting to
store them on the server, I'm happy.

>> If so, you may want to look out for servers that don't allow storing
>> custom flags. Look for the presence of \* in the PERMANENTFLAGS
>> response. I think the old nnimap stored these flags on the server if
>> supported, and only locally in .newsrc.eld otherwise.
>
> Is there some name space that one should use if storing private flags?
> Like $MyFlag or something?

Kind of. The system flags start with a backslash and the private flags
don't. (See 2.3.2 Flags Message Attribute in RFC 3501.) The old nnimap
used gnus-expire, gnus-dormant and so on.

-- 
Knut Anders

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:

>> Are you also storing Gnus-specific flags, like expire and dormant, on
>> the server?

> It's not doing so now, because I didn't think that would be useful.
> But it would if you were using Gnus to read from several different
> locations, wouldn't it?

Oh yes!  It's an important part of my usage pattern.  Currently three
different gnusen sharing the same folders.

Re: That newfangled IMAP thing...

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>
>> Are you also storing Gnus-specific flags, like expire and dormant, on
>> the server?
>
> It's not doing so now, because I didn't think that would be useful.  But
> it would if you were using Gnus to read from several different
> locations, wouldn't it?

Yes, very useful!  Being able to do so is the entire reason I use imap
at all.  It would be great if you kept the same flag names as the
current nnimap.

Dan

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

I just had the most brilliant idea (if I must say so myself) while
trying to fall asleep.

The way to get flags and stuff on `g' from the IMAP servers is to stream
"EXAMINE" and "FETCH 1:* FLAGS" (and variants thereof) commands on all
the subscribed groups, and then wait for all the data to arrive, and
then parse it and do stuff.  So 99% of the time is spent in
`process-accept-output'.

Here's the brilliant idea:

The network backends could have their
get-new-data-on-all-subscribed-groups functions split in two:  One
"early" bit and one "finish" bit.

So instead of doing

(dolist (method methods)
  (get-data method))

I'll rewrite that to  
  
(dolist (method methods)
  (get-data-early method))

(dolist (method methods)
  (get-data-finish method))

That way, data will be arriving from all the network methods (i.e., IMAP
and NNTP) in parallel, and reading 100 groups from 10 nntp/imap servers
each should be only vaguely slower than reading 1000 groups from 1
server.

And it should be trivial to implement.
  
-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> That way, data will be arriving from all the network methods (i.e., IMAP
> and NNTP) in parallel, and reading 100 groups from 10 nntp/imap servers
> each should be only vaguely slower than reading 1000 groups from 1
> server.

Actually, it should be much, much faster.  Today it's somewhat slower,
but with the change, it should be only vaguely slower than reading 100
groups from 1 server, I mean.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Actually, it should be much, much faster.  Today it's somewhat slower,
> but with the change, it should be only vaguely slower than reading 100
> groups from 1 server, I mean.

By golly.  It worked.  There's now zero waiting for the IMAP servers in
my setup.  So if the parsing and stuff is fast and stuff, then typical
NNTP + IMAP usage should be pretty snappy.  I hope.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Dan Christensen <jdc@uwo.ca>:

> ...  It would be great if you kept the same flag names as the current
> nnimap.

AOL!  My maildir+ dovecot mailboxes already contains heaps of articles
with the current custom flag names.

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Mon, 13 Sep 2010 23:55:25 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>> Are you also storing Gnus-specific flags, like expire and dormant, on
>> the server?

LMI> It's not doing so now, because I didn't think that would be useful.  But
LMI> it would if you were using Gnus to read from several different
LMI> locations, wouldn't it?

Yes.  There are other uses.  I could use it for gnus-registry.el and
gnus-sync.el storage, for instance.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Yes, very useful!  Being able to do so is the entire reason I use imap
> at all.  It would be great if you kept the same flag names as the
> current nnimap.

Sure.  They're called gnus-*.

But here's what I can't quite wrap my head around: When do you sync
these, and when don't you?

To take an example: You're using the FooIMAP server, which doesn't have
"*" in the PERMANENTFLAGS, so you don't store the Gnus flags.  Then
somebody upgrades FooIMAP, and then "*" arrives in PERMANENTFLAGS.  Now
we want to sync the extra flags, too.  Since, naturally, the first time
you use it after the upgrade, there's no gnus-* flags on the server, so
nnimap deletes all your local flags.

That's not very nice behaviour.

So...  nnimap has to store whether the group supports uploading flags.
If that status changes from "" to "*", then nnimap has to upload the
extra flags for all the articles.  Or something.  It seems somewhat
fragile.

Any ideas?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On Tue, Sep 14 2010, Lars Magne Ingebrigtsen wrote:

> Any ideas?

My idea would be that flag storage should be *always* done in the
backend (for mails at least :-).
So the backend is the authority for the flags.

In your example, if the server does not support flags, then, no flags
for you. When it will support it, well, you will haz flags.

My 2¢,

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> So...  nnimap has to store whether the group supports uploading flags.
> If that status changes from "" to "*", then nnimap has to upload the
> extra flags for all the articles.  Or something.  It seems somewhat
> fragile.

> Any ideas?

If you always also store them locally in the .newsrc.eld file, you can
compare them when reading the group.

If status is "*", some of the articles have custom flags in the local
information, but not in what's read from the IMAP mailbox, then it would
be the first time reading from a server that has been upgraded.

Of course, that's a lot of comparison to do, for something that will
only happen once in a while, and when it has happened almost never will
reoccur on that server.

It's probably a corner case, though.

I'm guessing that in most cases those servers present that already have
this capability will continue to have it, and those that don't, won't
get it.

But that's guessing since I don't know which ones support it, and which
ones don't.

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 17:10:10 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> But here's what I can't quite wrap my head around: When do you sync
LMI> these, and when don't you?

LMI> To take an example: You're using the FooIMAP server, which doesn't have
LMI> "*" in the PERMANENTFLAGS, so you don't store the Gnus flags.  Then
LMI> somebody upgrades FooIMAP, and then "*" arrives in PERMANENTFLAGS.  Now
LMI> we want to sync the extra flags, too.  Since, naturally, the first time
LMI> you use it after the upgrade, there's no gnus-* flags on the server, so
LMI> nnimap deletes all your local flags.

LMI> That's not very nice behaviour.

LMI> So...  nnimap has to store whether the group supports uploading flags.
LMI> If that status changes from "" to "*", then nnimap has to upload the
LMI> extra flags for all the articles.  Or something.  It seems somewhat
LMI> fragile.

This is so rare I wouldn't worry about it.  But you could make a special
case of "the server has NO flags BUT you have many AND you're
about to lose them" which would require yes-or-no confirmation.  I had
the same concerns with gnus-sync.el, which is why I plan to give it a UI
and much better merge logic.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> This is so rare I wouldn't worry about it.  But you could make a special
> case of "the server has NO flags BUT you have many AND you're
> about to lose them" which would require yes-or-no confirmation.  I had
> the same concerns with gnus-sync.el, which is why I plan to give it a UI
> and much better merge logic.

The user answers `n', and then reads a group.  Gnus then sets the
`gnus-forwarded' flag on an article.  Then the user hits `g'.  There's
now one gnus-specific mark in the group, so the test doesn't trigger,
and all the other marks are wiped out.  :-)

I think, as a first approximation, at least, that I'll just add a
`dont-sync' entry in the marks list if the group doesn't support
PERMANENTMARKS (for user-defined marks), and if that's present, it'll
never sync those marks, even if the server later supports the marks.
Perhaps. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ok, I think I've figured this out while walking home.

nnimap marks all the groups whether they are "*" or not.  Then, if it
sees that 1) the status has flipped from off to on, and 2) there are no
gnus-* marks on the server, and 3) there are marks in Gnus, then it just
uploads all the Gnus marks.  That's pretty simple to do.

The only problem is that to see whether a mailbox is "*" or not, nnimap
has to issue SELECT instead of EXAMINE for all the groups, and I don't
know whether that has a performance impact or not...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> The only problem is that to see whether a mailbox is "*" or not,
> nnimap has to issue SELECT instead of EXAMINE for all the groups, and
> I don't know whether that has a performance impact or not...

Well, as both Ted and I have said: this will be a pretty rare
occurrance.

And I think Julien's solution was good: if the server can't handle the
custom flags, disallow setting the custom flags in that server's
groups. 

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> And I think Julien's solution was good: if the server can't handle the
> custom flags, disallow setting the custom flags in that server's
> groups. 

Uhm.  The question was what to do if a server switches from "can't
store" to "can store".

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 18:21:49 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> The only problem is that to see whether a mailbox is "*" or not, nnimap
LMI> has to issue SELECT instead of EXAMINE for all the groups, and I don't
LMI> know whether that has a performance impact or not...

It's supposed to be slower and you can't batch it like EXAMINE.  Maybe
you can checksum the server capabilities as a group attribute in
newsrc.eld?

Ted

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> And I think Julien's solution was good: if the server can't handle the
>> custom flags, disallow setting the custom flags in that server's
>> groups. 

> Uhm.  The question was what to do if a server switches from "can't
> store" to "can store".

If you disallow saving custom flags when the value is "can't store" you
have no syncing problem if the server is upgraded to "can store".

The server is always the master.

You also avoid questions like "why does one of my gnusen show custom
marks while the others don't?", or "why does my gnusen have different
custom marks?".

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It's supposed to be slower and you can't batch it like EXAMINE.  Maybe
> you can checksum the server capabilities as a group attribute in
> newsrc.eld?

Perhaps it would suffice to ask one group once.  Is it likely that one
group would support "*" PERMANENTFLAGS, but other groups don't?  In
other words, is this (in real life) something that is a server-wide
capability, really, and not a mailbox parameter?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> If you disallow saving custom flags when the value is "can't store" you
> have no syncing problem if the server is upgraded to "can store".

Er.  I don't follow you.

> The server is always the master.

Except when it doesn't store flags.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 17:44:31 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> This is so rare I wouldn't worry about it.  But you could make a special
>> case of "the server has NO flags BUT you have many AND you're
>> about to lose them" which would require yes-or-no confirmation.  I had
>> the same concerns with gnus-sync.el, which is why I plan to give it a UI
>> and much better merge logic.

LMI> The user answers `n', and then reads a group.  Gnus then sets the
LMI> `gnus-forwarded' flag on an article.  Then the user hits `g'.  There's
LMI> now one gnus-specific mark in the group, so the test doesn't trigger,
LMI> and all the other marks are wiped out.  :-)

LMI> I think, as a first approximation, at least, that I'll just add a
LMI> `dont-sync' entry in the marks list if the group doesn't support
LMI> PERMANENTMARKS (for user-defined marks), and if that's present, it'll
LMI> never sync those marks, even if the server later supports the marks.
LMI> Perhaps. 

I think it's a really good idea to consider using gnus-sync.el as the
way to synchronize marks at this point.  Right now the backend choice is
just a file name, but we could have

((server1 'native) ; nnimap internal sync
 (server2 "file-path") ; external file (the only option in the current gnus-sync.el)
 (server3 "nnimap:server1:INBOX.synchronizer") ; use an IMAP server to hold the marks
 (server4 nil) ; native or 'newsrc, whatever is available
 (server5 'newsrc) ; just newsrc.eld
)

with nil (the default) equivalent to ((t nil)) meaning all servers use
their native marks sync method if available and fall back to 'newsrc.

If we move sync logic to a central place, we'll be able to reuse it for
all backends and DTRT in a general way instead of a nnimap-specific
way.  nntp, nnrss, etc. could really use that.

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> If we move sync logic to a central place, we'll be able to reuse it for
> all backends and DTRT in a general way instead of a nnimap-specific
> way.  nntp, nnrss, etc. could really use that.

But is it fast?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 21:24:22 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> It's supposed to be slower and you can't batch it like EXAMINE.  Maybe
>> you can checksum the server capabilities as a group attribute in
>> newsrc.eld?

LMI> Perhaps it would suffice to ask one group once.  Is it likely that one
LMI> group would support "*" PERMANENTFLAGS, but other groups don't?  In
LMI> other words, is this (in real life) something that is a server-wide
LMI> capability, really, and not a mailbox parameter?

Yes, I would just do an opportunistic SELECT.  Which makes it fast and
you don't need to store checksums or anything like that.

Ted

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

On Tue, Sep 14 2010, Ted Zlatanov wrote:

> If we move sync logic to a central place, we'll be able to reuse it for
> all backends and DTRT in a general way instead of a nnimap-specific
> way.  nntp, nnrss, etc. could really use that.

+1

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

On Tue, Sep 14 2010, Lars Magne Ingebrigtsen wrote:

> Perhaps it would suffice to ask one group once.  Is it likely that one
> group would support "*" PERMANENTFLAGS, but other groups don't?  In
> other words, is this (in real life) something that is a server-wide
> capability, really, and not a mailbox parameter?

You can have different mailboxes type on a IMAP server. For example, you
can have public mailboxes that you can only read but cannot write into,
or a shared mailbox where you can also do only some things your
co-worker give you the rights to do via some ACL.

So possibly some mailboxes won't have PERMANENTFLAGS.

Dovecot has some capabilities to make such setup with different
namespaces and all. I'm rather sure it still *can* allow PERMANENTFLAGS
almost everywhere (public, shared, private mailboxes), but well, in
theory you can't be sure.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes, I would just do an opportunistic SELECT.  Which makes it fast and
> you don't need to store checksums or anything like that.

Ok, I'll try that.

I've finally finished basic flag syncing now, but not QRESYNC handling.
Trying to avoid working on huge lists and working on ranges instead
(when selecting (e.g.) the last 100 articles to do a flags sync) kinda
makes my brain go "ouchie".

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> in theory you can't be sure.

That's why I was asking for how it is in reality.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Dan Christensen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I think it's a really good idea to consider using gnus-sync.el as the
> way to synchronize marks at this point.

I don't see how that addresses Lars' problem.  We still do want to be
able to store marks on the IMAP server using custom flags (when
supported), right?  Like many others, I currently run dovecot both on my
laptop and on a server, and keep them synced with offlineimap, so it's
important to me that the marks be natively stored using imap flags.
(Not to mention the archives of messages I have that have marks stored
using imap flags.)

Dan

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 21:28:40 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> If we move sync logic to a central place, we'll be able to reuse it for
>> all backends and DTRT in a general way instead of a nnimap-specific
>> way.  nntp, nnrss, etc. could really use that.

LMI> But is it fast?  :-)

When it falls back to native, it's as fast as the native method (minus a
few funcalls here and below)

When it falls back to 'newsrc if 'native doesn't have sync support, it's
as fast as now (e.g. nntp).

When it uses an external load file it's as fast as a newsrc read/write
unless the file is remote.  In that case the network time dominates.

If the external store is a nnimap mailbox, it's as fast as native
nnimap.  Here I'm thinking of storing fake messages with the particular
message-ID we are saving and setting flags on them.  And of course the
external store could be a database or memcached-style key-val store.

The value is in the uncommon use cases, like the "are you SURE you want
to wipe all your marks even though the server could be wrong?".  You'd
write the sync logic once and reuse it everywhere.

Ted

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 21:40:24 +0200 Julien Danjou <julien@danjou.info> wrote: 

JD> On Tue, Sep 14 2010, Lars Magne Ingebrigtsen wrote:
>> Perhaps it would suffice to ask one group once.  Is it likely that one
>> group would support "*" PERMANENTFLAGS, but other groups don't?  In
>> other words, is this (in real life) something that is a server-wide
>> capability, really, and not a mailbox parameter?

JD> You can have different mailboxes type on a IMAP server. For example, you
JD> can have public mailboxes that you can only read but cannot write into,
JD> or a shared mailbox where you can also do only some things your
JD> co-worker give you the rights to do via some ACL.

JD> So possibly some mailboxes won't have PERMANENTFLAGS.

JD> Dovecot has some capabilities to make such setup with different
JD> namespaces and all. I'm rather sure it still *can* allow PERMANENTFLAGS
JD> almost everywhere (public, shared, private mailboxes), but well, in
JD> theory you can't be sure.

I think that's over-engineering.  Do a SELECT on INBOX and be done.  If
we find edge cases we'll fix them, but it's not worth complicating
everyone's life for that extremely rare situation.

Ted

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Except when it doesn't store flags.  :-)

It can always store some of the flags in this list
 http://tools.ietf.org/html/rfc3501#section-2.3.2
(at least \Seen and \Deleted )

If not, client would have a hard time working.

The PERMANENTFLAGS response says exactly which ones are supported: 
 http://tools.ietf.org/html/rfc3501#section-7.1

If the PERMANENTFLAGS response contains \* then the server is able to
persist non-standard flags.  Ie. flags not listed in section 2.3.2 or
its own FLAGS response, and flags that may just have meaning to gnus and
no other IMAP clients.

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 15:44:15 -0400 Dan Christensen <jdc@uwo.ca> wrote: 

DC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I think it's a really good idea to consider using gnus-sync.el as the
>> way to synchronize marks at this point.

DC> I don't see how that addresses Lars' problem.  We still do want to be
DC> able to store marks on the IMAP server using custom flags (when
DC> supported), right?  Like many others, I currently run dovecot both on my
DC> laptop and on a server, and keep them synced with offlineimap, so it's
DC> important to me that the marks be natively stored using imap flags.
DC> (Not to mention the archives of messages I have that have marks stored
DC> using imap flags.)

The storage is easy if the backend supports it (which is why it's been
done like that so far).  The value is when we reuse the logic for
merging and resolving conflicts and we let users store marks outside of
the newsrc.eld (for NNTP access from home vs. work for instance).

Ted

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> It can always store some of the flags in this list

Yes, I know.  This discussion has been about "*", only.  That is,
client-specific flags.  The standard flags are irrelevant to the
discussion. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> The storage is easy if the backend supports it (which is why it's been
> done like that so far).  The value is when we reuse the logic for
> merging and resolving conflicts and we let users store marks outside of
> the newsrc.eld (for NNTP access from home vs. work for instance).

My vision is that Gnus in Emacs 24 should be so snappy and usable that a
normal nerdy programmer user would prefer using Gnus to, er, what,
Mozilla?  Or whatever the kids these days are using.

So a typical setup with a couple of IMAP servers and a couple of NNTP
servers should use less than 1 second on a `g', preferably much less
than that.  For NNTP that's easy enough, since it just gives you all the
data you need on a platter.  For IMAP, it's more difficult, since
there's a lot more data to transfer (i.e., all the flags per message).

I think I'm on the right track there with the streaming flags and stuff,
and a general gnus-sync thing, while sounding very useful, I can't
really see achieving that speed.  I'm cutting all the corners I can to
get Gnus snappy.  :-)

But that doesn't mean that Gnus couldn't use gnus-sync stuff in addition
to doing it in as sloppy manner as possible with "native" nnimap sync.

Of course, after getting nnimap up to speed, and ironing out the rest of
the problems that turned up after I rewrote the `g' function (I'm sure
there are more even though Katsumi has fixed a bunch), is writing an
HTML renderer in Emacs Lisp.  Then people can read their HTML email from
their IMAP servers without any problems, and Gnus will be
feature-complete.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Yes, I know.  This discussion has been about "*", only.  That is,
> client-specific flags.  The standard flags are irrelevant to the
> discussion. 

Ok.

The proposal is like this: if you can't save custom flags, live with
limited behaviour on that server (or folder).

Re: That newfangled IMAP thing...

[Ted Zlatanov]

On Tue, 14 Sep 2010 22:07:14 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> The storage is easy if the backend supports it (which is why it's been
>> done like that so far).  The value is when we reuse the logic for
>> merging and resolving conflicts and we let users store marks outside of
>> the newsrc.eld (for NNTP access from home vs. work for instance).

LMI> My vision is that Gnus in Emacs 24 should be so snappy and usable that a
LMI> normal nerdy programmer user would prefer using Gnus to, er, what,
LMI> Mozilla?  Or whatever the kids these days are using.

Pretty much everyone I know uses Outlook or web clients.  The GMail IMAP
support makes a big difference.

LMI> I think I'm on the right track there with the streaming flags and stuff,
LMI> and a general gnus-sync thing, while sounding very useful, I can't
LMI> really see achieving that speed.  I'm cutting all the corners I can to
LMI> get Gnus snappy.  :-)

LMI> But that doesn't mean that Gnus couldn't use gnus-sync stuff in addition
LMI> to doing it in as sloppy manner as possible with "native" nnimap sync.

OK, that's fine.  I may have to do some work in that area later then,
but definitely don't let it stop you.

Ted

Re: That newfangled IMAP thing...

[Frank Schmitt]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 14 Sep 2010 22:07:14 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
>
> LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> The storage is easy if the backend supports it (which is why it's been
>>> done like that so far).  The value is when we reuse the logic for
>>> merging and resolving conflicts and we let users store marks outside of
>>> the newsrc.eld (for NNTP access from home vs. work for instance).
>
> LMI> My vision is that Gnus in Emacs 24 should be so snappy and usable that a
> LMI> normal nerdy programmer user would prefer using Gnus to, er, what,
> LMI> Mozilla?  Or whatever the kids these days are using.
>
> Pretty much everyone I know uses Outlook or web clients.  The GMail IMAP
> support makes a big difference.

I hang around an IRC channel of guys who previously almost all used Gnus
(some of them did run mygnus.org in the olden days). Most of them have
switched in recent years, I think they now use Thunderbird or Sylpheed
Claws.

-- 
Have you ever considered how much text can fit in eighty columns?  Given that a
signature typically contains up to four lines of text, this space allows you to
attach a tremendous amount of valuable information to your messages.  Seize the
opportunity and don't waste your signature on bullshit that nobody cares about.

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Tue, 14 Sep 2010 22:07:14 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

>> My vision is that Gnus in Emacs 24

FWIW I don't know when I'll be on Emacs 24.  I still have an emacs 21
going, I've just recently switched to emacs 22 on one machine.  Debian
testing and Ubuntu Lucid Lynx both have emacs 23 as the default.  But I
had to downgrade to emacs22 on the Ubuntu box, because Ubuntu Network
Edition doesn't display emacs 23 correctly.

>> should be so snappy and usable that a normal nerdy programmer user
>> would prefer using Gnus to, er, what, Mozilla?  Or whatever the kids
>> these days are using.

> Pretty much everyone I know uses Outlook or web clients. 

Yes.  Especially web clients.  If I look at my family, they use Outlook
Express to get at their ISP-supplied email, and all have a gmail
account, and it's the gmail account they use to communicate.

> The GMail IMAP support makes a big difference.

How so?  I thought it was just us geeks using it?

Here's an article I stumbled over today, saying that linux email clients
are stagnating and should learn from gmail
 http://www.linuxplanet.com/linuxplanet/opinions/7171/1/

I'm not sure exactly what his point is, except that he thinks search is
important.

The reason I stay with Gnus is that it is an extremely efficient way of
reading email.  I'm not sure what makes it so efficient, but I suspect
that I run it entirely from the keyboard is an important part of it.

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Here's an article I stumbled over today, saying that linux email clients
> are stagnating and should learn from gmail
>  http://www.linuxplanet.com/linuxplanet/opinions/7171/1/
>
> I'm not sure exactly what his point is, except that he thinks search is
> important.

I think he has a point.  After the web took off in 1999, desktop user
interface development has died a silent death.  (He's saying that only
goes for mail programs, but I think it's across the board.  Not much has
happened the last decade, really.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

On Tue, Sep 14 2010, Ted Zlatanov wrote:

> I think that's over-engineering.  Do a SELECT on INBOX and be done.  If
> we find edge cases we'll fix them, but it's not worth complicating
> everyone's life for that extremely rare situation.

Lars' call. :-)

OTOH, there can be read-only boxes with no no flags at all (even \Seen & co
ones). Or simply read-only where you can't delete mails (we have such a
setup with public mailboxes here In Real Life).

Just saying that in case Lars did not think about that.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: That newfangled IMAP thing...

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Ted Zlatanov <tzz@lifelogs.com> writes:

>> If we move sync logic to a central place, we'll be able to reuse it
>> for all backends and DTRT in a general way instead of a
>> nnimap-specific way.  nntp, nnrss, etc. could really use that.

> But is it fast?  :-)

I use gnus-sync.el right now to sync those nntp groups that overlap
between my gnusen (such as eg. this one).

It's fast once it gets going.  The main slowdown for me is the initial
setup of tramp ssh (I sync using SSH to a file on a server out in the
cloud).

Re: That newfangled IMAP thing...

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Bjørn Mork <bjorn@mork.no> writes:
>
>> And while you are at it, please consider something like the attached
>> patch.  I've been using it since ... forever, and find it absolutely
>> necessary to even consider doing expiry of nnimap groups with a target
>> group.  Without this, all expired articles are downloaded and then
>> uploaded even if the target is on the same IMAP server.  That can be
>> quite annoying on a low bandwidth link...
>
> That sounds useful.  Do you have copyright assignment papers on file
> with the FSF for Emacs patches?

Don't know if it matters after your complete rewrite, but the FSF does
now have my copyright assignment papers.


Bjørn

Re: That newfangled IMAP thing...

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

>>> And while you are at it, please consider something like the attached
>>> patch.  I've been using it since ... forever, and find it absolutely
>>> necessary to even consider doing expiry of nnimap groups with a target
>>> group.  Without this, all expired articles are downloaded and then
>>> uploaded even if the target is on the same IMAP server.  That can be
>>> quite annoying on a low bandwidth link...

[...]

> Don't know if it matters after your complete rewrite, but the FSF does
> now have my copyright assignment papers.

The new nnimap expiry function also works the way you describe, so
something like your patch is still needed.  Could you rework it to fit
the new nnimap?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The new nnimap expiry function also works the way you describe, so
> something like your patch is still needed.  Could you rework it to fit
> the new nnimap?

Will try.  May take me some time as I'm really lisp illiterate (yeah,
about time to admit that...)

But I do see that you've already implemented something similar for the
mail splitting.  I think this code could be refactored into some
generic "nnimap-move-articles-to-target"


Bjørn

[PATCH] nnimap.el (nnimap-process-expiry-targets): avoid downloading articles unless necessary

[Bjørn Mork]

Using UID COPY to move the articles will always be faster.  And downloading articles
which we already know are to be deleted, seems a bit unnecessary...

Signed-off-by: Bjørn Mork <bjorn@mork.no>
---

Finally got around to porting this from my previous implementation.
Let me just repeat that I'm really lisp illiterate and add that I
don't know much about Gnus internals either.  So this code will
most likely look a bit strange to anyone familiar with either or
both.  Please fix as you find appropriate, or use as it is.

This Works For Me(tm).


 lisp/nnimap.el |   52 ++++++++++++++++++++++++++++++++++++----------------
 1 files changed, 36 insertions(+), 16 deletions(-)

diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index a53f9ac..4b4793d 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -793,22 +793,42 @@ textual parts.")
 
 (defun nnimap-process-expiry-targets (articles group server)
   (let ((deleted-articles nil))
-    (dolist (article articles)
-      (let ((target nnmail-expiry-target))
-	(with-temp-buffer
-          (mm-disable-multibyte)
-	  (when (nnimap-request-article article group server (current-buffer))
-	    (nnheader-message 7 "Expiring article %s:%d" group article)
-	    (when (functionp target)
-	      (setq target (funcall target group)))
-	    (when (and target
-		       (not (eq target 'delete)))
-	      (if (or (gnus-request-group target t)
-		      (gnus-request-create-group target))
-		  (nnmail-expiry-target-group target group)
-		(setq target nil)))
-	    (when target
-	      (push article deleted-articles))))))
+    (cond
+     ;; shortcut further processing if we're going to delete the articles
+     ((eq nnmail-expiry-target 'delete)
+      (setq deleted-articles articles)
+      t)
+     ;; or just move them to another folder on the same IMAP server
+     ((and (not (functionp nnmail-expiry-target))
+	   (gnus-server-equal (gnus-group-method nnmail-expiry-target)
+			      (gnus-server-to-method
+			       (format "nnimap:%s" server))))
+      (and (nnimap-possibly-change-group group server)
+	   (with-current-buffer (nnimap-buffer)
+	     (nnheader-message 7 "Expiring articles from %s: %s" group articles)
+	     (nnimap-command
+	      "UID COPY %s %S"
+	      (nnimap-article-ranges (gnus-compress-sequence articles))
+	      (utf7-encode (gnus-group-real-name nnmail-expiry-target) t))
+	     (setq deleted-articles articles)))
+      t)
+     (t
+      (dolist (article articles)
+	(let ((target nnmail-expiry-target))
+	  (with-temp-buffer
+	    (mm-disable-multibyte)
+	    (when (nnimap-request-article article group server (current-buffer))
+	      (nnheader-message 7 "Expiring article %s:%d" group article)
+	      (when (functionp target)
+		(setq target (funcall target group)))
+	      (when (and target
+			 (not (eq target 'delete)))
+		(if (or (gnus-request-group target t)
+			(gnus-request-create-group target))
+		    (nnmail-expiry-target-group target group)
+		  (setq target nil)))
+	      (when target
+		(push article deleted-articles))))))))
     ;; Change back to the current group again.
     (nnimap-possibly-change-group group server)
     (setq deleted-articles (nreverse deleted-articles))
-- 
1.7.2.3

expiry taking "forever" in both cpu time and real time when the IMAP UID is high

[Bjørn Mork]

And then after having implemented the IMAP internal copy speedup, I
noticed that the "fetch all articles which are about to be expired one
by one" code isn't the worst part of the expiration process...

After copying the articles, Gnus does "something" which evidently
depends on the UID value.  When expiring from my work INBOX, which now
is using UIDs in the 280000s although it usually only has a few hundred
messages, emacs will eat 100% of one CPU core for a loooong time
(several minutes).

So I did a simple experiment with my home courier-imap server: I changed
the UID value of one article to a high value (>100000) by editing the
courierimapuiddb file, and then tried to expire this article.  This made
Gnus go into the CPU burning mode.  There is obviously something going
on which doesn't handle high UIDs very well.  This happens right _after_
the actual article copying, while waiting for the *Group* buffer to
reappear.  

Any hints?  Anyone else seen this?


Bjørn

Re: expiry taking "forever" in both cpu time and real time when the IMAP UID is high

[Daniel Pittman]

Bjørn Mork <bjorn@mork.no> writes:

> And then after having implemented the IMAP internal copy speedup, I noticed
> that the "fetch all articles which are about to be expired one by one" code
> isn't the worst part of the expiration process...
>
> After copying the articles, Gnus does "something" which evidently depends on
> the UID value.  When expiring from my work INBOX, which now is using UIDs in
> the 280000s although it usually only has a few hundred messages, emacs will
> eat 100% of one CPU core for a loooong time (several minutes).

[...]

> Any hints?  Anyone else seen this?

Oh, yeah.  Gnus is expanding a range that includes that high number, so it can
walk through it, then compress it back into a range.  I /think/ this is the
stuff that Larsi was just talking about fixing up with the "range tree" stuff
and all.

I mostly ran into it on Zimbra which used a sparse UID space, so could
legitimately have gaps of 100 or even 1000 between consecutive message UIDs.

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: expiry taking "forever" in both cpu time and real time when the IMAP UID is high

[Bjørn Mork]

Daniel Pittman <daniel@rimspace.net> writes:
> Bjørn Mork <bjorn@mork.no> writes:
>
>> And then after having implemented the IMAP internal copy speedup, I noticed
>> that the "fetch all articles which are about to be expired one by one" code
>> isn't the worst part of the expiration process...
>>
>> After copying the articles, Gnus does "something" which evidently depends on
>> the UID value.  When expiring from my work INBOX, which now is using UIDs in
>> the 280000s although it usually only has a few hundred messages, emacs will
>> eat 100% of one CPU core for a loooong time (several minutes).
>
> [...]
>
>> Any hints?  Anyone else seen this?
>
> Oh, yeah.  Gnus is expanding a range that includes that high number, so it can
> walk through it, then compress it back into a range.  I /think/ this is the
> stuff that Larsi was just talking about fixing up with the "range tree" stuff
> and all.
>
> I mostly ran into it on Zimbra which used a sparse UID space, so could
> legitimately have gaps of 100 or even 1000 between consecutive message UIDs.

OK, then I hope it will be handled.  I managed to trace my particular
problem down to gnus-summary-expire-articles() in gnus-sum.el which will

	      (dolist (article expirable)

where "expirable" is the result of gnus-list-of-read-articles() in the
if total-expire is true.


I find this very odd:

(gnus-compress-sequence (gnus-list-of-read-articles "nnimap+mail.mork.no:INBOX.test.source"))
((2 . 93) (95 . 199))


when I have:

* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2010 Double Precision, Inc.  See COPYING for distribution information.
a login test secret
a OK LOGIN Ok.
a select INBOX.test.source
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1285784686] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
a OK [READ-WRITE] Ok
a uid fetch 1:* flags
* 1 FETCH (UID 2 FLAGS (\Seen))
* 2 FETCH (UID 90 FLAGS (\Seen))
* 3 FETCH (UID 94 FLAGS ())
* 4 FETCH (UID 200 FLAGS ())
a OK FETCH completed.


And if I change the UID of the last message to 200000 I get:

(gnus-compress-sequence (gnus-list-of-read-articles "nnimap+mail.mork.no:INBOX.test.source"))
((2 . 93) (95 . 199999))

and

a select INBOX.test.source
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1285784686] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
a OK [READ-WRITE] Ok
a uid fetch 1:* flags
* 1 FETCH (UID 2 FLAGS (\Seen))
* 2 FETCH (UID 90 FLAGS (\Seen))
* 3 FETCH (UID 94 FLAGS ())
* 4 FETCH (UID 200000 FLAGS ())
a OK FETCH completed.



I really do not understand why the list-of-read-articles isn't just (2 90)
in both cases...  Those are the only two articles with a \Seen flag. Why
care about the holes?

Anyway, you don't have to be a rocket scientist to see the problem with
the dolist and the 199996 non-existing UIDs.  Make that a large number
and you'll *really* be into trouble.


Bjørn

Re: [PATCH] nnimap.el (nnimap-process-expiry-targets): avoid downloading articles unless necessary

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> Finally got around to porting this from my previous implementation.
> Let me just repeat that I'm really lisp illiterate and add that I
> don't know much about Gnus internals either.  So this code will
> most likely look a bit strange to anyone familiar with either or
> both.  Please fix as you find appropriate, or use as it is.

Looks fine to me.  Thanks; applied.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: expiry taking "forever" in both cpu time and real time when the IMAP UID is high

[Lars Magne Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> OK, then I hope it will be handled.  I managed to trace my particular
> problem down to gnus-summary-expire-articles() in gnus-sum.el which will
>
> 	      (dolist (article expirable)
>
> where "expirable" is the result of gnus-list-of-read-articles() in the
> if total-expire is true.

Yes, that's going to be very slow, even after the sequence->rtree
makeover.

> (gnus-compress-sequence (gnus-list-of-read-articles
> "nnimap+mail.mork.no:INBOX.test.source"))
> ((2 . 93) (95 . 199999))
>
> and
>
> a select INBOX.test.source
> * FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
> * OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
> * 4 EXISTS
> * 0 RECENT
> * OK [UIDVALIDITY 1285784686] Ok
> * OK [MYRIGHTS "acdilrsw"] ACL
> a OK [READ-WRITE] Ok
> a uid fetch 1:* flags
> * 1 FETCH (UID 2 FLAGS (\Seen))
> * 2 FETCH (UID 90 FLAGS (\Seen))
> * 3 FETCH (UID 94 FLAGS ())
> * 4 FETCH (UID 200000 FLAGS ())
> a OK FETCH completed.
>
> I really do not understand why the list-of-read-articles isn't just (2 90)
> in both cases...  Those are the only two articles with a \Seen flag. Why
> care about the holes?

Gnus know nothing about what articles exist on the server -- just which
articles that it hasn't read, and the (low . high) range is.  (This
could be fixed, of course, but it hasn't.)

So looping over `gnus-list-of-read-articles' is a bad idea, since it can
be monstrously huge, as you've seen, and should be avoided.  I haven't
looked at the total-expire code, but does it really need to do that?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: That newfangled IMAP thing...

[Matthias Andree]

Am 07.09.2010 16:52, schrieb Richard Riley:
> Wes Hardaker <wes@hardakers.net> writes:
> 
>>>>>>> On Sun, 05 Sep 2010 13:20:27 +0200, Lars Magne Ingebrigtsen <larsi@gnus.org> said:
>>
>>>> Split incoming mail with procmail, or sieve, or the mail splitting tool
>>>> of your choice, directly into the imap server's folders.
>>
>> LMI> Yuck.  On the mail server?  Do people set up procmail rules still, in
>> LMI> this day and age?
>>
>> Procmail sucks (and needs to be rewritten).
> 
> What sucks about procmail? It's used throughout the world and heartily
> recommended by many. Certainly my (admittedly minor) use of it suggested
> it was very competent, reliable and easy to use.

Recommendations don't count, because you usually don't get scientific
recommendations, people will stop their search on the first tool that does the
job on initial testing and stick with it, without looking further if there's
better stuff.  Only later, long after they've sold their souls and mail, will
they notice if something goes wrong.


Procmail is anything, but not easy to use or reliable or maintained.
It's been abandonware for nearly a decade.

In .procmailrc, you need to write all the error handling yourself because
procmail will happily try the next recipe if the previous failed to deliver.
Thus, if you want to avoid mail ending up in the wrong box, you'd better be
spamming the whole procmailrc with

:0e
{
  EXITCODE=75
  HOST
}

after each and every delivering recipe.  And I've seen procmail exit with
non-/usr/include/sysexits.h error codes more often than I cared to count, and
I've seen procmail-induced inadvertent non-delivery notices (aka. bounce mail)
far more often than I care to remember.


Compare that with maildrop that automatically defers mail on write errors, and
has a rather intuitive syntax, here a snippet from my $HOME/.mailfilter

# ...
# check upstream's spam scanner status first
if (/^X-Spam-Status: Yes/)
        to "Mail/spam1"

xfilter "bogofilter -e -p -l"

if (/^X-Bogosity: Spam, tests=bogofilter/)
  to "Mail/spam-bf"

if (/^X-Bogosity: Unsure, tests=bogofilter/)
  cc "Mail/unsure"

# LBDB support
`/usr/bin/env lbdb-fetchaddr -d "%Y-%m-%d"`
# ...


Getting that right in procmail takes three screenfuls of text.

-- 
Matthias Andree

slow nnimap expiry (dubious patch)

[John Magorrian]

I've just upgraded from Gnus 5.11 and notice that NoGnus is unusably
slow at expiring msgs from my imap inbox.  Here's my set up:

  (setq gnus-total-expirable-newsgroups "INBOX$\\|personal$\\|...."
        nnmail-expiry-wait 'immediate
        nnmail-expiry-target 'my-gnus-expiry-target)

The reason for this slowness was isolated by Bjørn Mork almost a year
ago (http://article.gmane.org/gmane.emacs.gnus.general/74617): if a
group has only two articles with UIDs 1 and 9999, say, then Gnus will
try to expire all articles in the range 2..9998.  In response,

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
> Gnus know nothing about what articles exist on the server -- just which
> articles that it hasn't read, and the (low . high) range is.  (This
> could be fixed, of course, but it hasn't.)
>
> So looping over `gnus-list-of-read-articles' is a bad idea, since it can
> be monstrously huge, as you've seen, and should be avoided.  I haven't
> looked at the total-expire code, but does it really need to do that?

Here is a quick-and-dirty patch that seems to solve the problem (for
nnimap at least) by modifying 'nnimap-request-expire-articles' to ask
the server what articles it knows about.  I don't think this is an ideal
solution and I'm not even sure it's correct.  Are there any alternative
workarounds for those of us who like Gnus' idea of expiring articles,
but whose IMAP inboxes have vast voids in their UID distributions?

John



From 3cb9f595503fc88aa54b42c88c8c7461e9733187 Mon Sep 17 00:00:00 2001
From: John Magorrian <john.magorrian@ml1.net>
Date: Sun, 6 Nov 2011 21:33:55 +0000
Subject: [PATCH] An attempt at speeding up nnimap expiry.

nnimap.el (nnimap-request-expire-articles): take intersection of
'articles' argument with the article list fetched from IMAP server.
---
 lisp/nnimap.el |   23 +++++++++++++++++++++++
 1 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index cda17ba..9fec178 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -860,6 +860,29 @@ textual parts.")
 
 (deffoo nnimap-request-expire-articles (articles group &optional server force)
   (setq group (nnimap-decode-gnus-group group))
+
+  ;; Filter out any non-existent articles from 'articles'.
+  ;; Ask the server for a list of articles it knows about (the following code
+  ;; snippet is copied from nnimap-request-group) and then take intersection
+  ;; with 'articles'.
+  (let ((genuine-articles
+         (with-current-buffer (nnimap-buffer)
+           (erase-buffer)
+           (let ((marks)
+                 (group-sequence
+                  (nnimap-send-command "SELECT %S" (utf7-encode group t)))
+                 (flag-sequence
+                  (nnimap-send-command "UID FETCH 1:* FLAGS")))
+             (setf (nnimap-group nnimap-object) group)
+             (nnimap-wait-for-response flag-sequence)
+             (setq marks
+                   (nnimap-flags-to-marks
+                    (nnimap-parse-flags
+                     (list (list group-sequence flag-sequence
+                                 1 group "SELECT")))))
+             (cadr (car marks))))))
+    (setq articles (gnus-sorted-intersection articles genuine-articles)))
+
   (cond
    ((null articles)
     nil)
-- 
1.7.2.5

Re: slow nnimap expiry (dubious patch)

[Lars Magne Ingebrigtsen]

John Magorrian <john.magorrian@ml1.net> writes:

> Here is a quick-and-dirty patch that seems to solve the problem (for
> nnimap at least) by modifying 'nnimap-request-expire-articles' to ask
> the server what articles it knows about.  I don't think this is an ideal
> solution and I'm not even sure it's correct.

It looks OK, but I think the real long-time solution is to tell Gnus
what articles really exist so that this sort of thing wouldn't be
necessary.  However that's not going to happen during this development
cycle.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: slow nnimap expiry (dubious patch)

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> John Magorrian <john.magorrian@ml1.net> writes:
>
>> Here is a quick-and-dirty patch that seems to solve the problem (for
>> nnimap at least) by modifying 'nnimap-request-expire-articles' to ask
>> the server what articles it knows about.  I don't think this is an ideal
>> solution and I'm not even sure it's correct.
>
> It looks OK, but I think the real long-time solution is to tell Gnus
> what articles really exist so that this sort of thing wouldn't be
> necessary.  However that's not going to happen during this development
> cycle.

Which would imply that this patch is the only possible fix for no-gnus.
Please add it to that branch.

FWIW, I've been using the patch on top of no-gnus since John posted it
and haven't noticed any problems.  And the nnimap expiry speed of my
INBOX has gone from "will never ever finish during this universe's
lifetime" to "noticable, but acceptable".


Bjørn

Re: slow nnimap expiry (dubious patch)

[Lars Ingebrigtsen]

Bjørn Mork <bjorn@mork.no> writes:

> Which would imply that this patch is the only possible fix for no-gnus.
> Please add it to that branch.

It's a new feature, and Emacs 24 is in a feature freeze, so...

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Sent from my Rome

Re: slow nnimap expiry (dubious patch)

[Bjørn Mork]

Lars Ingebrigtsen <larsi@gnus.org> writes:
> Bjørn Mork <bjorn@mork.no> writes:
>
>> Which would imply that this patch is the only possible fix for no-gnus.
>> Please add it to that branch.
>
> It's a new feature, 

Well, that's your call to make, but to me it's a bugfix required for
usable nnimap expiry.



Bjørn

Re: slow nnimap expiry (dubious patch)

[Dave Abrahams]

on Tue Jan 03 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> John Magorrian <john.magorrian@ml1.net> writes:
>
>> Here is a quick-and-dirty patch that seems to solve the problem (for
>> nnimap at least) by modifying 'nnimap-request-expire-articles' to ask
>> the server what articles it knows about.  I don't think this is an ideal
>> solution and I'm not even sure it's correct.
>
> It looks OK, but I think the real long-time solution is to tell Gnus
> what articles really exist so that this sort of thing wouldn't be
> necessary.  However that's not going to happen during this development
> cycle.

Not sure what the bounds of the development cycle are, but my
understanding from <87aa4pz1wi.fsf@gnus.org> is that you've now done
this.  Have you also fixed expiry accordingly?

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: slow nnimap expiry (dubious patch)

[Lars Magne Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

>> It looks OK, but I think the real long-time solution is to tell Gnus
>> what articles really exist so that this sort of thing wouldn't be
>> necessary.  However that's not going to happen during this development
>> cycle.
>
> Not sure what the bounds of the development cycle are, but my
> understanding from <87aa4pz1wi.fsf@gnus.org> is that you've now done
> this.  Have you also fixed expiry accordingly?

I hadn't, but now I have.  I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: slow nnimap expiry (dubious patch)

[Dave Abrahams]

on Fri Mar 09 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>>> It looks OK, but I think the real long-time solution is to tell Gnus
>>> what articles really exist so that this sort of thing wouldn't be
>>> necessary.  However that's not going to happen during this development
>>> cycle.
>>
>> Not sure what the bounds of the development cycle are, but my
>> understanding from <87aa4pz1wi.fsf@gnus.org> is that you've now done
>> this.  Have you also fixed expiry accordingly?
>
> I hadn't, but now I have.  I think.

YAY!  Can't wait to try it.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: slow nnimap expiry (dubious patch)

[John Magorrian]

Dave Abrahams <dave@boostpro.com> writes:

> on Fri Mar 09 2012, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:
>
>> Dave Abrahams <dave@boostpro.com> writes:
>>
>>>> It looks OK, but I think the real long-time solution is to tell Gnus
>>>> what articles really exist so that this sort of thing wouldn't be
>>>> necessary.  However that's not going to happen during this development
>>>> cycle.
>>>
>>> Not sure what the bounds of the development cycle are, but my
>>> understanding from <87aa4pz1wi.fsf@gnus.org> is that you've now done
>>> this.  Have you also fixed expiry accordingly?
>>
>> I hadn't, but now I have.  I think.
>
> YAY!  Can't wait to try it.

It's been working well for me so far.  Thank you!

John