file storage in IMAP (eventually for Tramp) working and needs testing

file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Using imap.el and encrypt.el, I was able to store files in IMAP.

imap.el is in the Emacs core now, and encrypt.el can be replaced by
pgg.el or something similar if necessary (I personally find it
convenient to use encrypt.el but it's not the only way).  So this
package doesn't depend on Gnus functionality, although it can definitely
be used in Gnus.

The attached trimp.el (name is not final, but I like it) will eventually
provide a Tramp over IMAP interface.  For now it's simply a way to
put/get files from an IMAP mailbox.

Store a file (uses a file name):
(trimp-put-file mailbox file-to-insert)

Get a file (returns just contents):
; assumes mailbox is already selected
(trimp-get-file message-number)

The messages are build with a hardcoded headers string for now.  There's
no message ID header, for instance.  Duplicates are allowed for now
(this is useful IMO, since it provides a sort of version history, but
there will probably be a limit on the number of duplicates).

The files are stored as GPG-encrypted (AES cipher) data (encrypt.el
allows much more choice, this is just hard-coded for testing).  This
encryption is, IMO, much easier than the hassle of MIME over IMAP, plus
it gives us file encryption.  The sacrifice of having to store 1 file
per message is worthwhile, again IMO.

This is very experimental work, but it does work for me to a Courier
IMAP server.  If anyone wants to test this out, look at the function
trimp-test which encapsulates the current state of my testing.  

I think trimp.el could be very useful for storing things like marks and
passwords on an IMAP server for Gnus; the applications are literally
unlimited once it works with Tramp.  I need to make sure the IMAP basics
are solid before I get to that point, though.  That's why I'm not cc-ing
the Tramp devel list yet.

Simon and other Gnus users, if you could take a look and comment, I
would appreciate it.

Thanks
Ted


[-- Attachment #2: trimp.el --]
[-- Type: application/emacs-lisp, Size: 3396 bytes --]

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[David]

Ted Zlatanov <tzz@lifelogs.com> writes:
> Using imap.el and encrypt.el, I was able to store files in IMAP.
[...]
> I think trimp.el could be very useful for storing things like marks and
> passwords on an IMAP server for Gnus; the applications are literally
> unlimited once it works with Tramp.  I need to make sure the IMAP basics
> are solid before I get to that point, though.  That's why I'm not cc-ing
> the Tramp devel list yet.
>
> Simon and other Gnus users, if you could take a look and comment, I
> would appreciate it.

It works for me with the Dovecot IMAP server. Aside from the
applications you already mentioned, I think this could be particularly
useful for storing drafts on the IMAP server. IMHO this is one feature
which Gnus sorely lacks at the moment. As far as I understood, one would
have to implement an IMAP file handler so that e.g. auto-saving of
drafts can work. Another application would be to save files like .bbdb
and .gnus on the IMAP server, which is useful for syncing different Gnus
installations.

-David

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Sat, 29 Dec 2007 23:34:02 +0100 David <de_bb@arcor.de> wrote: 

D> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Using imap.el and encrypt.el, I was able to store files in IMAP.
D> [...]
>> I think trimp.el could be very useful for storing things like marks and
>> passwords on an IMAP server for Gnus; the applications are literally
>> unlimited once it works with Tramp.  I need to make sure the IMAP basics
>> are solid before I get to that point, though.  That's why I'm not cc-ing
>> the Tramp devel list yet.
>> 
>> Simon and other Gnus users, if you could take a look and comment, I
>> would appreciate it.

D> It works for me with the Dovecot IMAP server. Aside from the
D> applications you already mentioned, I think this could be particularly
D> useful for storing drafts on the IMAP server. IMHO this is one feature
D> which Gnus sorely lacks at the moment. As far as I understood, one would
D> have to implement an IMAP file handler so that e.g. auto-saving of
D> drafts can work.

I had not thought about using drafts over Tramp, but I think it's better
to support drafts to an IMAP server directly, so you can use them with
any other IMAP client (as Thunderbird does by default, for example).

D> Another application would be to save files like .bbdb and .gnus on
D> the IMAP server, which is useful for syncing different Gnus
D> installations.

Yes, those would be nice.  Splitting rules are another one, as a special
case of the .gnus storage.  The Gnus registry file is another one, as
long as it's not too big.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I had not thought about using drafts over Tramp, but I think it's better
> to support drafts to an IMAP server directly, so you can use them with
> any other IMAP client (as Thunderbird does by default, for example).

I don't get the point. Tramp is good for storing files somewhere else,
this could be a draft folder of an IMAP server(1). How would it prevent
other IMAP clients using it?

(1) It would need an imap method implementation, of course.

> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Mon, 31 Dec 2007 11:14:38 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I had not thought about using drafts over Tramp, but I think it's better
>> to support drafts to an IMAP server directly, so you can use them with
>> any other IMAP client (as Thunderbird does by default, for example).

MA> I don't get the point. Tramp is good for storing files somewhere else,
MA> this could be a draft folder of an IMAP server(1). How would it prevent
MA> other IMAP clients using it?

MA> (1) It would need an imap method implementation, of course.

I'm throwing the whole file, PGP-encrypted, into an article, and using
minimal headers plus a hacked-up Subject header to make it work.  I'm
not doing this with interoperability in mind, and I don't think as it
stands now it's usable by other IMAP clients.  If there's a standard for
file storage over IMAP (and I don't mean MIME) I don't know about it.
It needs to support deletion, overwriting, unambiguous naming, etc.

Gnus, whether accidentally or by design, will actually understand these
primitive articles and decode them without Trimp's help, but that was a
surprise to me :)

Incidentally since I mentioned "unambiguous naming" above, the header
currently has the file name stripped of any characters other than
[a-zA-Z/] which is nice for searching but will cause conflicts with many
file names.  I am thinking of using

Subject: trimp-subject-marker stripped-filename subject-encoded-filename

So the full filename is also visible without getting the article, but
the stripped filename lets us do IMAP searches easily.  The encoding
could be UTF-8.  As I said, there's no standard for this AFAIK, so any
suggestions are welcome.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> MA> I don't get the point. Tramp is good for storing files somewhere else,
> MA> this could be a draft folder of an IMAP server(1). How would it prevent
> MA> other IMAP clients using it?
>
> MA> (1) It would need an imap method implementation, of course.
>
> I'm throwing the whole file, PGP-encrypted, into an article, and using
> minimal headers plus a hacked-up Subject header to make it work.  I'm
> not doing this with interoperability in mind, and I don't think as it
> stands now it's usable by other IMAP clients.  If there's a standard for
> file storage over IMAP (and I don't mean MIME) I don't know about it.
> It needs to support deletion, overwriting, unambiguous naming, etc.

I don't know the IMAP protocol. But a short scan of RFC 3501 (IMAPv4)
gives me the impression one could use the SELECT, EXAMINE, CREATE,
DELETE, LIST, RENAME etc commands for mailbox operations. APPEND,
EXPUNGE, FETCH, STORE, COPY work over messages of a mailbox. If you
regard mailbox as a directory, and a message inside as a file, I see no
reason why you cannot apply Emacs' primitive file name handler
routines. Of course, the *contents* of the file needs some massage
during the transfer as you have said above.

> Incidentally since I mentioned "unambiguous naming" above, the header
> currently has the file name stripped of any characters other than
> [a-zA-Z/] which is nice for searching but will cause conflicts with many
> file names.  I am thinking of using
>
> Subject: trimp-subject-marker stripped-filename subject-encoded-filename
>
> So the full filename is also visible without getting the article, but
> the stripped filename lets us do IMAP searches easily.  The encoding
> could be UTF-8.  As I said, there's no standard for this AFAIK, so any
> suggestions are welcome.

A very simple approach would be to use the message UID or the message
sequence number as filename. That might introduce restrictions for
searching etc I don't know. But it would prevent ambiguous names.

> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Mon, 31 Dec 2007 14:35:29 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> I don't get the point. Tramp is good for storing files somewhere else,
MA> this could be a draft folder of an IMAP server(1). How would it prevent
MA> other IMAP clients using it?
>> 
MA> (1) It would need an imap method implementation, of course.
>> 
>> I'm throwing the whole file, PGP-encrypted, into an article, and using
>> minimal headers plus a hacked-up Subject header to make it work.  I'm
>> not doing this with interoperability in mind, and I don't think as it
>> stands now it's usable by other IMAP clients.  If there's a standard for
>> file storage over IMAP (and I don't mean MIME) I don't know about it.
>> It needs to support deletion, overwriting, unambiguous naming, etc.

MA> I don't know the IMAP protocol. But a short scan of RFC 3501 (IMAPv4)
MA> gives me the impression one could use the SELECT, EXAMINE, CREATE,
MA> DELETE, LIST, RENAME etc commands for mailbox operations. APPEND,
MA> EXPUNGE, FETCH, STORE, COPY work over messages of a mailbox. If you
MA> regard mailbox as a directory, and a message inside as a file, I see no
MA> reason why you cannot apply Emacs' primitive file name handler
MA> routines. Of course, the *contents* of the file needs some massage
MA> during the transfer as you have said above.

Yes, that's my plan basically.  I'm just making up the protocol instead
of using something already standardized :)  Also I don't want the
mailbox to be a directory, you'd end up with a lot of mailboxes.  Better
to have the IMAP server+folder be the root, and everything under it goes
in a single mailbox.

>> Incidentally since I mentioned "unambiguous naming" above, the header
>> currently has the file name stripped of any characters other than
>> [a-zA-Z/] which is nice for searching but will cause conflicts with many
>> file names.  I am thinking of using
>> 
>> Subject: trimp-subject-marker stripped-filename subject-encoded-filename
>> 
>> So the full filename is also visible without getting the article, but
>> the stripped filename lets us do IMAP searches easily.  The encoding
>> could be UTF-8.  As I said, there's no standard for this AFAIK, so any
>> suggestions are welcome.

MA> A very simple approach would be to use the message UID or the message
MA> sequence number as filename. That might introduce restrictions for
MA> searching etc I don't know. But it would prevent ambiguous names.

That won't work, unfortunately.  IMAP UIDs and message sequence IDs can
change (if I understand the IMAP protocol correctly) on subsequent
sessions.  I know at least one IMAP server will do that.  Also you can
compact a folder and renumber all the messages, which is sometimes
necessary.  Finally, if the filename is the message UID, that limits us
to numerical filenames.

I think we need to store metadata in the subject line and do a
search+retrieve every time instead of a straight retrieve to grab a
single message.  Fortunately IMAP is fast on searches so this is a
decent compromise.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[David]

Ted Zlatanov <tzz@lifelogs.com> writes:
> I had not thought about using drafts over Tramp, but I think it's better
> to support drafts to an IMAP server directly, so you can use them with
> any other IMAP client (as Thunderbird does by default, for example).

Yes, that would the best thing to do, but so far no one has coded
this. I only skimmed through nndraft.el and I don't really understand
how this all integrates with the Emacs file handlers, but since no one
has made nndraft.el to work over IMAP yet, it seems not a trivial thing
to do. If Tramp would support file storage over IMAP, it should be
fairly easy to implement things like auto-saving of unfinished message
buffers on an IMAP server. Even if continuing these messages only works
with Gnus, it's still better than nothing. By the way, your way of
storing files over IMAP seems to be more compatible than one might think
- I just tested it with Thunderbird with the Enigmail plugin and it
showed the files right away (this doesn't help for editing drafts,
though). 

-David

PS: I read ding over gmane, so no need to CC me.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes, that's my plan basically.  I'm just making up the protocol instead
> of using something already standardized :)  Also I don't want the
> mailbox to be a directory, you'd end up with a lot of mailboxes.  Better
> to have the IMAP server+folder be the root, and everything under it goes
> in a single mailbox.

Forgive me my ignorance, but what is the difference beetween a folder
and a mailbox on an IMAP server? I thought it is the same ... RFC 3501
says: "IMAP4rev1 permits manipulation of mailboxes (remote message
folders) in a way that is functionally equivalent to local folders".

> MA> A very simple approach would be to use the message UID or the message
> MA> sequence number as filename. That might introduce restrictions for
> MA> searching etc I don't know. But it would prevent ambiguous names.
>
> That won't work, unfortunately.  IMAP UIDs and message sequence IDs can
> change (if I understand the IMAP protocol correctly) on subsequent
> sessions.

RFC 3501: "The unique identifier of a message MUST NOT change during the
session, and SHOULD NOT change between sessions.  Any change of unique
identifiers between sessions MUST be detectable using the UIDVALIDITY
mechanism discussed below.  Persistent unique identifiers are required
for a client to resynchronize its state from a previous session with the
server (e.g., disconnected or offline access clients); this is discussed
further in [IMAP-DISC]."

OK, if there is a new IMAP session during Emacs' lifetime, opened files
must be retrieved again when the IMAP server reports a change. But this
shall be possible to handle, maybe with revert-buffer or so. Unsaved
files would be a problem, but they would be a problem anyway when an
IMAP session has been closed.

> I know at least one IMAP server will do that.  Also you can
> compact a folder and renumber all the messages, which is sometimes
> necessary.  Finally, if the filename is the message UID, that limits us
> to numerical filenames.

Not really. You could create a virtual filename in the metadata of the
subject line. This name is kept in buffer-file-name. In
buffer-file-truename, you could keep the UID of the corresponding
message. Reading/writing is performed by your IMAP file name handler
functions, which know what to do.

You would need kind of file name unicity per mailbox. But this task
could be given to Emacs: first, you create the buffer with the virtual
file name "Foo" from the subject line. If there is already such a buffer,
Emacs will use "Foo<1>" etc. The corresponding buffer-file-name for that
buffer is then (concat default-directory "/" (buffer-name))

> I think we need to store metadata in the subject line and do a
> search+retrieve every time instead of a straight retrieve to grab a
> single message.  Fortunately IMAP is fast on searches so this is a
> decent compromise.

Yes.

> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Mon, 31 Dec 2007 18:14:42 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Yes, that's my plan basically.  I'm just making up the protocol instead
>> of using something already standardized :)  Also I don't want the
>> mailbox to be a directory, you'd end up with a lot of mailboxes.  Better
>> to have the IMAP server+folder be the root, and everything under it goes
>> in a single mailbox.

MA> Forgive me my ignorance, but what is the difference beetween a folder
MA> and a mailbox on an IMAP server? I thought it is the same ... 

Sorry, my terminology is off.  Yes they are the same and I should have
been consistent.

MA> RFC 3501 says: "IMAP4rev1 permits manipulation of mailboxes (remote
MA> message folders) in a way that is functionally equivalent to local
MA> folders".

I know, but it's ugly (IMO) to do INBOX.a.b.c to store file /a/b/c/d so
I'd rather have INBOX.storage effectively act as a filesystem, and
/a/b/c/d or /e/f/g/h would both live inside INBOX.storage.  OTOH doing
mailboxes as directories would simplify file name handling and be a bit
more obvious to the users...  I'm not sure what's best.  My vote is for
a single mailbox.

MA> A very simple approach would be to use the message UID or the message
MA> sequence number as filename. That might introduce restrictions for
MA> searching etc I don't know. But it would prevent ambiguous names.
>> 
>> That won't work, unfortunately.  IMAP UIDs and message sequence IDs can
>> change (if I understand the IMAP protocol correctly) on subsequent
>> sessions.

MA> RFC 3501: "The unique identifier of a message MUST NOT change during the
MA> session, and SHOULD NOT change between sessions.  Any change of unique
MA> identifiers between sessions MUST be detectable using the UIDVALIDITY
MA> mechanism discussed below.  Persistent unique identifiers are required
MA> for a client to resynchronize its state from a previous session with the
MA> server (e.g., disconnected or offline access clients); this is discussed
MA> further in [IMAP-DISC]."

MA> OK, if there is a new IMAP session during Emacs' lifetime, opened files
MA> must be retrieved again when the IMAP server reports a change. But this
MA> shall be possible to handle, maybe with revert-buffer or so. Unsaved
MA> files would be a problem, but they would be a problem anyway when an
MA> IMAP session has been closed.

It seems like a lot of work.  I don't know imap.el well enough to tell
if it can do the dirty work for me and give me a unique identifier.  Can
anyone tell me?  I see UID and ID mixed in the imap.el functions and in
the documentation, but don't know how they relate.  Right now I'm just
using what imap.el gives me as message numbers without knowing their
persistence.

As I said, at least one IMAP server (Zimbra I think) does strange things
with the UIDVALIDITY and Gnus has to resynchronize every session.  Is it
really a win to use UIDs compared to a SEARCH+RETRIEVE?  Why not make
the file name be the unique ID of the message?  See below for the
approach I am proposing.

>> I know at least one IMAP server will do that.  Also you can
>> compact a folder and renumber all the messages, which is sometimes
>> necessary.  Finally, if the filename is the message UID, that limits us
>> to numerical filenames.

MA> Not really. You could create a virtual filename in the metadata of the
MA> subject line. This name is kept in buffer-file-name. In
MA> buffer-file-truename, you could keep the UID of the corresponding
MA> message. Reading/writing is performed by your IMAP file name handler
MA> functions, which know what to do.

MA> You would need kind of file name unicity per mailbox. But this task
MA> could be given to Emacs: first, you create the buffer with the virtual
MA> file name "Foo" from the subject line. If there is already such a buffer,
MA> Emacs will use "Foo<1>" etc. The corresponding buffer-file-name for that
MA> buffer is then (concat default-directory "/" (buffer-name))

Why not just use the file name as the identifier, embedded in the
Subject header?  At worst we'll end up with 2 or more articles with the
same file name, which is OK: we just work with the latest one, and the
others are backups that can be automatically removed if necessary.  Even
the Message-ID is not really necessary to identify the article of
interest, we just need to do a Subject search for the collapsed name and
check the matching articles' Subject headers against the full expansion
of the file name.  We use the latest article (by the Date header) no
matter how many matches there are, which protects us from older
versions.

I don't know Tramp too well, so if you think the UID approach is better
for Tramp integration, I'll go along.  Thank you very much for the
comments and suggestions.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> MA> RFC 3501 says: "IMAP4rev1 permits manipulation of mailboxes (remote
> MA> message folders) in a way that is functionally equivalent to local
> MA> folders".
>
> I know, but it's ugly (IMO) to do INBOX.a.b.c to store file /a/b/c/d so
> I'd rather have INBOX.storage effectively act as a filesystem, and
> /a/b/c/d or /e/f/g/h would both live inside INBOX.storage.  OTOH doing
> mailboxes as directories would simplify file name handling and be a bit
> more obvious to the users...  I'm not sure what's best.  My vote is for
> a single mailbox.

What about using "INBOX.a.b.c" as first level directory from root? This
would result in flat directories like "/INBOX.a.b.c/", "/INBOX.d.e.f/"
and "/DRAFT.x.y.z/" and no more subdirectories, but so what? Users would
recognize their mailbox names.

> MA> Not really. You could create a virtual filename in the metadata of the
> MA> subject line. This name is kept in buffer-file-name. In
> MA> buffer-file-truename, you could keep the UID of the corresponding
> MA> message. Reading/writing is performed by your IMAP file name handler
> MA> functions, which know what to do.
>
> MA> You would need kind of file name unicity per mailbox. But this task
> MA> could be given to Emacs: first, you create the buffer with the virtual
> MA> file name "Foo" from the subject line. If there is already such a buffer,
> MA> Emacs will use "Foo<1>" etc. The corresponding buffer-file-name for that
> MA> buffer is then (concat default-directory "/" (buffer-name))
>
> Why not just use the file name as the identifier, embedded in the
> Subject header?  At worst we'll end up with 2 or more articles with the
> same file name, which is OK: we just work with the latest one, and the
> others are backups that can be automatically removed if necessary.  Even
> the Message-ID is not really necessary to identify the article of
> interest, we just need to do a Subject search for the collapsed name and
> check the matching articles' Subject headers against the full expansion
> of the file name.  We use the latest article (by the Date header) no
> matter how many matches there are, which protects us from older
> versions.

I'm a little bit lost. Does it mean you don't want to offer *all*
messages in a mailbox?

> I don't know Tramp too well, so if you think the UID approach is better
> for Tramp integration, I'll go along.

Oh, there is no Tramp specific meaning how a file name should look
like. In fact I've only tried to sketch a scenario where file names
visible to the user are derived from the virtual file name kept in the
subject header line of the message, and the "file name" you speak with
IMAP is the UID. Solving also the problem of file name unicity.

The more serious work with Tramp is that all file operations towards
IMAP must be mapped to Emacs' file name handler primitives like
file-local-copy or write-region. If you want, I could support you a
little bit(1) in writing a tramp-imap.el.

> Ted

Best regards, Michael.

(1): As usual, I'm busy with other projects. Currently the D-Bus
integration into Emacs. But you've attracted me, somehow.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Mon, 31 Dec 2007 22:32:01 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> RFC 3501 says: "IMAP4rev1 permits manipulation of mailboxes (remote
MA> message folders) in a way that is functionally equivalent to local
MA> folders".
>> 
>> I know, but it's ugly (IMO) to do INBOX.a.b.c to store file /a/b/c/d so
>> I'd rather have INBOX.storage effectively act as a filesystem, and
>> /a/b/c/d or /e/f/g/h would both live inside INBOX.storage.  OTOH doing
>> mailboxes as directories would simplify file name handling and be a bit
>> more obvious to the users...  I'm not sure what's best.  My vote is for
>> a single mailbox.

MA> What about using "INBOX.a.b.c" as first level directory from root? This
MA> would result in flat directories like "/INBOX.a.b.c/", "/INBOX.d.e.f/"
MA> and "/DRAFT.x.y.z/" and no more subdirectories, but so what? Users would
MA> recognize their mailbox names.

I'm not sure what you're saying, sorry.

MA> I'm a little bit lost. Does it mean you don't want to offer *all*
MA> messages in a mailbox?

Correct, otherwise it's hard to handle invalid messages: are they
invalid files or something else?  I wanted also to treat duplicates as
automatic backups but if you don't like that idea then I'll drop it.

MA> The more serious work with Tramp is that all file operations towards
MA> IMAP must be mapped to Emacs' file name handler primitives like
MA> file-local-copy or write-region. If you want, I could support you a
MA> little bit(1) in writing a tramp-imap.el.

Sure.  In fact trimp.el is tramp-imap.el, shortened :) The name is not
important, if you want the latter.  Should it live in Tramp instead of
Gnus?  If so, imap.el will be needed, and it's only in Emacs after 22 I
believe.  Any opinions (Reiner?) are welcome.

Let me know what pieces you need from me, and if there's an example in
Tramp I should follow to implement this.

MA> (1): As usual, I'm busy with other projects. Currently the D-Bus
MA> integration into Emacs. But you've attracted me, somehow.

I appreciate your interest, and it's certainly nice to have your help.

Thanks
Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Reiner Steib]

[ The following message is a courtesy copy of an article that has
  been posted to news:gmane.emacs.gnus.general as well. ]

On Wed, Jan 02 2008, Ted Zlatanov wrote:
> Michael Albinus <michael.albinus@gmx.de> wrote:
> MA> The more serious work with Tramp is that all file operations towards
> MA> IMAP must be mapped to Emacs' file name handler primitives like
> MA> file-local-copy or write-region. If you want, I could support you a
> MA> little bit(1) in writing a tramp-imap.el.
>
> Sure.  In fact trimp.el is tramp-imap.el, shortened :) The name is not
> important, if you want the latter.  Should it live in Tramp instead of
> Gnus?  If so, imap.el will be needed, and it's only in Emacs after 22 I
> believe.  Any opinions (Reiner?) are welcome.

Emacs 21 (e.g. Gnus 5.9) already has imap.el.  Of course not a current
version, so I don't know if it would be sufficient for
trimp.el/tramp-imap.el.  But I don't think this matters much.  A
library for "file storage in IMAP" should probably be included in
Emacs, either as part of tramp or independently, but not as part of
Gnus (making it possible to save drafts on IMAP would be a very nice
feature for Gnus, of course!).  So you should discuss it on
emacs-devel sooner or later.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Wed, 02 Jan 2008 22:40:12 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> Emacs 21 (e.g. Gnus 5.9) already has imap.el.  Of course not a
RS> current version, so I don't know if it would be sufficient for
RS> trimp.el/tramp-imap.el.  But I don't think this matters much.  A
RS> library for "file storage in IMAP" should probably be included in
RS> Emacs, either as part of tramp or independently, but not as part of
RS> Gnus (making it possible to save drafts on IMAP would be a very nice
RS> feature for Gnus, of course!).  So you should discuss it on
RS> emacs-devel sooner or later.

All right, I'll work with Michael Albinus to put this functionality in
Tramp and from there it will go into Emacs.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> MA> I'm a little bit lost. Does it mean you don't want to offer *all*
> MA> messages in a mailbox?
>
> Correct, otherwise it's hard to handle invalid messages: are they
> invalid files or something else?  I wanted also to treat duplicates as
> automatic backups but if you don't like that idea then I'll drop it.

Maybe we should see real life examples. I don't know whether it is
always good to present selected contents only. If there are technical
restrictions - that's another game.

> MA> The more serious work with Tramp is that all file operations towards
> MA> IMAP must be mapped to Emacs' file name handler primitives like
> MA> file-local-copy or write-region. If you want, I could support you a
> MA> little bit(1) in writing a tramp-imap.el.
>
> Sure.  In fact trimp.el is tramp-imap.el, shortened :) The name is not
> important, if you want the latter.  Should it live in Tramp instead of
> Gnus?  If so, imap.el will be needed, and it's only in Emacs after 22 I
> believe.  Any opinions (Reiner?) are welcome.
>
> Let me know what pieces you need from me, and if there's an example in
> Tramp I should follow to implement this.

It would be nice if it could live in the Tramp repository. I'm not
fundamentalistic about names, but so far we have just tramp*.el files
there.

The only "piece" I would need from you are legal papers from the FSF for
Tramp and for Emacs. Guess you have it partly already because of gnus.

Once this is done, nothing will prevent you from writing in the Tramp CVS.

As starting point you might look at tramp-smb.el or tramp-gw.el. Both
are addons, like tramp-imap.el could be. tramp-fish.el might be examined
as well, but this method isn't used anywhere I believe - it was merely a
proof of concept I didn't want to throw away. And I never ever got a bug
report about ...

It wouldn't be a problem if imap.el is not present in all (X)Emacs
versions to be supported. tramp-gw.el lives with similar restrictions;
it is loaded only if some additional (external) packages are available.

I've added tramp-devel to the Cc, in order to answer the question from
your other message.

> Thanks
> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Reiner Steib]

On Thu, Jan 03 2008, Michael Albinus wrote:

> The only "piece" I would need from you are legal papers from the FSF for
> Tramp and for Emacs. Guess you have it partly already because of gnus.

Ted (Teodor Zlatanov) already has an assignment for Emacs on file.
I'd guess this covers Tramp as well, doesn't it?
(Like an assignment for Emacs covers Gnus.)

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 03 Jan 2008 12:32:44 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Thu, Jan 03 2008, Michael Albinus wrote:
>> The only "piece" I would need from you are legal papers from the FSF for
>> Tramp and for Emacs. Guess you have it partly already because of gnus.

RS> Ted (Teodor Zlatanov) already has an assignment for Emacs on file.
RS> I'd guess this covers Tramp as well, doesn't it?
RS> (Like an assignment for Emacs covers Gnus.)

I've submitted Emacs papers and Gnus papers, so no worries either way.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I've submitted Emacs papers and Gnus papers, so no worries either way.

Thanks. I have no idea whether an additional Tramp paper is needed
then, we will ask the FSF.

However, it is reasonable for me to give you write access to the Tramp
CVS repository. I've just added you on savannah.

> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 03 Jan 2008 00:50:56 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> I'm a little bit lost. Does it mean you don't want to offer *all*
MA> messages in a mailbox?
>> 
>> Correct, otherwise it's hard to handle invalid messages: are they
>> invalid files or something else?  I wanted also to treat duplicates as
>> automatic backups but if you don't like that idea then I'll drop it.

MA> Maybe we should see real life examples. I don't know whether it is
MA> always good to present selected contents only. If there are technical
MA> restrictions - that's another game.

(I'm leaving Ding on the CC in case anyone has comments)

Say you have these article UIDs and subjects, and UIDs are ordered by
date.  Note the ENCODED_FILENAME will be the filename encoded in some
way and /a/b/c is the filename without any risky characters; Gnus has
the subject encoding functionality already.  The subject format is
specifically to allow easy searching.

1 "tramp-imap-marker /a/b/c ENCODED_ABC"
2 "tramp-imap-marker /a/b/c ENCODED_ABC"
3 "tramp-imap-marker /a/b/c ENCODED_ABC"
4 "hello world"

Do we 

A) allow for three identically named files, dynamically renaming 2 and 3
with the Emacs <1> and <2> convention?

B) ignore 1 and 2, never create them intentionally, and (optionally)
erase duplicates when a file is saved?  This is probably easiest to
implement in code.

C) treat 1 and 2 as backups (there will be a tramp-imap-backups
parameter to control how many are allowed per filename, and older ones
over that limit are erased when a file is saved)?  We'll need a
mechanism to revert to a backup, which I don't think Tramp has built-in
at the moment.

Second question: is message 4 ignored?  I would prefer to do so, to
allow coexistence of tramp-imap.el with other messages (or even later
versions of tramp-imap.el).

I think it's fine to use message UIDs as the true file name, but we need
to decide what to do in the cases above.

Third question: namespaces.  I feel that it's much better
for the user to store all the files in a single mailbox:

INBOX.storage holds /a/b/c, /d/e/f, and /g/h/i

I believe you proposed that instead we should auto-create:

INBOX.storage.a.b to hold /a/b/c
INBOX.storage.d.e to hold /d/e/f
INBOX.storage.g.h to hold /g/h/i

Did I understand you correctly?  Maybe we could do both, allowing for a
"root mailbox" and a "root prefix".  In the first case, those would be

INBOX.storage /

and in the second case, they would be

INBOX.storage.a.b /a/b
INBOX.storage.d.e /d/e
INBOX.storage.g.h /g/h

And maybe the user can configure those mappings exactly.  I still think
it's too complicated for 99% of the users and they'll never need more
than one mailbox per virtual filesystem, but if you disagree I can do
the extra work.

I'll assume we've picked the single mailbox approach for the rest of the
message.  The implementations will change quite a bit as far as
directories are concerned if we use multiple mailboxes.

MA> As starting point you might look at tramp-smb.el or tramp-gw.el. Both
MA> are addons, like tramp-imap.el could be. tramp-fish.el might be examined
MA> as well, but this method isn't used anywhere I believe - it was merely a
MA> proof of concept I didn't want to throw away. And I never ever got a bug
MA> report about ...

OK, I'll look.  tramp-gw.el doesn't have any Emacs primitives
implemented at first glance, so I'll look at tramp-smb.el which defines
all those mappings nicely in tramp-smb-file-name-handler-alist.  That
list has quite a few methods; are they all necessary or do the default
handlers for some of them suffice?

Here's the list (omitting the commented-out items from tramp-smb.el)
with some questions and comments.  Sorry if most of this is obvious,
this is my first time diving into the Tramp code.  What function would
handle the truename-to-visualname translation?

add-name-to-file: could be a special "link message" or just a copy, like in Windows
copy-file: implemented as an APPEND
delete-directory: implemented with a search+delete for all matching messages
delete-file: search+delete of all matching messages
directory-file-name: tramp-handle-directory-file-name?
directory-files: search for matches
directory-files-and-attributes: search for matches, attributes always 777
dired-call-process: ignore
dired-compress-file: ignore
file-accessible-directory-p: always t
file-attributes: always 777
file-directory-p: needs a search, but we could have a file name that conflicts with a directory name
file-executable-p: always nil
file-exists-p: needs a search
file-local-copy: ?
file-remote-p: tramp-handle-file-remote-p
file-modes: tramp-handle-file-modes or hard-code
file-name-all-completions: needs a search
file-name-completion: tramp-handle-file-name-completion
file-name-directory: tramp-handle-file-name-directory
file-name-nondirectory: tramp-handle-file-name-nondirectory
file-newer-than-file-p: needs a search, plus backups may matter
file-ownership-preserved-p: ignore
file-readable-p: always t
file-regular-p: always t
file-symlink-p: always nil
file-truename: returns UID
file-writable-p: always t
find-backup-file-name: we need to decide
insert-directory: ?
insert-file-contents: search+retrieve
load: tramp-handle-load
make-directory, make-directory-internal: we need to decide what this should do
make-symbolic-link: ignore or implement as a special message
rename-file: append with new subject, delete original and its backups if
we decide to do backups
set-file-modes: ignore
set-visited-file-modtime: we could use a header for this
shell-command: ignore?
substitute-in-file-name: ?
unhandled-file-name-directory: tramp-handle-unhandled-file-name-directory?
vc-registered: always nil
verify-visited-file-modtime: ?
write-region: needs to do an append+delete of original+backups as needed; IMAP can't rewrite a message

Thanks
Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 03 Jan 2008 07:21:42 -0600 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Third question: namespaces.  I feel that it's much better
TZ> for the user to store all the files in a single mailbox:

TZ> INBOX.storage holds /a/b/c, /d/e/f, and /g/h/i

TZ> I believe you proposed that instead we should auto-create:

TZ> INBOX.storage.a.b to hold /a/b/c
TZ> INBOX.storage.d.e to hold /d/e/f
TZ> INBOX.storage.g.h to hold /g/h/i

I realized there is a third option: don't allow directories.  Everything
is at the top level /, and make-directory fails.  That would simplify
the code tremendously though users may not like it.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[James Cloos]

On namespaces in IMAP, do note that not every imapd uses . (full stop)
as the hierarchy separator character; many use / (virgule).  The LIST
command will show what that server uses.

Also, some (many?) users will probably not like the idea of turning
INBOX into a parent.  That convention seems to be a uw thing, AFAICT.

Were I to use this I’d want to put it all under something like
Emacs/Tramp/ or so.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 03 Jan 2008 17:31:14 +0000 James Cloos <cloos+math_uh-ding@jhcloos.com> wrote: 

JC> On namespaces in IMAP, do note that not every imapd uses . (full stop)
JC> as the hierarchy separator character; many use / (virgule).  The LIST
JC> command will show what that server uses.

Right, I did not mention that but knew it :)

JC> Also, some (many?) users will probably not like the idea of turning
JC> INBOX into a parent.  That convention seems to be a uw thing, AFAICT.

We'll definitely have any mailbox be the top parent (not just INBOX),
the question is whether other mailboxes under it will implement a
directory hierarchy, or if the hierarchy will be entirely inside a
single mailbox, or if we won't have directories at all.

Thanks
Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Say you have these article UIDs and subjects, and UIDs are ordered by
> date.  Note the ENCODED_FILENAME will be the filename encoded in some
> way and /a/b/c is the filename without any risky characters; Gnus has
> the subject encoding functionality already.  The subject format is
> specifically to allow easy searching.
>
> 1 "tramp-imap-marker /a/b/c ENCODED_ABC"
> 2 "tramp-imap-marker /a/b/c ENCODED_ABC"
> 3 "tramp-imap-marker /a/b/c ENCODED_ABC"
> 4 "hello world"
>
> Do we 
>
> A) allow for three identically named files, dynamically renaming 2 and 3
> with the Emacs <1> and <2> convention?
>
> B) ignore 1 and 2, never create them intentionally, and (optionally)
> erase duplicates when a file is saved?  This is probably easiest to
> implement in code.
>
> C) treat 1 and 2 as backups (there will be a tramp-imap-backups
> parameter to control how many are allowed per filename, and older ones
> over that limit are erased when a file is saved)?  We'll need a
> mechanism to revert to a backup, which I don't think Tramp has built-in
> at the moment.

If 1, 2 and 3 are created only by tramp-imap, then C might be the
preferred choice. B doesn't look good to me, I believe all files must
be accessible.

> Second question: is message 4 ignored?  I would prefer to do so, to
> allow coexistence of tramp-imap.el with other messages (or even later
> versions of tramp-imap.el).

Maybe. But then tramp-imap would only access files it has created
itself, this might be a restriction.

Of course it wouldn't be good if tramp-imap would change subject lines
of existing messages. Do we need to encode the filename in the subject
line? Couldn't it be any other header, like X-Tramp-Filename?

> Third question: namespaces.  I feel that it's much better
> for the user to store all the files in a single mailbox:

[...]

Honestly, I'm not so familar with IMAP conventions. I have no
precedence for this, you might decide yourself.

> Here's the list (omitting the commented-out items from tramp-smb.el)
> with some questions and comments.  Sorry if most of this is obvious,
> this is my first time diving into the Tramp code.  What function would
> handle the truename-to-visualname translation?

There is an "atomic" function tramp-smb-send-command, which makes the
communication with the server. Similar functions are
tramp-send-command and tramp-send-fish-command in their respective
packages. I guess you'll need a tramp-imap-send-command or so; this
could be a central place for the mapping. Other places might be needed
as well when *interpreting* the result of a sent command.

[...]

> add-name-to-file: could be a special "link message" or just a copy, like in Windows

Both is possible. Btw, on my todo is implementing a real link for
Windows, but this has no priority.

> copy-file: implemented as an APPEND
> delete-directory: implemented with a search+delete for all matching messages

OK.

> delete-file: search+delete of all matching messages

It depends how we decide with your first question. If there is
uniqueness in mapping file names to true names, than there is only one
message to be deleted.

> directory-file-name: tramp-handle-directory-file-name?
> directory-files: search for matches

OK.
> directory-files-and-attributes: search for matches, attributes always 777

It returns more than just file permissions.

> dired-call-process: ignore

OK.

> dired-compress-file: ignore

Likely yes, but there might be an implementation depending on how a
message is stored.

> file-accessible-directory-p: always t

No. Use default implementation (which checks both file-directory-p and
file-executable-p).

> file-attributes: always 777

No. file-attributes returns a list of 12 different attributes. Must be
implemented.

> file-directory-p: needs a search, but we could have a file name that conflicts with a directory name

We shall avoid such conflicts. A file name shall be unique.

> file-executable-p: always nil

Better is t (see file-accessible-directory-p). It doesn't hurt,
because we don't support processes.

> file-exists-p: needs a search

OK.

> file-local-copy: ?

That is the basic function for retrieving a file from the server. It
stores the file in the temp directory; some other functions use it
then (like insert-file-contents etc).

> file-remote-p: tramp-handle-file-remote-p

OK

> file-modes: tramp-handle-file-modes or hard-code

Both is possible. tramp-handle-file-modes is shorter to write.

> file-name-all-completions: needs a search
> file-name-completion: tramp-handle-file-name-completion
> file-name-directory: tramp-handle-file-name-directory
> file-name-nondirectory: tramp-handle-file-name-nondirectory
> file-newer-than-file-p: needs a search, plus backups may matter
> file-ownership-preserved-p: ignore

OK

> file-readable-p: always t
> file-regular-p: always t

No. You must return nil when the file does not exist.

> file-symlink-p: always nil
> file-truename: returns UID

OK

> file-writable-p: always t

No, again the case of non existing file names.

> find-backup-file-name: we need to decide
> insert-directory: ?

Both can be postponed, they are not basic functions.

> insert-file-contents: search+retrieve

file-local-copy and insert-file-contents of that local copy.

> load: tramp-handle-load
> make-directory, make-directory-internal: we need to decide what this should do
> make-symbolic-link: ignore or implement as a special message
> rename-file: append with new subject, delete original and its backups if
> we decide to do backups
> set-file-modes: ignore
> set-visited-file-modtime: we could use a header for this

OK

> shell-command: ignore?

Yes

> substitute-in-file-name: ?

tramp-handle-substitute-in-file-name

> unhandled-file-name-directory: tramp-handle-unhandled-file-name-directory?
> vc-registered: always nil

OK

> verify-visited-file-modtime: ?

Maybe ignore for the time being, it's not core functionality.

> write-region: needs to do an append+delete of original+backups as needed; IMAP can't rewrite a message

OK



I would start with file-local-copy and write-region, these are the
functions for retrieve and store a file.

> Thanks
> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Magnus Henoch]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Using imap.el and encrypt.el, I was able to store files in IMAP.
> [...]

A year later, I had the same idea and found this article :) What
happened to this?  Is there a more complete trimp.el / tramp-imap.el
that I can start hacking at?

[Followup-To set to tramp list]

Magnus

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Sun, 12 Apr 2009 15:04:24 +0200 Magnus Henoch <mange@freemail.hu> wrote: 

MH> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Using imap.el and encrypt.el, I was able to store files in IMAP.
>> [...]

MH> A year later, I had the same idea and found this article :) What
MH> happened to this?  Is there a more complete trimp.el / tramp-imap.el
MH> that I can start hacking at?

No news, unfortunately, I haven't had the time to play with it.  It was
next on my TODO list, actually, but that has a 1+ month granulatiry :)

I was about to start adding the TRAMP compatibility functions that
Michael Albinus kindly told me were needed.  The underlying store/fetch
functionality should be all right, except it doesn't do attachments so
the message is always one big text chunk.  You could, instead, make the
message contain metadata and put the payload in an attachment.

It's all yours if you want it.  I will gladly help with testing and
whatever else you need.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It's all yours if you want it.  I will gladly help with testing and
> whatever else you need.

As a first step, you could add the current version to the Tramp CVS
repository. This might ease to work on it.

> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Tue, 14 Apr 2009 10:44:56 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> It's all yours if you want it.  I will gladly help with testing and
>> whatever else you need.

MA> As a first step, you could add the current version to the Tramp CVS
MA> repository. This might ease to work on it.

It's really, really embarassing to check that messy pile of code into
Tramp.  Most of it is just TODO information.  I am attaching it for
yours and Magnus' review.  The encrypt-* calls should be replaced by
auth-source-* calls, as well.  If you really want me to check it in, I
will (and if this is a sneaky way to get me to fix it up and write the
missing code, it's working :)

Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 10760 bytes --]

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It's really, really embarassing to check that messy pile of code into
> Tramp.  Most of it is just TODO information.  I am attaching it for
> yours and Magnus' review.  The encrypt-* calls should be replaced by
> auth-source-* calls, as well.

No problem for me if you check it in. As long as we don't add it to the
makefiles, and we don't load it in tramp.el, it doesn't hurt.

Please adjust the copyright years, and replace GPLv2 by GPLv3.

> If you really want me to check it in, I
> will (and if this is a sneaky way to get me to fix it up and write the
> missing code, it's working :)

This sounds like a good plan :-)

Anyway, I will comment the code in more detail next days, likely.

> Ted

Best regards, Michael.

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2740 bytes --]

On Wed, 15 Apr 2009 23:03:16 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> No problem for me if you check it in. As long as we don't add it to the
MA> makefiles, and we don't load it in tramp.el, it doesn't hurt.

MA> Please adjust the copyright years, and replace GPLv2 by GPLv3.

Let's wait until it at least does completion, reads, and writes, please.

I made the year and GPL fixes.

The current code will:

- install many handlers, many unimplemented (copied from tramp-fish.el)

- handle "/imap:blockstar.com:/INBOX.test/" for filename expansion (but
  the actual expansion is not working; tramp-imap-get-file-entries does
  not get the correct file name from Emacs)

- save /imap:blockstar.com:/INBOX.test/new via tramp-imap-put-file (uses PGG)

- read /imap:blockstar.com:/INBOX.test/new via tramp-imap-get-file (uses PGG)

- use auth-source to get user name and password for a server

- maintain an IMAP connection in the Tramp buffer specific for this
  connection

- show you how much nasty code I can produce in a few hours

My biggest problem right now is that I can't get filename completion to
work (as mentioned above, tramp-imap-get-file-entries does not get the
right name).  I'm sure I'm missing a lot of details here, but ideally I
should parse the filename myself without any interferences from Emacs or
from Tramp (everything else--port, username, server, etc.--can be parsed
by Tramp).

Here's how I want the parsing to work:

/imap:SERVER:/MAILBOX/

hitting TAB after MAILBOX or after MAILBOX/ should complete
"/imap:SERVER:/MAILBOX" with or without the trailing slash.  MAILBOX is
the directory.

/imap:SERVER:/MAILBOX/anything

hitting TAB at the end should complete all the messages that match
"anything".

/imap:SERVER:/MAILBOX/anything/else

The mailbox here will be "MAILBOX/anything" and completion will look for
messages that match "else"

So basically the mailbox name is everything up to the last / character,
like a `dirname' call.  IMAP mailboxes can contain many characters,
including /, so we need to forbid those characters in the file name.
The base file name is then just `basename' on the full name.  I tried to
implement this by hand in tramp-imap-file-name-mailbox-or-name but there
must be a better way.

My second problem is that messages are not yet sorted by date so you are
not guaranteed to get the latest version (we agreed not to delete the
older versions, though there probably should be a flexible way to mark
them deleted past N backups).

Third, ranges are not supported for most commands.

The last major issue is that error handling and PGG integration are not
good.  Usability is not good.

If anyone can help me with the above items, I'd appreciate it.

Thanks
Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 25900 bytes --]

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Daiki Ueno]

>>>>> In <86k55cb4ye.fsf@jumptrading.com> 
>>>>>	Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Wed, 15 Apr 2009 23:03:16 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

> - save /imap:blockstar.com:/INBOX.test/new via tramp-imap-put-file (uses PGG)
> - read /imap:blockstar.com:/INBOX.test/new via tramp-imap-get-file (uses PGG)

I think this (implicit encryption) is a misdesgined feature which should
be removed.  Why don't you leave the job to auto-encryption-mode so that
a user can choose whether to encrypt a file by adding ".gpg"?

Regards,
-- 
Daiki Ueno

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 23 Apr 2009 11:45:04 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

>>>>>> In <86k55cb4ye.fsf@jumptrading.com> 
>>>>>> Ted Zlatanov <tzz@lifelogs.com> wrote:
>> On Wed, 15 Apr 2009 23:03:16 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

>> - save /imap:blockstar.com:/INBOX.test/new via tramp-imap-put-file (uses PGG)
>> - read /imap:blockstar.com:/INBOX.test/new via tramp-imap-get-file (uses PGG)

DU> I think this (implicit encryption) is a misdesgined feature which should
DU> be removed.  Why don't you leave the job to auto-encryption-mode so that
DU> a user can choose whether to encrypt a file by adding ".gpg"?

Unlike filesystems, IMAP servers are a semi-public space, in the sense
that the vast majority of the users can't expect security or privacy
beyond what the server administrators and owners are willing to grant.
Very, very few users actually own and control their IMAP servers.  Thus,
encrypting by default is IMHO the best option and the only one I
provided in this initial version.

I disagree the user should choose whether to encrypt, especially by the
burdensome mechanism of changing the file extension.  The majority of
users will use the default setting, whatever it is.  I believe the
default setting of tramp-imap.el should be to encrypt.

If you or anyone else likes, feel free to provide a patch that will
support a nil passphrase to mean no encryption should be performed.  The
usual Emacs-style settings for the passphrase (alist and function in
addition to a string) which I plan to add eventually can return nil too
for the same effect.  I can't accept that as the tramp-imap default,
though, and I don't have the time currently to write that support
myself.

Thanks
Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Daiki Ueno]

>>>>> In <86fxfzbkic.fsf@lifelogs.com> 
>>>>>	Ted Zlatanov <tzz@lifelogs.com> wrote:
> On Thu, 23 Apr 2009 11:45:04 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 
> >> - save /imap:blockstar.com:/INBOX.test/new via tramp-imap-put-file (uses PGG)
> >> - read /imap:blockstar.com:/INBOX.test/new via tramp-imap-get-file (uses PGG)

DU> I think this (implicit encryption) is a misdesgined feature which should
DU> be removed.  Why don't you leave the job to auto-encryption-mode so that
DU> a user can choose whether to encrypt a file by adding ".gpg"?

> Unlike filesystems, IMAP servers are a semi-public space,

Really?  I use 3 IMAP servers for everyday use and they properly
implement access control like file systems.  In other words, no one can
read my private emails unless s/he shares the passphrase with me.

Otherwise, it's a horrible situation...

> in the sense that the vast majority of the users can't expect security
> or privacy beyond what the server administrators and owners are
> willing to grant.  Very, very few users actually own and control their
> IMAP servers.  Thus, encrypting by default is IMHO the best option and
> the only one I provided in this initial version.

I think you always lack "use-case analysis".

Regards,
-- 
Daiki Ueno

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 23 Apr 2009 22:09:52 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

>>>>>> In <86fxfzbkic.fsf@lifelogs.com> 
>>>>>> Ted Zlatanov <tzz@lifelogs.com> wrote:
>> On Thu, 23 Apr 2009 11:45:04 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 
>> >> - save /imap:blockstar.com:/INBOX.test/new via tramp-imap-put-file (uses PGG)
>> >> - read /imap:blockstar.com:/INBOX.test/new via tramp-imap-get-file (uses PGG)

DU> I think this (implicit encryption) is a misdesgined feature which should
DU> be removed.  Why don't you leave the job to auto-encryption-mode so that
DU> a user can choose whether to encrypt a file by adding ".gpg"?

>> Unlike filesystems, IMAP servers are a semi-public space,

DU> Really?  I use 3 IMAP servers for everyday use and they properly
DU> implement access control like file systems.  In other words, no one can
DU> read my private emails unless s/he shares the passphrase with me.

Yes, really.  I was not talking about other IMAP users.  Whoever owns
your IMAP servers can read your unencrypted e-mails with little effort.
If you trust your 3 servers, that's great.  The vast majority of IMAP
users, as I said, have no expectation of privacy on their IMAP server
(consider GMail, for instance).  This is not a filesystem on a remote
server where you can trust user permissions to be managed correctly.

>> Very, very few users actually own and control their IMAP servers.
>> Thus, encrypting by default is IMHO the best option and the only one
>> I provided in this initial version.

DU> I think you always lack "use-case analysis".

I guess we'll have to disagree.  As I said, feel free to provide patches
to make turning encryption off easier, and of course if a vote goes
against my design I'll reconsider it.

Thanks
Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Daiki Ueno]

>>>>> In <86zle7a3in.fsf@lifelogs.com> 
>>>>>	Ted Zlatanov <tzz@lifelogs.com> wrote:
> If you trust your 3 servers, that's great.  The vast majority of IMAP
> users, as I said, have no expectation of privacy on their IMAP server
> (consider GMail, for instance).  This is not a filesystem on a remote
> server where you can trust user permissions to be managed correctly.

I agree with the facts.  But in that case, you cannot even trust your
ISP which delivers original e-mails, can you?

Do you think that Gnus should encrypt all the outgoing e-mails by
default?
-- 
Daiki Ueno

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 23 Apr 2009 22:48:39 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

>>>>>> In <86zle7a3in.fsf@lifelogs.com> 
>>>>>> Ted Zlatanov <tzz@lifelogs.com> wrote:
>> If you trust your 3 servers, that's great.  The vast majority of IMAP
>> users, as I said, have no expectation of privacy on their IMAP server
>> (consider GMail, for instance).  This is not a filesystem on a remote
>> server where you can trust user permissions to be managed correctly.

DU> I agree with the facts.  But in that case, you cannot even trust your
DU> ISP which delivers original e-mails, can you?

Users expect a higher level of security with their files than with
e-mail.  I think tramp-imap.el should provide it by default.  Doing less
is easier and lets us blame the user later ("Why didn't you read the
manual?  It clearly says to turn encryption on or specify the .gpg
extension!") but I don't like that approach.

DU> Do you think that Gnus should encrypt all the outgoing e-mails by
DU> default?

No, that's completely different from what we're discussing.  We're
discussing file storage over IMAP, not e-mail security.

Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

>>>>> In <7d371a60-fbac-4b0e-a6b3-2001292f0205@broken.deisui.org> 
>>>>>	Daiki Ueno <ueno@unixuser.org> wrote:
> >>>>> In <86zle7a3in.fsf@lifelogs.com> 
> >>>>>	Ted Zlatanov <tzz@lifelogs.com> wrote:
> > If you trust your 3 servers, that's great.  The vast majority of IMAP
> > users, as I said, have no expectation of privacy on their IMAP server
> > (consider GMail, for instance).  This is not a filesystem on a remote
> > server where you can trust user permissions to be managed correctly.

> I agree with the facts.  But in that case, you cannot even trust your
> ISP which delivers original e-mails, can you?

Ah, I see my confusion about e-mail systems and IMAP as a file system.
That makes sense.  I apologize to you for that.

But still, I'm against the use of PGG (which is on the way to
obsolescence) in the new package.  Here is a patch to use EPG instead.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tramp-imap.el.diff --]
[-- Type: text/x-diff, Size: 2840 bytes --]

--- tramp-imap.el~	2009-04-23 23:13:51.000000000 +0900
+++ tramp-imap.el	2009-04-23 23:25:46.000000000 +0900
@@ -247,7 +247,7 @@
 (require 'tramp-cache)
 (require 'tramp-compat)
 (require 'imap)
-(require 'pgg)
+(require 'epa)
 (autoload 'auth-source-user-or-password "auth-source")
 
 ;; Define Tramp IMAP method ...
@@ -627,8 +627,7 @@
       (goto-char (point-min))
       (while (re-search-forward "
" nil t)
 	(replace-match "" nil nil))
-      ;; TODO: catch errors from PGG here
-      (tramp-imap-pgg-decode-buffer))))
+      (tramp-imap-decode-buffer))))
 
 ;;; (tramp-imap-collapse-name "a b c / where ; strange ! characters $ abound") 
 ;;; => "abcwherestrangecharactersabound"
@@ -655,29 +654,40 @@
 		(with-current-buffer filename-or-buffer
 		  (buffer-string)))
 	     (insert-file-contents filename-or-buffer))
-	   (tramp-imap-pgg-encode-buffer))))
+	   (tramp-imap-encode-buffer))))
       (current-buffer)))
 
-(defun tramp-imap-pgg-encode-buffer ()
-  (tramp-imap-pgg-process-buffer t))
-
-(defun tramp-imap-pgg-decode-buffer ()
-  (tramp-imap-pgg-process-buffer))
-
-(defun tramp-imap-pgg-process-buffer (&optional encode)
-  "Use PGG to encode or decode the current buffer."
-  (let ((pfft (if encode 'pgg-encrypt-symmetric 'pgg-decrypt))
-	(default-enable-multibyte-characters nil)
-	(input (buffer-substring-no-properties (point-min) (point-max)))
-	exit-data)
-    (with-temp-buffer
-      (insert input)
-      ;; note that we call pfft before pgg-display-output-buffer
-      (pgg-display-output-buffer (point-min) (point-max) (funcall pfft (point-min) (point-max) tramp-imap-passphrase))
-      (setq exit-data
-	    (buffer-substring-no-properties (point-min) (point-max))))
-;    (debug exit-data)
-    exit-data))
+(defun tramp-imap-encode-buffer ()
+  (let ((context (epg-make-context 'OpenPGP))
+	cipher)
+    (epg-context-set-armor context t)
+    (epg-context-set-passphrase-callback context
+					 #'epa-passphrase-callback-function)
+    (epg-context-set-progress-callback context
+				       (cons #'epa-progress-callback-function
+					     "Encrypting..."))
+    (message "Encrypting...")
+    (setq cipher (epg-encrypt-string
+		  context
+		  (encode-coding-string (buffer-string) 'utf-8)
+		  nil))
+    (message "Encrypting...done")
+    cipher))
+
+(defun tramp-imap-decode-buffer ()
+  (let ((context (epg-make-context 'OpenPGP))
+	plain)
+    (epg-context-set-passphrase-callback context
+					 #'epa-passphrase-callback-function)
+    (epg-context-set-progress-callback context
+				       (cons #'epa-progress-callback-function
+					     "Decrypting..."))
+    (message "Decrypting...")
+    (setq plain (decode-coding-string
+		 (epg-decrypt-string context (buffer-string))
+		 'utf-8))
+    (message "Decrypting...done")
+    plain))
 
 (provide 'tramp-imap)
 ;;; tramp-imap.el ends here

[-- Attachment #3: Type: text/plain, Size: 25 bytes --]


Regards,
-- 
Daiki Ueno

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

On Thu, 23 Apr 2009 23:28:08 +0900 Daiki Ueno <ueno@unixuser.org> wrote: 

DU> But still, I'm against the use of PGG (which is on the way to
DU> obsolescence) in the new package.  Here is a patch to use EPG instead.

Wonderful.  Two questions:

- EPG asks for the passphrase.  It should try tramp-imap-passphrase as a
  string first (I'll add the alist and function call support later).  As
  an extra courtesy to the user, if the passphrase is nil, we should ask
  interactively if encryption is wanted, and if yes, read the
  passphrase.  Would you be willing to make that addition?

- I had to clean ^M characters from the buffer before; EPG seems to
  handle them automatically so I removed this code

      ;; TODO: do this better, removes all ^M in the buffer
      (goto-char (point-min))
       (while (re-search-forward "^M" nil t) ;; this was the literal ^M
 	(replace-match "" nil nil))

  from tramp-imap-get-file.  Is that OK?

Thank you
Ted

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

On Wed, 22 Apr 2009 19:15:36 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> So basically the mailbox name is everything up to the last / character,
TZ> like a `dirname' call.  IMAP mailboxes can contain many characters,
TZ> including /, so we need to forbid those characters in the file name.
TZ> The base file name is then just `basename' on the full name.  I tried to
TZ> implement this by hand in tramp-imap-file-name-mailbox-or-name but there
TZ> must be a better way.

I fixed this in the code, and now it seems like filename completion
works too.  I always convert "/INBOX.test/a/b/filename" to
mailbox "INBOX.test/a/b" and message name "filename".  In addition, I
collapse the file name to make searching and usage easier by subject,
while keeping the real file name in the X-Tramp-IMAP header.

I made many other fixes so please use the attached file as a baseline if
you want to submit patches for any of the issues I listed before, or any
new issues you notice.  It's getting close to usable.

Thanks
Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 29868 bytes --]

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Daiki Ueno]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

>>>>> In <86r5zj9z96.fsf@lifelogs.com> 
>>>>>	Ted Zlatanov <tzz@lifelogs.com> wrote:
> - EPG asks for the passphrase.  It should try tramp-imap-passphrase as a
>   string first (I'll add the alist and function call support later).  As
>   an extra courtesy to the user, if the passphrase is nil, we should ask
>   interactively if encryption is wanted, and if yes, read the
>   passphrase.  Would you be willing to make that addition?

Yes, here is a patch to do that.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tramp-imap.el.diff --]
[-- Type: text/x-diff, Size: 1300 bytes --]

--- tramp-imap.el~	2009-04-24 08:59:11.000000000 +0900
+++ tramp-imap.el	2009-04-24 09:01:16.000000000 +0900
@@ -738,12 +738,17 @@
 	(error "Could not find the body separator in the encoded message!")))
       (current-buffer)))
 
+(defun tramp-imap-passphrase-callback-function (context key-id handback)
+  (or tramp-imap-passphrase
+      (epa-passphrase-callback-function context key-id handback)))
+
 (defun tramp-imap-encode-buffer ()
   (let ((context (epg-make-context 'OpenPGP))
 	cipher)
     (epg-context-set-armor context t)
-    (epg-context-set-passphrase-callback context
-					 #'epa-passphrase-callback-function)
+    (epg-context-set-passphrase-callback
+     context
+     #'tramp-imap-passphrase-callback-function)
     (epg-context-set-progress-callback context
 				       (cons #'epa-progress-callback-function
 					     "Encrypting..."))
@@ -758,8 +763,9 @@
 (defun tramp-imap-decode-buffer ()
   (let ((context (epg-make-context 'OpenPGP))
 	plain)
-    (epg-context-set-passphrase-callback context
-					 #'epa-passphrase-callback-function)
+    (epg-context-set-passphrase-callback
+     context
+     #'tramp-imap-passphrase-callback-function)
     (epg-context-set-progress-callback context
 				       (cons #'epa-progress-callback-function
 					     "Decrypting..."))

[-- Attachment #3: Type: text/plain, Size: 476 bytes --]


> - I had to clean ^M characters from the buffer before; EPG seems to
>   handle them automatically so I removed this code

>       ;; TODO: do this better, removes all ^M in the buffer
>       (goto-char (point-min))
>        (while (re-search-forward "^M" nil t) ;; this was the literal ^M
>  	(replace-match "" nil nil))

>   from tramp-imap-get-file.  Is that OK?

I think so - if data do not roundtrip before/after encryption, it seems
a bug...

Regards,
-- 
Daiki Ueno

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Michael Albinus]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 10404 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I made many other fixes so please use the attached file as a baseline if
> you want to submit patches for any of the issues I listed before, or any
> new issues you notice.  It's getting close to usable.

I've started to play with it. First error:

Debugger entered--Lisp error: (wrong-number-of-arguments #[nil "ŠdbˆÀÁÂÃ#…\x13

" nil t delete-char 1] 4 ("/usr/local/share/emacs/23.0.92/lisp/gnus/nnheader.elc" . 25599)] 1)
  nnheader-ms-strip-cr("X-Tramp-Imap: *scratch*
\nFrom: Tramp-Imap <tramp-imap@tramp-imap.tramp.emacs.org>
\nTo: Tramp-Imap <tramp-imap@tramp-imap.tramp.emacs.org>
\nSubject: tramp-imap-subject-marker fstab
\nDate: Sat, 25 Apr 2009 19:08:13 +0200
\nMessage-ID: <87prf04q6a.fsf@gmx.de>
\n
\n")
  tramp-imap-get-message-headers(1)
  (nnheader-ms-strip-cr (tramp-imap-get-message-headers msgs))
  (let ((headers ...)) (when (string-match "^X-Tramp-IMAP: \\(.+\\)" headers) (match-string 1 headers)))
  tramp-imap-get-message-x-tramp-imap(1)
  (setq sname (tramp-imap-get-message-x-tramp-imap msg))
  (while --cl-dolist-temp-- (setq msg (car --cl-dolist-temp--)) (setq sname (tramp-imap-get-message-x-tramp-imap msg)) (when (and sname ...) (push ... res) (tramp-debug-message vec "found message %s: %s" msg ...)) (setq --cl-dolist-temp-- (cdr --cl-dolist-temp--)))
  (let ((--cl-dolist-temp-- ...) msg) (while --cl-dolist-temp-- (setq msg ...) (setq sname ...) (when ... ... ...) (setq --cl-dolist-temp-- ...)) nil)
  (catch (quote --cl-block-nil--) (let (... msg) (while --cl-dolist-temp-- ... ... ... ...) nil))
  (cl-block-wrapper (catch (quote --cl-block-nil--) (let ... ... nil)))
  (block nil (let (... msg) (while --cl-dolist-temp-- ... ... ... ...) nil))
  (dolist (msg (imap-search ...)) (setq sname (tramp-imap-get-message-x-tramp-imap msg)) (when (and sname ...) (push ... res) (tramp-debug-message vec "found message %s: %s" msg ...)))
  (let (sname) (tramp-debug-message vec "looking for '%s'" search-name) (dolist (msg ...) (setq sname ...) (when ... ... ...)))
  (if (and name (imap-mailbox-select mbox)) (let (sname) (tramp-debug-message vec "looking for '%s'" search-name) (dolist ... ... ...)) (tramp-error vec (quote none) "bad name %s or mailbox %s" name mbox))
  (let* ((mbox ...) (name ...) (truename ...) (search-name ...) res) (tramp-debug-message vec "selecting mbox %s" mbox) (if (and name ...) (let ... ... ...) (tramp-error vec ... "bad name %s or mailbox %s" name mbox)) res)
  (save-current-buffer (set-buffer (tramp-imap-buffer vec)) (let* (... ... ... ... res) (tramp-debug-message vec "selecting mbox %s" mbox) (if ... ... ...) res))
  (with-current-buffer (tramp-imap-buffer vec) (let* (... ... ... ... res) (tramp-debug-message vec "selecting mbox %s" mbox) (if ... ... ...) res))
  tramp-imap-get-file-entries(["imap" "albinus" "localhost" "/INBOX.test"])
  (car (tramp-imap-get-file-entries v))
  (cdr (car (tramp-imap-get-file-entries v)))
  (progn (cdr (car ...)))
  (setq value (progn (cdr ...)))
  (progn (setq value (progn ...)) (tramp-set-file-property v localname (format "file-attributes-%s" id-format) value))
  (if (eq value (quote undef)) (progn (setq value ...) (tramp-set-file-property v localname ... value)))
  (when (eq value (quote undef)) (setq value (progn ...)) (tramp-set-file-property v localname (format "file-attributes-%s" id-format) value))
  (let ((value ...)) (when (eq value ...) (setq value ...) (tramp-set-file-property v localname ... value)) value)
  (if (file-name-absolute-p localname) (let (...) (when ... ... ...) value) (cdr (car ...)))
  (with-file-property v localname (format "file-attributes-%s" id-format) (cdr (car ...)))
  (let* ((v ...) (method ...) (user ...) (host ...) (localname ...)) (with-file-property v localname (format "file-attributes-%s" id-format) (cdr ...)))
  (with-parsed-tramp-file-name (expand-file-name filename) nil (with-file-property v localname (format "file-attributes-%s" id-format) (cdr ...)))
  tramp-imap-handle-file-attributes("/imap:albinus@localhost:/INBOX.test")
  apply(tramp-imap-handle-file-attributes "/imap:albinus@localhost:/INBOX.test")
  (progn (apply (cdr fn) args))
  (unwind-protect (progn (apply ... args)) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal ...)) (unwind-protect (progn ...) (set-match-data save-match-data-internal ...)))
  (save-match-data (apply (cdr fn) args))
  (if fn (save-match-data (apply ... args)) (tramp-run-real-handler operation args))
  (let ((fn ...)) (if fn (save-match-data ...) (tramp-run-real-handler operation args)))
  tramp-imap-file-name-handler(file-attributes "/imap:albinus@localhost:/INBOX.test")
  apply(tramp-imap-file-name-handler file-attributes "/imap:albinus@localhost:/INBOX.test")
  (cond ((and completion ... ...) t) ((and completion ... ...) filename) (foreign (apply foreign operation args)) (t (tramp-run-real-handler operation args)))
  (let* ((v ...) (method ...) (user ...) (host ...) (localname ...)) (cond (... t) (... filename) (foreign ...) (t ...)))
  (with-parsed-tramp-file-name filename nil (cond (... t) (... filename) (foreign ...) (t ...)))
  (let* ((filename ...) (completion ...) (foreign ...)) (with-parsed-tramp-file-name filename nil (cond ... ... ... ...)))
  (progn (let* (... ... ...) (with-parsed-tramp-file-name filename nil ...)))
  (unwind-protect (progn (let* ... ...)) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal ...)) (unwind-protect (progn ...) (set-match-data save-match-data-internal ...)))
  (save-match-data (let* (... ... ...) (with-parsed-tramp-file-name filename nil ...)))
  (if tramp-mode (save-match-data (let* ... ...)) (tramp-run-real-handler operation args))
  tramp-file-name-handler(file-attributes "/imap:albinus@localhost:/INBOX.test")
  file-attributes("/imap:albinus@localhost:/INBOX.test")
  (car (file-attributes filename))
  (let ((x ...)) (when (stringp x) (if ... ... x)))
  (let* ((v ...) (method ...) (user ...) (host ...) (localname ...)) (let (...) (when ... ...)))
  (with-parsed-tramp-file-name filename nil (let (...) (when ... ...)))
  tramp-handle-file-symlink-p("/imap:albinus@localhost:/INBOX.test")
  apply(tramp-handle-file-symlink-p "/imap:albinus@localhost:/INBOX.test")
  (progn (apply (cdr fn) args))
  (unwind-protect (progn (apply ... args)) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal ...)) (unwind-protect (progn ...) (set-match-data save-match-data-internal ...)))
  (save-match-data (apply (cdr fn) args))
  (if fn (save-match-data (apply ... args)) (tramp-run-real-handler operation args))
  (let ((fn ...)) (if fn (save-match-data ...) (tramp-run-real-handler operation args)))
  tramp-imap-file-name-handler(file-symlink-p "/imap:albinus@localhost:/INBOX.test")
  apply(tramp-imap-file-name-handler file-symlink-p "/imap:albinus@localhost:/INBOX.test")
  (cond ((and completion ... ...) t) ((and completion ... ...) filename) (foreign (apply foreign operation args)) (t (tramp-run-real-handler operation args)))
  (let* ((v ...) (method ...) (user ...) (host ...) (localname ...)) (cond (... t) (... filename) (foreign ...) (t ...)))
  (with-parsed-tramp-file-name filename nil (cond (... t) (... filename) (foreign ...) (t ...)))
  (let* ((filename ...) (completion ...) (foreign ...)) (with-parsed-tramp-file-name filename nil (cond ... ... ... ...)))
  (progn (let* (... ... ...) (with-parsed-tramp-file-name filename nil ...)))
  (unwind-protect (progn (let* ... ...)) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal ...)) (unwind-protect (progn ...) (set-match-data save-match-data-internal ...)))
  (save-match-data (let* (... ... ...) (with-parsed-tramp-file-name filename nil ...)))
  (if tramp-mode (save-match-data (let* ... ...)) (tramp-run-real-handler operation args))
  tramp-file-name-handler(file-symlink-p "/imap:albinus@localhost:/INBOX.test")
  file-symlink-p("/imap:albinus@localhost:/INBOX.test")
  file-truename("/imap:albinus@localhost:/INBOX.test" (98) (nil))
  file-truename("/imap:albinus@localhost:/INBOX.test/bbb")
  apply(file-truename "/imap:albinus@localhost:/INBOX.test/bbb")
  (let* ((inhibit-file-name-handlers ...) (inhibit-file-name-operation operation)) (apply operation args))
  tramp-run-real-handler(file-truename ("/imap:albinus@localhost:/INBOX.test/bbb"))
  (if fn (save-match-data (apply ... args)) (tramp-run-real-handler operation args))
  (let ((fn ...)) (if fn (save-match-data ...) (tramp-run-real-handler operation args)))
  tramp-imap-file-name-handler(file-truename "/imap:albinus@localhost:/INBOX.test/bbb")
  apply(tramp-imap-file-name-handler file-truename "/imap:albinus@localhost:/INBOX.test/bbb")
  (cond ((and completion ... ...) t) ((and completion ... ...) filename) (foreign (apply foreign operation args)) (t (tramp-run-real-handler operation args)))
  (let* ((v ...) (method ...) (user ...) (host ...) (localname ...)) (cond (... t) (... filename) (foreign ...) (t ...)))
  (with-parsed-tramp-file-name filename nil (cond (... t) (... filename) (foreign ...) (t ...)))
  (let* ((filename ...) (completion ...) (foreign ...)) (with-parsed-tramp-file-name filename nil (cond ... ... ... ...)))
  (progn (let* (... ... ...) (with-parsed-tramp-file-name filename nil ...)))
  (unwind-protect (progn (let* ... ...)) (set-match-data save-match-data-internal (quote evaporate)))
  (let ((save-match-data-internal ...)) (unwind-protect (progn ...) (set-match-data save-match-data-internal ...)))
  (save-match-data (let* (... ... ...) (with-parsed-tramp-file-name filename nil ...)))
  (if tramp-mode (save-match-data (let* ... ...)) (tramp-run-real-handler operation args))
  tramp-file-name-handler(file-truename "/imap:albinus@localhost:/INBOX.test/bbb")
  file-truename("/imap:albinus@localhost:/INBOX.test/bbb")
  set-visited-file-name("/imap:localhost:INBOX.test/bbb" nil)
  write-file("/imap:localhost:INBOX.test/bbb" t)
  call-interactively(write-file nil nil)

I'm using gnus as built in GNU Emacs 23.0.92. `nnheader-ms-strip-cr'
has no argument there.

You use by default 'ssl for connection. Can we expect this to be the
standard? Or shall we let the user decide, for example by two methods
"imap" and "imaps"?

I've also applied some few patches, you might check. See appended.

> Thanks
> Ted

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 5345 bytes --]

*** /home/albinus/lisp/tramp-imap.el.~1~	2009-04-25 18:27:52.000000000 +0200
--- /home/albinus/lisp/tramp-imap.el	2009-04-25 19:03:27.000000000 +0200
***************
*** 261,269 ****
  ;; ... and add it to the method list.
  (add-to-list 'tramp-methods (cons tramp-imap-method nil))
  
- ;; ... and add it to the method list.
- (add-to-list 'tramp-methods (cons tramp-imap-method nil))
- 
  ;; Add a default for `tramp-default-user-alist'. Default is the local user.
  (add-to-list 'tramp-default-user-alist
  	     `(,tramp-imap-method nil ,(user-login-name)))
--- 261,266 ----
***************
*** 336,354 ****
    "Alist of handler functions for Tramp IMAP method.
  Operations not mentioned here will be handled by the default Emacs primitives.")
  
- (defgroup tramp-imap nil
-   "Tramp over IMAP configuration."
-   :group 'applications)
- 
  (defcustom tramp-imap-subject-marker "tramp-imap-subject-marker"
    "The subject marker that Tramp-Imap will use."
    :type 'string
!   :group 'tramp-imap)
  
  (defcustom tramp-imap-passphrase "my passphrase"
    "The passphrase that Tramp-Imap will use.  TODO: allow function calls, etc. here"
    :type 'string
!   :group 'tramp-imap)
  
  (defun tramp-imap-file-name-p (filename)
    "Check if it's a filename for IMAP protocol."
--- 333,347 ----
    "Alist of handler functions for Tramp IMAP method.
  Operations not mentioned here will be handled by the default Emacs primitives.")
  
  (defcustom tramp-imap-subject-marker "tramp-imap-subject-marker"
    "The subject marker that Tramp-Imap will use."
    :type 'string
!   :group 'tramp)
  
  (defcustom tramp-imap-passphrase "my passphrase"
    "The passphrase that Tramp-Imap will use.  TODO: allow function calls, etc. here"
    :type 'string
!   :group 'tramp)
  
  (defun tramp-imap-file-name-p (filename)
    "Check if it's a filename for IMAP protocol."
***************
*** 410,416 ****
         (save-match-data
  	 (let ((entries
  		(with-file-property v localname "file-entries"
! 		  (tramp-imap-get-file-entries v localname))))
  	   (mapcar
  	    (lambda (x)
  	      (list
--- 403,409 ----
         (save-match-data
  	 (let ((entries
  		(with-file-property v localname "file-entries"
! 		  (tramp-imap-get-file-entries v))))
  	   (mapcar
  	    (lambda (x)
  	      (list
***************
*** 421,431 ****
  
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:blockstar.com:/INBOX.test/fstab") t)
  
! (defun tramp-imap-get-file-entries (vec localname &optional exact)
    "Read entries returned by IMAP server. EXACT limits to exact matches.
  Result is a list of (LOCALNAME LINK COUNT UID GID ATIME MTIME CTIME
  SIZE MODE WEIRD INODE DEVICE)."
!   (tramp-debug-message vec "working on %s" localname)
    (with-current-buffer (tramp-imap-buffer vec)
      (let* ((mbox (tramp-imap-file-name-mailbox vec))
  	   (name (tramp-imap-file-name-name vec))
--- 414,424 ----
  
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:blockstar.com:/INBOX.test/fstab") t)
  
! (defun tramp-imap-get-file-entries (vec &optional exact)
    "Read entries returned by IMAP server. EXACT limits to exact matches.
  Result is a list of (LOCALNAME LINK COUNT UID GID ATIME MTIME CTIME
  SIZE MODE WEIRD INODE DEVICE)."
!   (tramp-debug-message vec "working on %s" (tramp-file-name-localname vec))
    (with-current-buffer (tramp-imap-buffer vec)
      (let* ((mbox (tramp-imap-file-name-mailbox vec))
  	   (name (tramp-imap-file-name-name vec))
***************
*** 516,522 ****
        (let ((point (point))
  	    size data)
  	(tramp-message v 4 "Fetching file %s..." filename)
! 	(let* ((found (tramp-imap-get-file-entries v filename))
  	       ;; inode is the message UID; we use the first message found
  	       (uid (nth 11 (nth 0 found))))
  	  (when found
--- 509,515 ----
        (let ((point (point))
  	    size data)
  	(tramp-message v 4 "Fetching file %s..." filename)
! 	(let* ((found (tramp-imap-get-file-entries v))
  	       ;; inode is the message UID; we use the first message found
  	       (uid (nth 11 (nth 0 found))))
  	  (when found
***************
*** 547,553 ****
    "Like `file-attributes' for Tramp files."
    (with-parsed-tramp-file-name (expand-file-name filename) nil
      (with-file-property v localname (format "file-attributes-%s" id-format)
!       (cdr (car (tramp-imap-get-file-entries v localname))))))
  
  (defun tramp-imap-handle-file-executable-p (filename)
    "Like `file-executable-p' for Tramp files.  False for IMAP."
--- 540,546 ----
    "Like `file-attributes' for Tramp files."
    (with-parsed-tramp-file-name (expand-file-name filename) nil
      (with-file-property v localname (format "file-attributes-%s" id-format)
!       (cdr (car (tramp-imap-get-file-entries v))))))
  
  (defun tramp-imap-handle-file-executable-p (filename)
    "Like `file-executable-p' for Tramp files.  False for IMAP."
***************
*** 651,664 ****
  ;;; (with-current-buffer (tramp-imap-buffer (tramp-dissect-file-name "/imap:blockstar.com:/INBOX.test/new")) (tramp-imap-get-message-subject 24))
  (defun tramp-imap-get-message-subject (msgs)
  "Get message subject over IMAP."
!   (let ((allsubject 
! 	 (nnheader-ms-strip-cr 
! 	  (nth 2 (nth 0 (imap-fetch 
  			 msgs
! 			 "BODY[HEADER.FIELDS (SUBJECT)]" 
  			 'BODYDETAIL))))))
!     (when (string-match 

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Tramp-devel mailing list
Tramp-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/tramp-devel

Re: file storage in IMAP (eventually for Tramp) working and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

On Sat, 25 Apr 2009 19:25:58 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> I made many other fixes so please use the attached file as a baseline if
>> you want to submit patches for any of the issues I listed before, or any
>> new issues you notice.  It's getting close to usable.

MA> I've started to play with it. First error:

MA> Debugger entered--Lisp error: (wrong-number-of-arguments #[nil "ŠdbˆÀÁÂÃ#…
MA> " nil t delete-char 1] 4 ("/usr/local/share/emacs/23.0.92/lisp/gnus/nnheader.elc" . 25599)] 1)
MA>   nnheader-ms-strip-cr("X-Tramp-Imap: *scratch*

This is fixed, thanks.  It was simplest to just copy those functions to
tramp-imap.el, considering it shouldn't depend on Gnus anyhow.

MA> You use by default 'ssl for connection. Can we expect this to be the
MA> standard? Or shall we let the user decide, for example by two methods
MA> "imap" and "imaps"?

That makes sense, yes.  Do you think you could add it?  For now I left
it nil, so imap.el will pick the best available stream.

MA> I've also applied some few patches, you might check. See appended.

I applied them all, thank you.

Current status: passphrase management works, but I should hook
auth-source into it.  Saving a file works, both to a new name and to an
existing name.  Opening a file works.  Deleting a file doesn't work yet.
I hope users can start testing this so I'm not just testing it against
my Courier IMAP server in GNU Emacs (XEmacs should work too, for
example).

I think it's OK to throw this into Tramp when file deletion is added and
all the extraneous comments are removed.  All the TODO items should also
be done.  By then we should have some test cases too.

Thanks for your help and sorry for the long delay...
Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 32232 bytes --]

Tramp-IMAP works and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Fri, 10 Jul 2009 16:34:49 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I think it's OK to throw this into Tramp when file deletion is added and
TZ> all the extraneous comments are removed.  All the TODO items should also
TZ> be done.  By then we should have some test cases too.

All right, attached is one more release before the ready-for-Tramp one:

- allow file deletion (all matching files are deleted)
- allow imap and imaps method, connecting over SSL when appropriate
- use auth-source for the symmetric passphrase
- much more complex logic for passphrases, allowing caching them
- give a small setup example
- remove my mail server and username from the test cases littered throughout

I need to do more cleanup, but at least this version is usable.  The
next release will be for wide release and will go into Tramp itself, so
PLEASE test this version and let me know if you find problems.

Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 27423 bytes --]

Re: Tramp-IMAP works and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I need to do more cleanup, but at least this version is usable.  The
> next release will be for wide release and will go into Tramp itself, so
> PLEASE test this version and let me know if you find problems.

Don't panic, it is on my todo list. As usual, this list is much too long.

I hope to find time for test over the weekend.

> Ted

Best regards, Michael.

Re: Tramp-IMAP works and needs testing

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 2802 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I need to do more cleanup, but at least this version is usable.  The
> next release will be for wide release and will go into Tramp itself, so
> PLEASE test this version and let me know if you find problems.

I've played a little bit around:

- Several functions in `tramp-imap-file-name-handler-alist' are not
  implemented yet.

- I doubt there will be implementation for `shell-command',
  `executable-find', and `process-file'. I've disabled them in
  `tramp-imap-file-name-handler-alist'.

- You have used `tramp-debug-message'. I recommend to use
  `tramp-message' instead, the verbose levels allow a better analysis of
  the debug buffer.

- `tramp-imap-handle-file-directory-p' must not return nil in all
  cases. When I apply (save-buffer), the imap folder is checked for
  being a directory. As workaround, I've changed
  `tramp-imap-handle-file-directory-p' to return `t', but it must be
  more precise.

- When I have opened "/imap:albinus@localhost:/INBOX.test/1", and try to
  save it under the name "/imap:albinus@localhost:/INBOX.test/2", I get

  Debugger entered--Lisp error: (none "bad name 1 or mailbox INBOX.test/2")
  signal(none ("bad name 1 or mailbox INBOX.test/2"))
  tramp-error(["imap" "albinus" "localhost" "/INBOX.test/2/1"] none "bad name %s or mailbox %s" "1" "INBOX.test/2")
  (if (and name (imap-mailbox-select mbox)) (let (sname) (tramp-debug-message vec "looking for '%s'" search-name) (dolist ... ... ...)) (tramp-error vec (quote none) "bad name %s or mailbox %s" name mbox))
  (let* ((mbox ...) (name ...) (truename ...) (search-name ...) res) (tramp-debug-message vec "selecting mbox %s" mbox) (if (and name ...) (let ... ... ...) (tramp-error vec ... "bad name %s or mailbox %s" name mbox)) res)
  (save-current-buffer (set-buffer (tramp-imap-buffer vec)) (let* (... ... ... ... res) (tramp-debug-message vec "selecting mbox %s" mbox) (if ... ... ...) res))
  (with-current-buffer (tramp-imap-buffer vec) (let* (... ... ... ... res) (tramp-debug-message vec "selecting mbox %s" mbox) (if ... ... ...) res))
  tramp-imap-get-file-entries(["imap" "albinus" "localhost" "/INBOX.test/2/1"] "/INBOX.test/2/1")

  [...]

- I've fixed a small error in `tramp-imap-handle-delete-file' and
  `tramp-imap-delete-files'.

- `tramp-retrieve-data', used in `tramp-imap-handle-file-local-copy',
  does not exist.

- In `tramp-imap-passphrase-callback-function', you use the non-existing
  variable `v'.

- The email address of stored messages looks like "a real one", one
  could try to send messages there. Better might be to use an invalid
  address as defined in RFC 2606, like "Tramp-IMAP
  <tramp-imap@tramp-imap.invalid>".

My current patch is appended. Now I must go; my kids want to see HP6.

> Ted

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 16444 bytes --]

*** /home/albinus/lisp/tramp-imap.el.~1~	2009-07-18 10:45:43.000000000 +0200
--- /home/albinus/lisp/tramp-imap.el	2009-07-18 12:58:21.000000000 +0200
***************
*** 60,65 ****
--- 60,66 ----
  (defcustom tramp-imap-method "imap"
    "*Method to connect via IMAP protocol."
    :group 'tramp
+   :version "23.2"
    :type 'string)
  
  (add-to-list 'tramp-methods (cons tramp-imap-method nil))
***************
*** 72,77 ****
--- 73,79 ----
  (defcustom tramp-imaps-method "imaps"
    "*Method to connect via secure IMAP protocol."
    :group 'tramp
+   :version "23.2"
    :type 'string)
  
  ;; ... and add it to the method list.
***************
*** 84,89 ****
--- 86,92 ----
  ;; Add completion function for IMAP method.
  ;; (tramp-set-completion-function
  ;;  tramp-imap-method tramp-completion-function-alist-ssh) ; TODO: test this
+ ;;  tramp-imaps-method tramp-completion-function-alist-ssh) ; TODO: test this
  
  ;; New handlers should be added here.
  (defconst tramp-imap-file-name-handler-alist
***************
*** 136,161 ****
      (set-file-modes . tramp-imap-handle-set-file-modes)
      (set-file-times . tramp-imap-handle-set-file-times)
      (set-visited-file-modtime . ignore)
!     (shell-command . tramp-handle-shell-command)
      (substitute-in-file-name . tramp-handle-substitute-in-file-name)
      (unhandled-file-name-directory . tramp-handle-unhandled-file-name-directory)
      (vc-registered . ignore)
      (verify-visited-file-modtime . ignore)
      (write-region . tramp-imap-handle-write-region)
!     (executable-find . tramp-imap-handle-executable-find)
      (start-file-process . ignore)
!     (process-file . tramp-imap-handle-process-file)
  )
    "Alist of handler functions for Tramp IMAP method.
  Operations not mentioned here will be handled by the default Emacs primitives.")
  
  (defgroup tramp-imap nil
    "Tramp over IMAP configuration."
    :group 'applications)
  
  (defcustom tramp-imap-subject-marker "tramp-imap-subject-marker"
    "The subject marker that Tramp-IMAP will use."
    :type 'string
    :group 'tramp-imap)
  
  ;; TODO: these will be defcustoms later
--- 139,166 ----
      (set-file-modes . tramp-imap-handle-set-file-modes)
      (set-file-times . tramp-imap-handle-set-file-times)
      (set-visited-file-modtime . ignore)
!     (shell-command . ignore)
      (substitute-in-file-name . tramp-handle-substitute-in-file-name)
      (unhandled-file-name-directory . tramp-handle-unhandled-file-name-directory)
      (vc-registered . ignore)
      (verify-visited-file-modtime . ignore)
      (write-region . tramp-imap-handle-write-region)
!     (executable-find . ignore)
      (start-file-process . ignore)
!     (process-file . ignore)
  )
    "Alist of handler functions for Tramp IMAP method.
  Operations not mentioned here will be handled by the default Emacs primitives.")
  
  (defgroup tramp-imap nil
    "Tramp over IMAP configuration."
+   :version "23.2"
    :group 'applications)
  
  (defcustom tramp-imap-subject-marker "tramp-imap-subject-marker"
    "The subject marker that Tramp-IMAP will use."
    :type 'string
+   :version "23.2"
    :group 'tramp-imap)
  
  ;; TODO: these will be defcustoms later
***************
*** 253,265 ****
  	    (tramp-debug-message vec "looking for '%s'" search-name)
  	    (dolist (msg (imap-search
  			  (format "SUBJECT \"%s %s\""
! 				  tramp-imap-subject-marker 
  				  search-name)))
  
  	      ;; this is the name of the file
  	      (setq sname (tramp-imap-get-message-x-tramp-imap msg))
! 	      
! 	      (when (and 
  		     sname		; we need a valid X-Tramp-IMAP header
  		     (or (not exact) (equal sname truename)))
  		;; Return entry in file-attributes format
--- 258,270 ----
  	    (tramp-debug-message vec "looking for '%s'" search-name)
  	    (dolist (msg (imap-search
  			  (format "SUBJECT \"%s %s\""
! 				  tramp-imap-subject-marker
  				  search-name)))
  
  	      ;; this is the name of the file
  	      (setq sname (tramp-imap-get-message-x-tramp-imap msg))
! 
! 	      (when (and
  		     sname		; we need a valid X-Tramp-IMAP header
  		     (or (not exact) (equal sname truename)))
  		;; Return entry in file-attributes format
***************
*** 356,362 ****
  
  (defun tramp-imap-handle-file-directory-p (filename)
    "Like `file-directory-p' for Tramp files.  False for IMAP."
!   nil)
  
  (defun tramp-imap-handle-file-attributes (filename &optional id-format)
    "Like `file-attributes' for Tramp files."
--- 361,367 ----
  
  (defun tramp-imap-handle-file-directory-p (filename)
    "Like `file-directory-p' for Tramp files.  False for IMAP."
!   t);nil)
  
  (defun tramp-imap-handle-file-attributes (filename &optional id-format)
    "Like `file-attributes' for Tramp files."
***************
*** 380,388 ****
    "Like `delete-file' for Tramp files."
    (cond
     ((not (file-exists-p file)) nil)
!    (t (tramp-imap-delete-files
!        ;; inode is the message UID; we use any messages found
!        (tramp-imap-get-file-entries v filename)))))
  
  ;; TODO: fix this in tramp-imap-get-file-entries
  (defun tramp-imap-handle-file-newer-than-file-p (file1 file2)
--- 385,395 ----
    "Like `delete-file' for Tramp files."
    (cond
     ((not (file-exists-p file)) nil)
!    (t (with-parsed-tramp-file-name (expand-file-name file) nil
! 	(with-current-buffer (tramp-imap-buffer v)
! 	  (tramp-imap-delete-files
! 	   ;; inode is the message UID; we use any messages found
! 	   (tramp-imap-get-file-entries v file)))))))
  
  ;; TODO: fix this in tramp-imap-get-file-entries
  (defun tramp-imap-handle-file-newer-than-file-p (file1 file2)
***************
*** 402,407 ****
--- 409,415 ----
         "Cannot make local copy of non-existing file `%s'" filename))
      (let ((tmpfile (tramp-compat-make-temp-file filename)))
        (tramp-message v 4 "Fetching %s to tmp file %s..." filename tmpfile)
+       ;; TODO: tramp-retrieve-data does not exist!!!
        (when (tramp-imap-retrieve-data v)
  	;; Save file
  	(with-current-buffer (tramp-get-buffer v)
***************
*** 437,460 ****
  (defun tramp-imap-file-name-truename (vec)
    (tramp-imap-file-name-mailbox-or-name vec nil))
  
! (defun tramp-imap-buffer (v)
!   (if (imap-opened (tramp-get-buffer v))
!       (tramp-get-buffer v)
!     (let* ((server (tramp-file-name-real-host v))
! 	   (port (tramp-file-name-port v))
  	   (auth-info
  	    (auth-source-user-or-password '("login" "password") server port))
  	   (auth-user (nth 0 auth-info))
  	   (auth-passwd (nth 1 auth-info))
! 	   (buffer (imap-open 
! 		    server 
! 		    port 
  		    ;; this is the only place where IMAP vs IMAPS matters
! 		    (if (string= (tramp-file-name-method v) tramp-imap-method)
  			nil
  		      'ssl)
  		    nil
! 		    (tramp-get-buffer v))))
        (imap-authenticate auth-user auth-passwd buffer)
        buffer)))
  
--- 445,468 ----
  (defun tramp-imap-file-name-truename (vec)
    (tramp-imap-file-name-mailbox-or-name vec nil))
  
! (defun tramp-imap-buffer (vec)
!   (if (imap-opened (tramp-get-buffer vec))
!       (tramp-get-buffer vec)
!     (let* ((server (tramp-file-name-real-host vec))
! 	   (port (tramp-file-name-port vec))
  	   (auth-info
  	    (auth-source-user-or-password '("login" "password") server port))
  	   (auth-user (nth 0 auth-info))
  	   (auth-passwd (nth 1 auth-info))
! 	   (buffer (imap-open
! 		    server
! 		    port
  		    ;; this is the only place where IMAP vs IMAPS matters
! 		    (if (string= (tramp-file-name-method vec) tramp-imap-method)
  			nil
  		      'ssl)
  		    nil
! 		    (tramp-get-buffer vec))))
        (imap-authenticate auth-user auth-passwd buffer)
        buffer)))
  
***************
*** 462,468 ****
  ;;; (with-current-buffer (tramp-imap-buffer (tramp-dissect-file-name "/imap:yourhosthere.com:/INBOX.test/new")) (tramp-imap-put-file "INBOX.test" "/etc/fstab" "new5"))
  
  (defun tramp-imap-put-file (mailbox filename-or-buffer &optional subject)
!   (imap-message-append 
     mailbox
     ;;TODO: use better buffer name
     (tramp-imap-message-buffer "*tramp-imap-encode*" filename-or-buffer subject)))
--- 470,476 ----
  ;;; (with-current-buffer (tramp-imap-buffer (tramp-dissect-file-name "/imap:yourhosthere.com:/INBOX.test/new")) (tramp-imap-put-file "INBOX.test" "/etc/fstab" "new5"))
  
  (defun tramp-imap-put-file (mailbox filename-or-buffer &optional subject)
!   (imap-message-append
     mailbox
     ;;TODO: use better buffer name
     (tramp-imap-message-buffer "*tramp-imap-encode*" filename-or-buffer subject)))
***************
*** 479,504 ****
        (tramp-imap-decode-buffer))))
  
  (defun tramp-imap-delete-files (entries)
!   (with-current-buffer (tramp-imap-buffer v)
!     (dolist (entry entries)
!       ;; TODO: this should be aware of IMAP large integers
!       (when (integerp (nth 11 entry))
! 	(imap-message-flags-add (format "%d" (nth 11 entry))
! 				"\\Seen \\Deleted")))
!     (imap-mailbox-expunge)))
  
  ;; TODO: make it use tramp-imap-get-message-details, so it's compatible with non-IMAP4rev1 IMAP servers
  ;;; (with-current-buffer (tramp-imap-buffer (tramp-dissect-file-name "/imap:yourhosthere.com:/INBOX.test/new")) (tramp-imap-get-message-subject 24))
  (defun tramp-imap-get-message-subject (msgs)
  "Get message subject over IMAP."
!   (let ((allsubject 
! 	 (tramp-imap-ms-strip-cr 
! 	  (nth 2 (nth 0 (imap-fetch 
  			 msgs
! 			 "BODY[HEADER.FIELDS (SUBJECT)]" 
  			 'BODYDETAIL))))))
!     (when (string-match 
! 	   (format "Subject: %s \\(.+\\)" tramp-imap-subject-marker) 
  	   allsubject)
        (match-string 1 allsubject))))
  
--- 487,511 ----
        (tramp-imap-decode-buffer))))
  
  (defun tramp-imap-delete-files (entries)
!   (dolist (entry entries)
!     ;; TODO: this should be aware of IMAP large integers
!     (when (integerp (nth 11 entry))
!       (imap-message-flags-add (format "%d" (nth 11 entry))
! 			      "\\Seen \\Deleted")))
!   (imap-mailbox-expunge))
  
  ;; TODO: make it use tramp-imap-get-message-details, so it's compatible with non-IMAP4rev1 IMAP servers
  ;;; (with-current-buffer (tramp-imap-buffer (tramp-dissect-file-name "/imap:yourhosthere.com:/INBOX.test/new")) (tramp-imap-get-message-subject 24))
  (defun tramp-imap-get-message-subject (msgs)
  "Get message subject over IMAP."
!   (let ((allsubject
! 	 (tramp-imap-ms-strip-cr
! 	  (nth 2 (nth 0 (imap-fetch
  			 msgs
! 			 "BODY[HEADER.FIELDS (SUBJECT)]"
  			 'BODYDETAIL))))))
!     (when (string-match
! 	   (format "Subject: %s \\(.+\\)" tramp-imap-subject-marker)
  	   allsubject)
        (match-string 1 allsubject))))
  
***************
*** 514,520 ****
  (defun tramp-imap-get-message-headers (msgs)
    "Get message headers over IMAP."
    ;; nil means "all headers"
!   (tramp-imap-ms-strip-cr 
     (nth 2 (nth 0 (tramp-imap-get-message-details msgs nil)))))
  
  ;; from nnheader.el
--- 521,527 ----
  (defun tramp-imap-get-message-headers (msgs)
    "Get message headers over IMAP."
    ;; nil means "all headers"
!   (tramp-imap-ms-strip-cr
     (nth 2 (nth 0 (tramp-imap-get-message-details msgs nil)))))
  
  ;; from nnheader.el
***************
*** 536,542 ****
  
  (defun tramp-imap-get-message-body (msgs)
    "Get message body over IMAP."
!   (tramp-imap-ms-strip-cr 
     (nth 2 (nth 0 (tramp-imap-get-message-details msgs 'body)))))
  
  (defun tramp-imap-get-message-details (msgs body-or-header)
--- 543,549 ----
  
  (defun tramp-imap-get-message-body (msgs)
    "Get message body over IMAP."
!   (tramp-imap-ms-strip-cr
     (nth 2 (nth 0 (tramp-imap-get-message-details msgs 'body)))))
  
  (defun tramp-imap-get-message-details (msgs body-or-header)
***************
*** 549,563 ****
  	 (if body "BODY.PEEK[TEXT]" "BODY.PEEK[HEADER]")
         (if body "RFC822.TEXT.PEEK" "RFC822.HEADER"))
       ;; the RECEIVE parameter
!      (if i4r1 
  	 'BODYDETAIL
         (if body 'RFC822.TEXT 'RFC822.HEADER)))))
  
! ;;; (tramp-imap-collapse-name "a b c / where ; strange ! characters $ abound") 
  ;;; => "abcwherestrangecharactersabound"
  (defun tramp-imap-collapse-name (name)
    "Return NAME with only [A-Za-z0-9] characters"
!   (when name 
      (replace-regexp-in-string "[^A-Za-z0-9]" "" name)))
  
  ;;; (tramp-imap-message-buffer "testimap" "/etc/fstab" "fstab")
--- 556,570 ----
  	 (if body "BODY.PEEK[TEXT]" "BODY.PEEK[HEADER]")
         (if body "RFC822.TEXT.PEEK" "RFC822.HEADER"))
       ;; the RECEIVE parameter
!      (if i4r1
  	 'BODYDETAIL
         (if body 'RFC822.TEXT 'RFC822.HEADER)))))
  
! ;;; (tramp-imap-collapse-name "a b c / where ; strange ! characters $ abound")
  ;;; => "abcwherestrangecharactersabound"
  (defun tramp-imap-collapse-name (name)
    "Return NAME with only [A-Za-z0-9] characters"
!   (when name
      (replace-regexp-in-string "[^A-Za-z0-9]" "" name)))
  
  ;;; (tramp-imap-message-buffer "testimap" "/etc/fstab" "fstab")
***************
*** 579,586 ****
        (message-setup
         `((To . "Tramp-IMAP <tramp-imap@tramp-imap.tramp.emacs.org>")
  	 (From . "Tramp-IMAP <tramp-imap@tramp-imap.tramp.emacs.org>")
! 	 (Subject . ,(format 
! 		      "%s %s" 
  		      tramp-imap-subject-marker
  		      (or subject (tramp-imap-collapse-name sname))))
  	 ;; TODO: make sure this can handle non-ASCII data
--- 586,593 ----
        (message-setup
         `((To . "Tramp-IMAP <tramp-imap@tramp-imap.tramp.emacs.org>")
  	 (From . "Tramp-IMAP <tramp-imap@tramp-imap.tramp.emacs.org>")
! 	 (Subject . ,(format
! 		      "%s %s"
  		      tramp-imap-subject-marker
  		      (or subject (tramp-imap-collapse-name sname))))
  	 ;; TODO: make sure this can handle non-ASCII data
***************
*** 601,606 ****
--- 608,614 ----
  CONTEXT is the encryption/decryption EPG context.
  HANDBACK is just carried through.
  KEY-ID can be 'SYM or 'PIN among others."
+   ;; TODO: Variable `v' is not declared!!! Existing due to side-effects only.
    (let* ((server (tramp-file-name-real-host v))
  	 (port "tramp-imap")		; this is NOT the server password!
  	 (auth-passwd
***************
*** 613,625 ****
  	 ;; do we reuse it?
  	 (if (y-or-n-p "Reuse the passphrase? ")
  	     (copy-sequence tramp-imap-passphrase)
! 	   ;; don't reuse: revert caching behavior to nil, erase passphrase, 
  	   ;; call ourselves again
  	   (setq tramp-imap-passphrase-cache nil)
  	   (setq tramp-imap-passphrase nil)
  	   (tramp-imap-passphrase-callback-function context key-id handback))
         (let ((p (if (eq key-id 'SYM)
! 		    (read-passwd 
  		     "Tramp-IMAP passphrase for symmetric encryption: "
  		     (eq (epg-context-operation context) 'encrypt)
  		     tramp-imap-passphrase)
--- 621,633 ----
  	 ;; do we reuse it?
  	 (if (y-or-n-p "Reuse the passphrase? ")
  	     (copy-sequence tramp-imap-passphrase)
! 	   ;; don't reuse: revert caching behavior to nil, erase passphrase,
  	   ;; call ourselves again
  	   (setq tramp-imap-passphrase-cache nil)
  	   (setq tramp-imap-passphrase nil)
  	   (tramp-imap-passphrase-callback-function context key-id handback))
         (let ((p (if (eq key-id 'SYM)
! 		    (read-passwd
  		     "Tramp-IMAP passphrase for symmetric encryption: "
  		     (eq (epg-context-operation context) 'encrypt)
  		     tramp-imap-passphrase)
***************
*** 628,643 ****
  		       "Tramp-IMAP passphrase for PIN: "
  		     (let ((entry (assoc key-id epg-user-id-alist)))
  		       (if entry
! 			   (format "Tramp-IMAP passphrase for %s %s: " 
  				   key-id (cdr entry))
  			 (format "Tramp-IMAP passphrase for %s: " key-id))))
  		   nil
  		   tramp-imap-passphrase))))
  
! 	 ;; if we have an answer, the passphrase has changed, 
! 	 ;; the user hasn't declined keeping the passphrase, 
  	 ;; and they answer yes to keep it now...
! 	 (when (and 
  		p
  		(not (equal tramp-imap-passphrase p))
  		(not (eq tramp-imap-passphrase-cache 'never))
--- 636,651 ----
  		       "Tramp-IMAP passphrase for PIN: "
  		     (let ((entry (assoc key-id epg-user-id-alist)))
  		       (if entry
! 			   (format "Tramp-IMAP passphrase for %s %s: "
  				   key-id (cdr entry))
  			 (format "Tramp-IMAP passphrase for %s: " key-id))))
  		   nil
  		   tramp-imap-passphrase))))
  
! 	 ;; if we have an answer, the passphrase has changed,
! 	 ;; the user hasn't declined keeping the passphrase,
  	 ;; and they answer yes to keep it now...
! 	 (when (and
  		p
  		(not (equal tramp-imap-passphrase p))
  		(not (eq tramp-imap-passphrase-cache 'never))

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Tramp-devel mailing list
Tramp-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/tramp-devel

Re: Tramp-IMAP works and needs testing

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 4426 bytes --]

On Sat, 18 Jul 2009 13:11:51 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> - Several functions in `tramp-imap-file-name-handler-alist' are not
MA>   implemented yet.

I'm trying to get basic file read/write working before those other
functions.  Should I just set them to nil for now?

Sorry for the delay, also.  I've been busy and the weekends are too short...

MA> - I doubt there will be implementation for `shell-command',
MA> `executable-find', and `process-file'. I've disabled them in
MA> `tramp-imap-file-name-handler-alist'.

OK.

MA> - You have used `tramp-debug-message'. I recommend to use
MA>   `tramp-message' instead, the verbose levels allow a better analysis of
MA>   the debug buffer.

OK.

MA> - `tramp-imap-handle-file-directory-p' must not return nil in all
MA>   cases. When I apply (save-buffer), the imap folder is checked for
MA>   being a directory. As workaround, I've changed
MA>   `tramp-imap-handle-file-directory-p' to return `t', but it must be
MA>   more precise.

This is actually tough, because an entry could be both.  We could have
the files INBOX.test/b and INBOX.test/b/c at the same time.  I will have
to make the directory name the mailbox name and everything above that
the filename.  Thus no subdirectories are allowed, which simplifies the
logic.  Otherwise I don't think Tramp can handle it.  So:

/imap:albinus@localhost:/INBOX.test = directory
/imap:albinus@localhost:/INBOX.test/ = directory
/imap:albinus@localhost:/INBOX.test/1 = file "1"
/imap:albinus@localhost:/INBOX.test/1/2 = illegal file name

I think that's sufficient for most users.  I hate to kill functionality
this way, but otherwise things will get too confusing.  Does that sound
OK to you?

MA> - When I have opened "/imap:albinus@localhost:/INBOX.test/1", and try to
MA>   save it under the name "/imap:albinus@localhost:/INBOX.test/2", I get

MA>   Debugger entered--Lisp error: (none "bad name 1 or mailbox INBOX.test/2")
MA>   signal(none ("bad name 1 or mailbox INBOX.test/2"))
MA>   tramp-error(["imap" "albinus" "localhost" "/INBOX.test/2/1"] none "bad name %s or mailbox %s" "1" "INBOX.test/2")
MA>   (if (and name (imap-mailbox-select mbox)) (let (sname) (tramp-debug-message vec "looking for '%s'" search-name) (dolist ... ... ...)) (tramp-error vec (quote none) "bad name %s or mailbox %s" name mbox))
MA>   (let* ((mbox ...) (name ...) (truename ...) (search-name ...) res) (tramp-debug-message vec "selecting mbox %s" mbox) (if (and name ...) (let ... ... ...) (tramp-error vec ... "bad name %s or mailbox %s" name mbox)) res)
MA>   (save-current-buffer (set-buffer (tramp-imap-buffer vec)) (let* (... ... ... ... res) (tramp-debug-message vec "selecting mbox %s" mbox) (if ... ... ...) res))
MA>   (with-current-buffer (tramp-imap-buffer vec) (let* (... ... ... ... res) (tramp-debug-message vec "selecting mbox %s" mbox) (if ... ... ...) res))
MA>   tramp-imap-get-file-entries(["imap" "albinus" "localhost" "/INBOX.test/2/1"] "/INBOX.test/2/1")

MA>   [...]

OK.  I think the code is a mess, I'll improve it as stated above and below.

MA> - I've fixed a small error in `tramp-imap-handle-delete-file' and
MA>   `tramp-imap-delete-files'.

MA> - `tramp-retrieve-data', used in `tramp-imap-handle-file-local-copy',
MA>   does not exist.

MA> - In `tramp-imap-passphrase-callback-function', you use the non-existing
MA>   variable `v'.

Thanks for these.  For tramp-imap-retrieve-data I was looking at the
tramp-fish.el code, which has that function.  I think in general my code
needs to be redone, it's too messy.  I was just playing around when I
put it together; the core functionality is OK but it's a mess of
spaghetti code that I would hate to support.  I'll try to rewrite it
before the next Tramp release.  I'm attaching the last spaghetti
version, I hope.

In the passphrase callback, I don't have access to the Tramp context.  I
only have the EPG context.  

Daiki Ueno: is it OK if I insert the Tramp context (basically a cons) at
the end of the EPG context?  It will make the EPG context longer by 1
element so I worry it may break things in EPG.  If not, how can I pass
the Tramp context to the passphrase callback?

MA> - The email address of stored messages looks like "a real one", one
MA>   could try to send messages there. Better might be to use an invalid
MA>   address as defined in RFC 2606, like "Tramp-IMAP
MA>   <tramp-imap@tramp-imap.invalid>".

OK.

Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 27603 bytes --]

Re: Tramp-IMAP works and needs testing

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 3152 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sat, 18 Jul 2009 13:11:51 +0200 Michael Albinus <michael.albinus@gmx.de> wrote:
>
> MA> - Several functions in `tramp-imap-file-name-handler-alist' are not
> MA>   implemented yet.
>
> I'm trying to get basic file read/write working before those other
> functions.  Should I just set them to nil for now?

We could set them to `ignore', and add a comment. I've done it for all
functions with a missing implementation.

> MA> - `tramp-imap-handle-file-directory-p' must not return nil in all
> MA>   cases. When I apply (save-buffer), the imap folder is checked for
> MA>   being a directory. As workaround, I've changed
> MA>   `tramp-imap-handle-file-directory-p' to return `t', but it must be
> MA>   more precise.
>
> This is actually tough, because an entry could be both.  We could have
> the files INBOX.test/b and INBOX.test/b/c at the same time.  I will have
> to make the directory name the mailbox name and everything above that
> the filename.  Thus no subdirectories are allowed, which simplifies the
> logic.  Otherwise I don't think Tramp can handle it.  So:
>
> /imap:albinus@localhost:/INBOX.test = directory
> /imap:albinus@localhost:/INBOX.test/ = directory
> /imap:albinus@localhost:/INBOX.test/1 = file "1"
> /imap:albinus@localhost:/INBOX.test/1/2 = illegal file name
>
> I think that's sufficient for most users.  I hate to kill functionality
> this way, but otherwise things will get too confusing.  Does that sound
> OK to you?

OK. I've added a corresponding implementation.

Problems:

`tramp-imap-buffer' can return nil, if the connection to IMAP fails. You
always use (with-current-buffer (tramp-imap-buffer vec) ...), which
gives an error then. It might be better to check the result of
`tramp-imap-buffer' first.

`tramp-imap-handle-name-all-completions' does not complete mailbox names.

If you apply `write-file', and the file exists already on the IMAP
server, a second file with the same name (subject) will be written
instead of overwriting the existing one.

In `tramp-imap-buffer', you use auth-source in order to determine user
and password. I believe, it shall be rather done via `tramp-read-passwd'
(which includes auth-source). By this, the user name given in the Tramp
filename is respected. Furthermore, I plan to add more password handling
mechanisms there, for example the Secret Service API for gnome-keyring
and kwallet.

Furthermore, I've changed the following:

- Change the copyright owner to FSF. I hope you are OK with this;
  otherwise we have no chance to add your package to Tramp and Emacs.

- Make `tramp-imap-method' and `tramp-imaps-method' a defconst. I know
  that in some other tramp-*.el files the methods are also defcustom's,
  but I don't believe it is necessary (and I shall change it everywhere).

- Change implementation of `tramp-imap-handle-file-readable-p ' and
  `tramp-imap-handle-file-writable-p'.

After all, it is working already pretty fine. I believe we shall add it
to Tramp's CVS repository, and likely we shall add it also to the
upcoming Tramp 2.1.17 release (maybe marked as experimental).

> Ted

Best regards, Michael.


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 27836 bytes --]

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Tramp-devel mailing list
Tramp-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/tramp-devel

Re: Tramp-IMAP works and needs testing

[Ted Zlatanov]

On Tue, 08 Sep 2009 15:33:11 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 
...
MA> Furthermore, I've changed the following:

MA> - Change the copyright owner to FSF. I hope you are OK with this;
MA>   otherwise we have no chance to add your package to Tramp and Emacs.

MA> - Make `tramp-imap-method' and `tramp-imaps-method' a defconst. I know
MA>   that in some other tramp-*.el files the methods are also defcustom's,
MA>   but I don't believe it is necessary (and I shall change it everywhere).

MA> - Change implementation of `tramp-imap-handle-file-readable-p ' and
MA>   `tramp-imap-handle-file-writable-p'.

MA> After all, it is working already pretty fine. I believe we shall add it
MA> to Tramp's CVS repository, and likely we shall add it also to the
MA> upcoming Tramp 2.1.17 release (maybe marked as experimental).

Thanks for all your help.  I am working on a storage layer
(imap-hash.el) that will let me treat a mailbox as a hash table, making
retrieval and storage much easier.  That way I can make tramp-imap.el
correspondingly simpler.  

I will incorporate all your changes as well; my changes are at the
storage layer and don't interfere with the Tramp functionality.  I am
working hard on this so I expect imap-hash.el to be ready soon.
Afterwards I can merge your changes and overhaul the tramp-imap.el code.
Can you hold on before adding the old tramp-imap.el to Tramp?  I don't
want my bad code harming other programmers' minds ;)  I think I can
make the 2.1.17 release; I'll follow up with a progress report in a week
if I'm not done by then.

imap-hash.el is relevant to the Gnus project as well.  I plan to use it
to store things like the Gnus registry and maybe the Gnus newsrc file,
essentially providing a distributed hashtable in an IMAP mailbox.  This
was another motive for developing a separate storage layer.

Thanks
Ted

Re: Tramp-IMAP works and needs testing

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> MA> After all, it is working already pretty fine. I believe we shall add it
> MA> to Tramp's CVS repository, and likely we shall add it also to the
> MA> upcoming Tramp 2.1.17 release (maybe marked as experimental).
>
> Can you hold on before adding the old tramp-imap.el to Tramp?  I don't
> want my bad code harming other programmers' minds ;)  I think I can
> make the 2.1.17 release; I'll follow up with a progress report in a week
> if I'm not done by then.

That's easy. Starting on Saturday, I'll be almost offline for a week.

> Thanks
> Ted

Best regards, Michael.

imap-hash.el and tramp-imap.el

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

Attached are imap-hash.el and tramp-imap.el, I worked on them this
weekend.

imap-hash.el lets you treat an IMAP mailbox as a hash.  It lets you get
individual keys' headers and body data (keys are message UIDs) or map a
function across *all the messages in the mailbox*.  The function is only
called for those that match a given subject, but really I should be
using SEARCH.  So the library is not optimized for speed.  Also, it does
not handle invalid mailbox names gracefully yet.  Consider it an alpha
version.

I plan to eventually use imap-hash.el to store distributed Gnus data,
specifically the Gnus registry, the newsrc file, scoring files,
etc.

tramp-imap.el uses imap-hash.el to provide a Tramp interface to read and
write files as messages in an IMAP mailbox.  It's also an alpha version,
but it will at least let you (given a valid IMAP mailbox) read and write
directory contents.  It uses EPG (thanks to Daiki Ueno's help) to
encrypt the file contents when they are stored as messages.

Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 25494 bytes --]

[-- Attachment #3: imap-hash.el --]
[-- Type: application/emacs-lisp, Size: 14078 bytes --]

Re: imap-hash.el and tramp-imap.el

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> Attached are imap-hash.el and tramp-imap.el, I worked on them this
> weekend.

Thanks for this. I've played a little bit with them (no real test yet).

> imap-hash.el lets you treat an IMAP mailbox as a hash.  It lets you get
> individual keys' headers and body data (keys are message UIDs) or map a
> function across *all the messages in the mailbox*.  The function is only
> called for those that match a given subject, but really I should be
> using SEARCH.  So the library is not optimized for speed.  Also, it does
> not handle invalid mailbox names gracefully yet.  Consider it an alpha
> version.

Could you, please, add "(require 'assoc)"? It's not loaded by default.

> tramp-imap.el uses imap-hash.el to provide a Tramp interface to read and
> write files as messages in an IMAP mailbox.  It's also an alpha version,
> but it will at least let you (given a valid IMAP mailbox) read and write
> directory contents.  It uses EPG (thanks to Daiki Ueno's help) to
> encrypt the file contents when they are stored as messages.

As usual, I have some few patches :-)

* imaps didn't work for me any longer. I've patched
  `tramp-imap-make-iht' until it worked ... maybe my sledge-hammer patch
  could be done more elegant. There is still an error message in the
  mini-buffer when opening the IMAP folder, but it doesn't seem to hurt.

* I have renamed `tramp-imap-handle-file-inode' to
  `tramp-imap-get-file-inode'. tramp-*-handle-* function names shall be
  reserved to basic file name operations implementation.

* I have used `tramp-current-host' to solve the problem of unresolved
  vector in `tramp-imap-passphrase-callback-function'.  That variable
  was introduced in tramp.el for exactly this kind of problems.

* When you byte-compile tramp-imap.el, there is a warning about unknown
  function `tramp-imap-get-message-headers'. No idea, what to do here.

My patch is appended.

I really would like to add tramp-imap.el to the CVS repository. I test
on different machines, and the current situation is inconvenient for
me. Do you agree for a checkin?

> Ted

Best regards, Michael.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 11446 bytes --]

*** /tmp/ediff21621-xQ	2009-09-22 16:49:14.000000000 +0200
--- /tmp/ediff21621L8W	2009-09-22 16:49:14.000000000 +0200
***************
*** 260,266 ****
  			"-rw-rw-rw-"
  			nil
  			uid
! 			1))) 
  		   iht t)))
  
  (defun tramp-imap-handle-write-region (start end filename &optional append visit lockname confirm)
--- 260,266 ----
  			"-rw-rw-rw-"
  			nil
  			uid
! 			1)))
  		   iht t)))
  
  (defun tramp-imap-handle-write-region (start end filename &optional append visit lockname confirm)
***************
*** 275,281 ****
  	(tramp-error v 'file-error "File not overwritten")))
      (tramp-flush-file-property v localname)
      (let* ((old-buffer (current-buffer))
! 	   (inode (tramp-imap-handle-file-inode filename))
  	   (min 1)
  	   (max (point-max))
  	   ;; make sure we have good start and end values
--- 275,281 ----
  	(tramp-error v 'file-error "File not overwritten")))
      (tramp-flush-file-property v localname)
      (let* ((old-buffer (current-buffer))
! 	   (inode (tramp-imap-get-file-inode filename))
  	   (min 1)
  	   (max (point-max))
  	   ;; make sure we have good start and end values
***************
*** 290,298 ****
  			     (with-current-buffer old-buffer
  			       (buffer-substring-no-properties start end)))
  			    (current-buffer)))
! 	(tramp-imap-put-file v 
! 			     temp-buffer 
! 			     (tramp-imap-file-name-name v) 
  			     inode)))
      (when (eq visit t)
        (set-visited-file-modtime))))
--- 290,298 ----
  			     (with-current-buffer old-buffer
  			       (buffer-substring-no-properties start end)))
  			    (current-buffer)))
! 	(tramp-imap-put-file v
! 			     temp-buffer
! 			     (tramp-imap-file-name-name v)
  			     inode)))
      (when (eq visit t)
        (set-visited-file-modtime))))
***************
*** 307,313 ****
    (with-parsed-tramp-file-name filename nil
      (if (not (file-exists-p filename))
  	(tramp-error
! 	 v 'file-error "File '%s' not found on remote host" filename)
        (let ((point (point))
  	    size data)
  	(tramp-message v 4 "Fetching file %s..." filename)
--- 307,313 ----
    (with-parsed-tramp-file-name filename nil
      (if (not (file-exists-p filename))
  	(tramp-error
! 	 v 'file-error "File `%s' not found on remote host" filename)
        (let ((point (point))
  	    size data)
  	(tramp-message v 4 "Fetching file %s..." filename)
***************
*** 339,346 ****
      ;;   (cdr (nth 0 (tramp-imap-get-file-entries v localname))))
      (cdr-safe (nth 0 (tramp-imap-get-file-entries v localname)))))
  
! (defun tramp-imap-handle-file-inode (filename &optional id-format)
!   "Get inode equivalent (actually the UID) for Tramp-IMAP FILENAME."
    (nth 10 (file-attributes filename)))
  
  (defun tramp-imap-handle-file-executable-p (filename)
--- 339,346 ----
      ;;   (cdr (nth 0 (tramp-imap-get-file-entries v localname))))
      (cdr-safe (nth 0 (tramp-imap-get-file-entries v localname)))))
  
! (defun tramp-imap-get-file-inode (filename &optional id-format)
!   "Get inode equivalent \(actually the UID) for Tramp-IMAP FILENAME."
    (nth 10 (file-attributes filename)))
  
  (defun tramp-imap-handle-file-executable-p (filename)
***************
*** 361,367 ****
     ((not (file-exists-p file)) nil)
     (t (with-parsed-tramp-file-name (expand-file-name file) nil
  	(let ((iht (tramp-imap-make-iht v)))
! 	  (imap-hash-rem (tramp-imap-handle-file-inode filename) iht))))))
  
  ;; TODO: fix this in tramp-imap-get-file-entries
  (defun tramp-imap-handle-file-newer-than-file-p (file1 file2)
--- 361,367 ----
     ((not (file-exists-p file)) nil)
     (t (with-parsed-tramp-file-name (expand-file-name file) nil
  	(let ((iht (tramp-imap-make-iht v)))
! 	  (imap-hash-rem (tramp-imap-get-file-inode filename) iht))))))
  
  ;; TODO: fix this in tramp-imap-get-file-entries
  (defun tramp-imap-handle-file-newer-than-file-p (file1 file2)
***************
*** 387,402 ****
  	tmpfile))))
  
  (defun tramp-imap-put-file (vec filename-or-buffer &optional subject inode)
! "Write contents of FILENAME-OR-BUFFER to Tramp-IMAP file VEC with name SUBJECT.
  When INODE is given, delete that old remote file after writing the new one
! (normally this is the old file with the same name)."
!   (let ((iht (tramp-imap-make-iht vec)))
      (imap-hash-put (list
  		    (list (cons
  			   'Subject
! 			   (format 
! 			    "%s%s" 
! 			    tramp-imap-subject-marker 
  			    (or subject "no subject"))))
  		    (cond ((bufferp filename-or-buffer)
  			   (with-current-buffer filename-or-buffer
--- 387,404 ----
  	tmpfile))))
  
  (defun tramp-imap-put-file (vec filename-or-buffer &optional subject inode)
!   "Write contents of FILENAME-OR-BUFFER to Tramp-IMAP file VEC with name SUBJECT.
  When INODE is given, delete that old remote file after writing the new one
! \(normally this is the old file with the same name)."
!   ;; `tramp-current-host' is used in `tramp-imap-passphrase-callback-function'.
!   (let ((tramp-current-host (tramp-file-name-real-host vec))
! 	(iht (tramp-imap-make-iht vec)))
      (imap-hash-put (list
  		    (list (cons
  			   'Subject
! 			   (format
! 			    "%s%s"
! 			    tramp-imap-subject-marker
  			    (or subject "no subject"))))
  		    (cond ((bufferp filename-or-buffer)
  			   (with-current-buffer filename-or-buffer
***************
*** 408,435 ****
  
  
  (defun tramp-imap-get-file (filename)
!   ;; (debug (tramp-imap-handle-file-inode filename))
    (with-parsed-tramp-file-name (expand-file-name filename) nil
      (condition-case ()
! 	(let ((iht (tramp-imap-make-iht v))
! 	      (inode (tramp-imap-handle-file-inode filename))
! 	      (data (imap-hash-get (tramp-imap-handle-file-inode 
! 				    filename)
! 				   iht t)))
  	  (with-temp-buffer
  	    (insert (nth 1 data))
  	    ;;(debug inode (buffer-string))
  	    (tramp-imap-decode-buffer)))
        (error (tramp-error
! 	      v 'file-error "File '%s' could not be read" filename)))))
  
  (defun tramp-imap-passphrase-callback-function (context key-id handback)
    "Called by EPG to get a passphrase for Tramp-IMAP.
  CONTEXT is the encryption/decryption EPG context.
  HANDBACK is just carried through.
  KEY-ID can be 'SYM or 'PIN among others."
!   ;; TODO: Variable `v' is not declared!!! Existing due to side-effects only.
!   (let* ((server (tramp-file-name-real-host v))
  	 (port "tramp-imap")		; this is NOT the server password!
  	 (auth-passwd
  	  (auth-source-user-or-password "password" server port)))
--- 410,437 ----
  
  
  (defun tramp-imap-get-file (filename)
!   ;; (debug (tramp-imap-get-file-inode filename))
    (with-parsed-tramp-file-name (expand-file-name filename) nil
      (condition-case ()
! 	;; `tramp-current-host' is used in
! 	;; `tramp-imap-passphrase-callback-function'.
! 	(let* ((tramp-current-host (tramp-file-name-real-host v))
! 	       (iht (tramp-imap-make-iht v))
! 	       (inode (tramp-imap-get-file-inode filename))
! 	       (data (imap-hash-get inode iht t)))
  	  (with-temp-buffer
  	    (insert (nth 1 data))
  	    ;;(debug inode (buffer-string))
  	    (tramp-imap-decode-buffer)))
        (error (tramp-error
! 	      v 'file-error "File `%s' could not be read" filename)))))
  
  (defun tramp-imap-passphrase-callback-function (context key-id handback)
    "Called by EPG to get a passphrase for Tramp-IMAP.
  CONTEXT is the encryption/decryption EPG context.
  HANDBACK is just carried through.
  KEY-ID can be 'SYM or 'PIN among others."
!   (let* ((server tramp-current-host)
  	 (port "tramp-imap")		; this is NOT the server password!
  	 (auth-passwd
  	  (auth-source-user-or-password "password" server port)))
***************
*** 532,545 ****
  (defun tramp-imap-make-iht (vec &optional needed-subject)
    "Translate the Tramp vector VEC to the imap-hash structure.
  With NEEDED-SUBJECT, alters the imap-hash test accordingly."
!   (let ((mbox (tramp-imap-file-name-mailbox vec))
! 	(server (tramp-file-name-real-host vec))
! 	(port (or (tramp-file-name-port vec) "imap")))
      ;; return the IHT with a test override to look for the subject marker
!     (plist-put (imap-hash-make server port mbox) 
! 	       :test (format "^%s%s" 
! 			     tramp-imap-subject-marker
! 			     (if needed-subject needed-subject "")))))
  
  (provide 'tramp-imap)
  ;;; tramp-imap.el ends here
--- 534,550 ----
  (defun tramp-imap-make-iht (vec &optional needed-subject)
    "Translate the Tramp vector VEC to the imap-hash structure.
  With NEEDED-SUBJECT, alters the imap-hash test accordingly."
!   (let* ((mbox (tramp-imap-file-name-mailbox vec))
! 	 (server (tramp-file-name-real-host vec))
! 	 (ssl (string-equal (tramp-file-name-method vec) tramp-imaps-method))
! 	 (port (or (tramp-file-name-port vec) (if ssl 993 143))))
      ;; return the IHT with a test override to look for the subject marker
!     (plist-put
!      (plist-put (imap-hash-make server port mbox)
! 		:test (format "^%s%s"
! 			      tramp-imap-subject-marker
! 			      (if needed-subject needed-subject "")))
!      :ssl ssl)))
  
  (provide 'tramp-imap)
  ;;; tramp-imap.el ends here
***************
*** 553,559 ****
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:yourhosthere.com:/test/") t)
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:yourhosthere.com:/test/welcommen") t)
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:yourhosthere.com:/test/welcommen") t t)
! ;;;(tramp-imap-handle-file-inode "/imap:yourhosthere.com:/test/welcome")
  ;;; (dired-copy-file "/etc/fstab" "/imap:yourhosthere.com:/test/welcome" t)
  ;;; (write-region 1 100 "/imap:yourhosthere.com:/test/welcome")
  ;;; (tramp-imap-get-file "/imap:yourhosthere.com:/test/welcome")
--- 558,564 ----
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:yourhosthere.com:/test/") t)
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:yourhosthere.com:/test/welcommen") t)
  ;;; (tramp-imap-get-file-entries (tramp-dissect-file-name "/imap:yourhosthere.com:/test/welcommen") t t)
! ;;;(tramp-imap-get-file-inode "/imap:yourhosthere.com:/test/welcome")
  ;;; (dired-copy-file "/etc/fstab" "/imap:yourhosthere.com:/test/welcome" t)
  ;;; (write-region 1 100 "/imap:yourhosthere.com:/test/welcome")
  ;;; (tramp-imap-get-file "/imap:yourhosthere.com:/test/welcome")
***************
*** 574,581 ****
  ;;;(file-exists-p "/imap:yourhosthere.com:/test/welcome2")
  ;;;(setq tramp-cache-data (make-hash-table :test 'equal))
  ;;;(tramp-imap-handle-file-attributes "/imap:yourhosthere.com:/test/welcome")
! ;;;(tramp-imap-handle-file-inode "/imap:yourhosthere.com:/test/welcommen")
! ;;;(tramp-imap-handle-file-inode "/imap:yourhosthere.com:/test/welcome")
  ;;;(file-writable-p "/imap:yourhosthere.com:/test/welcome2")
  ;;; (delete-file "/imap:yourhosthere.com:/test/welcome")
  ;;; (tramp-imap-get-file "/imap:yourhosthere.com:/test/welcommen")
--- 579,586 ----
  ;;;(file-exists-p "/imap:yourhosthere.com:/test/welcome2")
  ;;;(setq tramp-cache-data (make-hash-table :test 'equal))
  ;;;(tramp-imap-handle-file-attributes "/imap:yourhosthere.com:/test/welcome")
! ;;;(tramp-imap-get-file-inode "/imap:yourhosthere.com:/test/welcommen")
! ;;;(tramp-imap-get-file-inode "/imap:yourhosthere.com:/test/welcome")
  ;;;(file-writable-p "/imap:yourhosthere.com:/test/welcome2")
  ;;; (delete-file "/imap:yourhosthere.com:/test/welcome")
  ;;; (tramp-imap-get-file "/imap:yourhosthere.com:/test/welcommen")

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
Tramp-devel mailing list
Tramp-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/tramp-devel

Re: imap-hash.el and tramp-imap.el

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]

On Tue, 22 Sep 2009 16:52:59 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Could you, please, add "(require 'assoc)"? It's not loaded by default.

OK.

MA> * imaps didn't work for me any longer. I've patched
MA>   `tramp-imap-make-iht' until it worked ... maybe my sledge-hammer patch
MA>   could be done more elegant. There is still an error message in the
MA>   mini-buffer when opening the IMAP folder, but it doesn't seem to hurt.

Looks OK.  I haven't tested imaps thoroughly and it's not crucial to
releasing IMO.  The approach is OK; I want the clients of imap-hash.el
to modify the IHT as needed after it's created, so it can carry extra
information as appropriate to the particular client.

MA> * I have renamed `tramp-imap-handle-file-inode' to
MA>   `tramp-imap-get-file-inode'. tramp-*-handle-* function names shall be
MA>   reserved to basic file name operations implementation.

MA> * I have used `tramp-current-host' to solve the problem of unresolved
MA>   vector in `tramp-imap-passphrase-callback-function'.  That variable
MA>   was introduced in tramp.el for exactly this kind of problems.

Great.  Thanks so much for looking into this.

MA> * When you byte-compile tramp-imap.el, there is a warning about unknown
MA>   function `tramp-imap-get-message-headers'. No idea, what to do here.

I don't see that warning.  I do see:

tramp-imap.el:364:53:Warning: reference to free variable `filename'

which is strange, because with-parsed-tramp-file-name should set
`filename'.

MA> My patch is appended.

MA> I really would like to add tramp-imap.el to the CVS repository. I test
MA> on different machines, and the current situation is inconvenient for
MA> me. Do you agree for a checkin?

That's fine now, it's sufficiently presentable and maintaineable.
Latest attached for your review; no changes other than the require 'assoc.

I'll ask on emacs-devel about imap-hash.el; it will be used in Tramp and
Gnus so I would prefer to host it inside Emacs itself, probably under
lisp/net/ together with imap.el.  Tramp, in order to be self-sufficient
on XEmacs, may need to package it as well.

Ted


[-- Attachment #2: tramp-imap.el --]
[-- Type: application/emacs-lisp, Size: 25739 bytes --]

Re: imap-hash.el and tramp-imap.el

[Reiner Steib]

On Mon, Sep 21 2009, Ted Zlatanov wrote:

>     ;; TODO: make this search better
>     (if (search-forward "--text follows this line--" nil t)
> 	(delete-region (line-beginning-position) (line-end-position))
>       (error "Could not find the body separator in the encoded message!"))))

Why not use `mail-header-separator'?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: imap-hash.el and tramp-imap.el

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> MA> * imaps didn't work for me any longer. I've patched
> MA>   `tramp-imap-make-iht' until it worked ... maybe my sledge-hammer patch
> MA>   could be done more elegant. There is still an error message in the
> MA>   mini-buffer when opening the IMAP folder, but it doesn't seem to hurt.
>
> Looks OK.  I haven't tested imaps thoroughly and it's not crucial to
> releasing IMO.  The approach is OK; I want the clients of imap-hash.el
> to modify the IHT as needed after it's created, so it can carry extra
> information as appropriate to the particular client.

I have polished my change a little bit. The default port numbers are
kept now in `tramp-methods', like it is done for other methods as
well. And I also update the user tag in the structure.

> MA> * When you byte-compile tramp-imap.el, there is a warning about unknown
> MA>   function `tramp-imap-get-message-headers'. No idea, what to do here.
>
> I don't see that warning.  I do see:
>
> tramp-imap.el:364:53:Warning: reference to free variable `filename'
>
> which is strange, because with-parsed-tramp-file-name should set
> `filename'.

This I have fixed. However, there is still the warning for
`tramp-imap-get-message-headers. You could check the source code; this
function does not seem to be defined, indeed.

> MA> I really would like to add tramp-imap.el to the CVS repository. I test
> MA> on different machines, and the current situation is inconvenient for
> MA> me. Do you agree for a checkin?
>
> That's fine now, it's sufficiently presentable and maintaineable.
> Latest attached for your review; no changes other than the require 'assoc.

Thanks. There seems to be a misunderstanding; 'assoc needs to be
required in imap-hash.el, not in tramp-imap.el.

I have added tramp-imap.el to Tramp's CVS now. It is also loaded from
tramp.el, when imap-hash.el is available. There are some other cleanups
I have applied before checkin; you might check whether there is
something I have misunderstood ...

In tramp.texi, I have added some short sentences about imap and imaps
methods. I'm not so good in documentation; maybe you find a better
wording.

> I'll ask on emacs-devel about imap-hash.el; it will be used in Tramp and
> Gnus so I would prefer to host it inside Emacs itself, probably under
> lisp/net/ together with imap.el.  Tramp, in order to be self-sufficient
> on XEmacs, may need to package it as well.

OK. Once imap-hash.el has arrived Emacs CVS, I will synchronize the
corresponding changes from Tramp.

With XEmacs, we could wait until it is requested. Tramp hasn't even been
upgraded to 2.1 in the XEmacs package repository (due to license
reasons), so I'm a little bit lazy to work on it. Of course, I won't
protest if somebody else works on the XEmacs part :-)

I will continue to test tramp-imap.el. And likely, I will add an
implementation for `tramp-imap-insert-directory'; I'm using dired a lot.

> Ted

Best regards, Michael.

Re: imap-hash.el and tramp-imap.el

[Ted Zlatanov]

On Thu, 24 Sep 2009 12:38:43 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
MA> * imaps didn't work for me any longer. I've patched
MA> `tramp-imap-make-iht' until it worked ... maybe my sledge-hammer patch
MA> could be done more elegant. There is still an error message in the
MA> mini-buffer when opening the IMAP folder, but it doesn't seem to hurt.
>> 
>> Looks OK.  I haven't tested imaps thoroughly and it's not crucial to
>> releasing IMO.  The approach is OK; I want the clients of imap-hash.el
>> to modify the IHT as needed after it's created, so it can carry extra
>> information as appropriate to the particular client.

MA> I have polished my change a little bit. The default port numbers are
MA> kept now in `tramp-methods', like it is done for other methods as
MA> well. And I also update the user tag in the structure.

OK.  We'll assume Tramp's version of tramp-imap.el is authoritative from
now on, and I'll commit my updates there.

MA> This I have fixed. However, there is still the warning for
MA> `tramp-imap-get-message-headers. You could check the source code; this
MA> function does not seem to be defined, indeed.

OK, I'll make a patch for this and also to reflect Reiner's comment
about the message-mode body separator.

MA> I really would like to add tramp-imap.el to the CVS repository. I test
MA> on different machines, and the current situation is inconvenient for
MA> me. Do you agree for a checkin?
>> 
>> That's fine now, it's sufficiently presentable and maintaineable.
>> Latest attached for your review; no changes other than the require 'assoc.

MA> Thanks. There seems to be a misunderstanding; 'assoc needs to be
MA> required in imap-hash.el, not in tramp-imap.el.

Oops.  I'll make the change in the imap-hash.el that's going into Emacs.

MA> I have added tramp-imap.el to Tramp's CVS now. It is also loaded from
MA> tramp.el, when imap-hash.el is available. There are some other cleanups
MA> I have applied before checkin; you might check whether there is
MA> something I have misunderstood ...

OK, something to do over the weekend.

MA> In tramp.texi, I have added some short sentences about imap and imaps
MA> methods. I'm not so good in documentation; maybe you find a better
MA> wording.

OK.

MA> With XEmacs, we could wait until it is requested. Tramp hasn't even been
MA> upgraded to 2.1 in the XEmacs package repository (due to license
MA> reasons), so I'm a little bit lazy to work on it. Of course, I won't
MA> protest if somebody else works on the XEmacs part :-)

MA> I will continue to test tramp-imap.el. And likely, I will add an
MA> implementation for `tramp-imap-insert-directory'; I'm using dired a lot.

Wonderful.  I'll soon improve imap-hash.el to add SEARCH when the :test
key is set in the IHT, instead of FETCH, so only the matching messages
are retrieved.  Also I need to fix the error when a nonexistent mailbox
is specified.  If you see any other issues, please let me know.

Ted

Re: imap-hash.el and tramp-imap.el

[Ted Zlatanov]

On Thu, 24 Sep 2009 08:44:50 +0200 Reiner Steib <reinersteib+gmane@imap.cc> wrote: 

RS> On Mon, Sep 21 2009, Ted Zlatanov wrote:
>> ;; TODO: make this search better
>> (if (search-forward "--text follows this line--" nil t)
>> (delete-region (line-beginning-position) (line-end-position))
>> (error "Could not find the body separator in the encoded message!"))))

RS> Why not use `mail-header-separator'?

I looked for "message-*" variable names, didn't find anything relevant,
put a TODO in there and then promptly forgot about it :)

Thanks for the suggestion, I'll update this in the Tramp CVS.

Ted

Re: imap-hash.el and tramp-imap.el

[Ted Zlatanov]

On Fri, 25 Sep 2009 15:48:07 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

MA> This I have fixed. However, there is still the warning for
MA> `tramp-imap-get-message-headers. You could check the source code; this
MA> function does not seem to be defined, indeed.

I removed the file-local-copy handler.  From looking at the source it
seems that it's automatically filled-in by Tramp, so the implementation
was not necessary.  The function I removed (commented out until you
decide what's next) was left over from the old tramp-imap and I didn't
hit it in any of my testing; sorry for the annoyance.

TZ> OK, I'll make a patch for this and also to reflect Reiner's comment
TZ> about the message-mode body separator.

This was done in imap-hash.el, which is now in the Emacs CVS.

I didn't do the docs or much else.  I did see your comment fixes--thanks
for cleaning them up.  Your other changes look fine too.

Thanks
Ted

Re: imap-hash.el and tramp-imap.el

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> MA> This I have fixed. However, there is still the warning for
> MA> `tramp-imap-get-message-headers. You could check the source code; this
> MA> function does not seem to be defined, indeed.
>
> I removed the file-local-copy handler.  From looking at the source it
> seems that it's automatically filled-in by Tramp, so the implementation
> was not necessary.  The function I removed (commented out until you
> decide what's next) was left over from the old tramp-imap and I didn't
> hit it in any of my testing; sorry for the annoyance.

OK, I'll check it.

I've checked in some changes as well.

* 'assoc must also be required in tramp-imap.el, you were right.

* I've implemented handlers for `copy-file', `insert-directory' and
  `rename-file'. `copy-file' and `rename-file' need optimization, when
  both files are located on the IMAP server, maybe you can have a look
  on them. I did also extend `tramp-imap-put-file' and
  `tramp-imap-get-file' because of this.

* Some few other, minor fixes.

Next days, after some further tests, I'll bring tramp-imap.el to Emacs CVS.

> Thanks
> Ted

Best regards, Michael.

Re: imap-hash.el and tramp-imap.el

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Next days, after some further tests, I'll bring tramp-imap.el to Emacs CVS.

I've added handlers for `directory-files-and-attributes' and
`file-local-copy'. Furthermore, I've added a TODO section at the end of
the file, containing some problems I've observed during testing. Maybe
you can have a look on them; all those items are out of my capabilities.

tramp-imap.el is added now to Emacs CVS as well.

>> Thanks
>> Ted

Best regards, Michael.