downloading attachments on demand

downloading attachments on demand

[Sam Steingold]

Is there a way to open a message without downloading all the attachments?

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 4 (Stentz)
http://www.iris.org.il http://www.savegushkatif.org http://www.camera.org
http://ffii.org http://www.openvotingconsortium.org
There is an exception to every rule, including this one.

Re: downloading attachments on demand

[Lars Magne Ingebrigtsen]

Sam Steingold <sds@podval.org> writes:

> Is there a way to open a message without downloading all the attachments?

If it's a multipart/mixed message, for instance, then there's
probably no way to query the storage back ends for just the textual
bits.  At least nntp or pop doesn't support anything like that...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: downloading attachments on demand

[Sam Steingold]

> * Lars Magne Ingebrigtsen <ynefv@tahf.bet> [2006-04-11 08:13:26 +0200]:
>
> Sam Steingold <sds@podval.org> writes:
>
>> Is there a way to open a message without downloading all the attachments?
>
> If it's a multipart/mixed message, for instance, then there's
> probably no way to query the storage back ends for just the textual
> bits.  At least nntp or pop doesn't support anything like that...

All I care about is IMAP (and it appears that Evolution and Mozilla can
download just the text parts and the list of attachments, not the
attachments themselves).

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 4 (Stentz)
http://pmw.org.il http://www.camera.org http://www.jihadwatch.org
http://www.honestreporting.com http://www.dhimmi.com http://ffii.org
char*a="char*a=%c%s%c;main(){printf(a,34,a,34);}";main(){printf(a,34,a,34);}

Re: downloading attachments on demand

[Jouni K Seppanen]

Sam Steingold <sds@podval.org> writes:

>> * Lars Magne Ingebrigtsen <ynefv@tahf.bet> [2006-04-11 08:13:26 +0200]:
>> Sam Steingold <sds@podval.org> writes:
>>> Is there a way to open a message without downloading all the attachments?
>>
>> If it's a multipart/mixed message, for instance, then there's
>> probably no way to query the storage back ends for just the textual
>> bits.  At least nntp or pop doesn't support anything like that...
>
> All I care about is IMAP (and it appears that Evolution and Mozilla can
> download just the text parts and the list of attachments, not the
> attachments themselves).

Yes, you can do that with IMAP. I just sent myself a message
containing the word "foo", an attachment consisting of the word "bar",
and my signature. Here's me querying the IMAP server about the
message:

a fetch 3475 bodystructure
* 3475 FETCH (BODYSTRUCTURE (("text" "plain" NIL NIL NIL "8bit" 5 1 NIL NIL NIL)("text" "plain" NIL NIL NIL "8bit" 3 0 NIL ("attachment" NIL) NIL)("text" "plain" NIL NIL NIL "8bit" 12 2 NIL NIL NIL) "mixed" ("boundary" "=-=-=") NIL NIL))
a OK FETCH completed.

You could almost use the Lisp reader on that... Here's how to get the
first part:

a fetch 3475 body[1]
* 3475 FETCH (BODY[1] {5}
foo
)
* 3475 FETCH (FLAGS (\Seen \Recent))
a OK FETCH completed.

Here's the attachment:

a fetch 3475 body[2]
* 3475 FETCH (BODY[2] {3}
bar)
a OK FETCH completed.

And the final part:

a fetch 3475 body[3]
* 3475 FETCH (BODY[3] {12}
-- 
Jouni
)
a OK FETCH completed.

(This is all documented in RFC 2060.)

From what I understand about the internal workings of Gnus, you'd have
to alter the backend protocol to make use of this. Instead of getting
a buffer filled with a message that it can parse for MIME structure,
the frontend would ask for some sort of a pre-parsed message object,
with stubs for the attachments, and only if the user wants to do
anything with them, would the frontend query the backend for their
contents.

-- 
Jouni

Re: downloading attachments on demand

[Lars Magne Ingebrigtsen]

Jouni K Seppanen <jks@iki.fi> writes:

> From what I understand about the internal workings of Gnus, you'd have
> to alter the backend protocol to make use of this. Instead of getting
> a buffer filled with a message that it can parse for MIME structure,
> the frontend would ask for some sort of a pre-parsed message object,
> with stubs for the attachments, and only if the user wants to do
> anything with them, would the frontend query the backend for their
> contents.

Well, nnimap could just return a valid MIME document with
message/external-body parts, I think.  Gnus and nnimap would have to
implement a special external-body scheme to download the parts, but
that shouldn't be too much work...

I don't use imap myself, so I'm probably not the person to implement
this. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: downloading attachments on demand

[Sam Steingold]

> * Lars Magne Ingebrigtsen <ynefv@tahf.bet> [2006-04-12 06:38:01 +0200]:
>
> Jouni K Seppanen <jks@iki.fi> writes:
>
>> From what I understand about the internal workings of Gnus, you'd have
>> to alter the backend protocol to make use of this. Instead of getting
>> a buffer filled with a message that it can parse for MIME structure,
>> the frontend would ask for some sort of a pre-parsed message object,
>> with stubs for the attachments, and only if the user wants to do
>> anything with them, would the frontend query the backend for their
>> contents.
>
> Well, nnimap could just return a valid MIME document with
> message/external-body parts, I think.  Gnus and nnimap would have to
> implement a special external-body scheme to download the parts, but
> that shouldn't be too much work...
>
> I don't use imap myself, so I'm probably not the person to implement
> this.

you are the Gnus maintainer, so you might prod nnimap.el authors into
doing this.

-- 
Sam Steingold (http://www.podval.org/~sds) on Fedora Core release 4 (Stentz)
http://www.memri.org http://www.camera.org http://www.openvotingconsortium.org
http://www.mideasttruth.com http://www.iris.org.il http://pmw.org.il
Is there another word for synonym?

Re: downloading attachments on demand

[Lars Magne Ingebrigtsen]

Sam Steingold <sds@podval.org> writes:

> you are the Gnus maintainer, so you might prod nnimap.el authors into
> doing this.

*prod* *prod*

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: downloading attachments on demand

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Jouni K Seppanen <jks@iki.fi> writes:
>
>> From what I understand about the internal workings of Gnus, you'd have
>> to alter the backend protocol to make use of this. Instead of getting
>> a buffer filled with a message that it can parse for MIME structure,
>> the frontend would ask for some sort of a pre-parsed message object,
>> with stubs for the attachments, and only if the user wants to do
>> anything with them, would the frontend query the backend for their
>> contents.
>
> Well, nnimap could just return a valid MIME document with
> message/external-body parts, I think.  Gnus and nnimap would have to
> implement a special external-body scheme to download the parts, but
> that shouldn't be too much work...

I think we've discussed this approach before, and I'm not sure it is
the right one.  For example, how will this work with the various
caches in Gnus?  There is the backlog and the agent article cache.
Maybe others?

I'm afraid I don't have time to work on this.  Would someone else be
willing to make a stab at it?  Perhaps just ignore the backlog and
agent cache issues, and see if it is possible to make nnimap return
only partial articles if the article is larger than x KB should not be
too complex.  Then add a command that asks nnimap to really download
the entire article.  Once this is done, we can see if it is possible
to prune the agent/backlog caches correctly.

Alternatively, as I believe I have proposed before, we could add a new
backend interface to allow Gnus to query the MIME structure of
articles, and then only request partial articles, and handle the Gnus
cache coherency stuff in Gnus instead of in some nnimap-specific
command.

Re: downloading attachments on demand

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> Alternatively, as I believe I have proposed before, we could add a new
> backend interface to allow Gnus to query the MIME structure of
> articles, and then only request partial articles, and handle the Gnus
> cache coherency stuff in Gnus instead of in some nnimap-specific
> command.

Instead of issuing `nn*-request-article' calls, Gnus could call
`nn*-request-structure' (or the like) which would return the MIME
structure.  `nn*-request-structure' wouldn't do any caching -- the
structures should be pretty small, anyway.  Then we'd need a call like
`nn*-request-part' that would request some content.

I don't think it would be difficult to implement this, but as I don't
use imap, I'm not really equipped to do so.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: downloading attachments on demand

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Alternatively, as I believe I have proposed before, we could add a new
>> backend interface to allow Gnus to query the MIME structure of
>> articles, and then only request partial articles, and handle the Gnus
>> cache coherency stuff in Gnus instead of in some nnimap-specific
>> command.
>
> Instead of issuing `nn*-request-article' calls, Gnus could call
> `nn*-request-structure' (or the like) which would return the MIME
> structure.  `nn*-request-structure' wouldn't do any caching -- the
> structures should be pretty small, anyway.  Then we'd need a call like
> `nn*-request-part' that would request some content.

Yup, sounds good.

> I don't think it would be difficult to implement this, but as I don't
> use imap, I'm not really equipped to do so.

I hope someone will step forward to work on this...

Re: downloading attachments on demand

[Ted Zlatanov]

On 18 Apr 2006, jas@extundo.com wrote:

> Alternatively, as I believe I have proposed before, we could add a
> new backend interface to allow Gnus to query the MIME structure of
> articles, and then only request partial articles, and handle the
> Gnus cache coherency stuff in Gnus instead of in some
> nnimap-specific command.

Can we combine this work with the flags work we discussed a while ago
(arbitrary flags/tags)?  It would probably be easier to do a major
backend addition like that in one effort instead of twice.

It may also be worthwhile to look at the entire nn* backend system and
how it's implemented in each backend.  I'm sure there are lessons to
be learned and code that could be rewritten for better performance or
reliability.

Ted

Re: downloading attachments on demand

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On 18 Apr 2006, jas@extundo.com wrote:
>
>> Alternatively, as I believe I have proposed before, we could add a
>> new backend interface to allow Gnus to query the MIME structure of
>> articles, and then only request partial articles, and handle the
>> Gnus cache coherency stuff in Gnus instead of in some
>> nnimap-specific command.
>
> Can we combine this work with the flags work we discussed a while ago
> (arbitrary flags/tags)?

I don't see much connection, actually.  How would you combine these
two things?

> It would probably be easier to do a major backend addition like that
> in one effort instead of twice.

Adding a backend interface isn't that difficult.

> It may also be worthwhile to look at the entire nn* backend system and
> how it's implemented in each backend.  I'm sure there are lessons to
> be learned and code that could be rewritten for better performance or
> reliability.

Yup.

Re: downloading attachments on demand

[Ted Zlatanov]

On 18 Apr 2006, jas@extundo.com wrote:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>
>> On 18 Apr 2006, jas@extundo.com wrote:
>>
>>> Alternatively, as I believe I have proposed before, we could add a
>>> new backend interface to allow Gnus to query the MIME structure of
>>> articles, and then only request partial articles, and handle the
>>> Gnus cache coherency stuff in Gnus instead of in some
>>> nnimap-specific command.
>>
>> Can we combine this work with the flags work we discussed a while ago
>> (arbitrary flags/tags)?
>
> I don't see much connection, actually.  How would you combine these
> two things?

I don't mean they are directly connected, but that they both require a
rethinking of the backend API.  If they both require it, it's better
to do a major change like that just once.

>> It would probably be easier to do a major backend addition like that
>> in one effort instead of twice.
>
> Adding a backend interface isn't that difficult.

I mean the backend API.  It should change to allow body structure
queries and arbitrary tags.  I don't think in its current form it can
do either easily, but perhaps I'm wrong.

>> It may also be worthwhile to look at the entire nn* backend system and
>> how it's implemented in each backend.  I'm sure there are lessons to
>> be learned and code that could be rewritten for better performance or
>> reliability.
>
> Yup.

This, too, could require significant changes...  So I would suggest
combining all that work in one project instead of spreading the pain
and knowledge over several projects.

Ted