fetch only a portion of an article? (nnimap mainly)

fetch only a portion of an article? (nnimap mainly)

[Ted Zlatanov]

In Gnus, you call gnus-request-article to get an article.  What if you
know you only need the first N lines (or bytes)?  Can nnimap.el
support that?  I can change the gnus-request-article function to
accomodate such an option, and it will be passed to
nnimap-request-article.  I just don't know enough about the IMAP
protocol to implement it.

I need this specifically for nnimap interactions, but I'm sure it can
be implemented for other backends as well.  It's useful, for instance,
when summarizing something about the contents of the article, but you
don't want to fetch more than N lines of it.  I would use it to
summarize qmail bounce reports - right now, I have to fetch the whole
article just because I need to see what's in the first 10 lines.

Ted

Re: fetch only a portion of an article? (nnimap mainly)

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> In Gnus, you call gnus-request-article to get an article.  What if you
> know you only need the first N lines (or bytes)?  Can nnimap.el
> support that?  I can change the gnus-request-article function to
> accomodate such an option, and it will be passed to
> nnimap-request-article.  I just don't know enough about the IMAP
> protocol to implement it.
>
> I need this specifically for nnimap interactions, but I'm sure it can
> be implemented for other backends as well.  It's useful, for instance,
> when summarizing something about the contents of the article, but you
> don't want to fetch more than N lines of it.  I would use it to
> summarize qmail bounce reports - right now, I have to fetch the whole
> article just because I need to see what's in the first 10 lines.

IMAP support partial retrieval (byte counts), and even retrieval based
on the selected MIME structures (i.e., retrieve entire message except
one big attachment).  I think the problem may be in designing the back
end interface properly, so it is flexible enough.  Ideally there
should be some way to retrieve the MIME structure of an article, and
then make Gnus chose which MIME parts to retrieve.  But having a line
count may be sufficient as a start.  However, there is a big problem
with caching.  If an incomplete article ends up in the asynchronous
pre-fetcher, agent cache, user cache, etc, it will require some
tinkering to invalidate that copy and force a re-query on the entire
article.

Re: fetch only a portion of an article? (nnimap mainly)

[Ted Zlatanov]

On Fri, 04 Mar 2005, jas@extundo.com wrote:

> IMAP support partial retrieval (byte counts), and even retrieval based
> on the selected MIME structures (i.e., retrieve entire message except
> one big attachment).  I think the problem may be in designing the back
> end interface properly, so it is flexible enough.  

Can you write the nnimap function to get N bytes with an extra
parameter (you can merge it with nnimap-request-article-part or not,
up to you)?

I'll do the rest of the API.

> Ideally there should be some way to retrieve the MIME structure of
> an article, and then make Gnus chose which MIME parts to retrieve.

I'll take a look at that stuff.  Unfortunately the other backends
don't support this, so it's less useful than a line count IMO
(although it can make the Gnus IMAP experience much better).

> But having a line count may be sufficient as a start.  However,
> there is a big problem with caching.  If an incomplete article ends
> up in the asynchronous pre-fetcher, agent cache, user cache, etc, it
> will require some tinkering to invalidate that copy and force a
> re-query on the entire article.

I can deal with this, I think.  I looked at the code and it's not too
bad.  I just won't enter the article in the cache if it was fetched
partially.  Do you think this is OK?  When and if retrieval by MIME
part is done I can change this, but for now it seems like a good
compromise.

Ted

Re: fetch only a portion of an article? (nnimap mainly)

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Fri, 04 Mar 2005, jas@extundo.com wrote:
>
>> IMAP support partial retrieval (byte counts), and even retrieval based
>> on the selected MIME structures (i.e., retrieve entire message except
>> one big attachment).  I think the problem may be in designing the back
>> end interface properly, so it is flexible enough.  
>
> Can you write the nnimap function to get N bytes with an extra
> parameter (you can merge it with nnimap-request-article-part or not,
> up to you)?

I can try.  Two parameters, right?  The start offset, and the length?

>> Ideally there should be some way to retrieve the MIME structure of
>> an article, and then make Gnus chose which MIME parts to retrieve.
>
> I'll take a look at that stuff.  Unfortunately the other backends
> don't support this, so it's less useful than a line count IMO
> (although it can make the Gnus IMAP experience much better).

Sure.

Re: fetch only a portion of an article? (nnimap mainly)

[Ted Zlatanov]

On Mon, 07 Mar 2005, jas@extundo.com wrote:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:
> 
>> On Fri, 04 Mar 2005, jas@extundo.com wrote:
>>
>>> IMAP support partial retrieval (byte counts), and even retrieval based
>>> on the selected MIME structures (i.e., retrieve entire message except
>>> one big attachment).  I think the problem may be in designing the back
>>> end interface properly, so it is flexible enough.  
>>
>> Can you write the nnimap function to get N bytes with an extra
>> parameter (you can merge it with nnimap-request-article-part or not,
>> up to you)?
> 
> I can try.  Two parameters, right?  The start offset, and the
> length?

I can't imagine a reason to need an offset other than 0, but sure :)

I'll watch for the CVS commit.

Ted

Re: fetch only a portion of an article? (nnimap mainly)

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Mon, 07 Mar 2005, jas@extundo.com wrote:
>
>> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>> 
>>> On Fri, 04 Mar 2005, jas@extundo.com wrote:
>>>
>>>> IMAP support partial retrieval (byte counts), and even retrieval based
>>>> on the selected MIME structures (i.e., retrieve entire message except
>>>> one big attachment).  I think the problem may be in designing the back
>>>> end interface properly, so it is flexible enough.  
>>>
>>> Can you write the nnimap function to get N bytes with an extra
>>> parameter (you can merge it with nnimap-request-article-part or not,
>>> up to you)?
>> 
>> I can try.  Two parameters, right?  The start offset, and the
>> length?
>
> I can't imagine a reason to need an offset other than 0, but sure :)

Chunked downloading?  Another feature request has been to cancel
downloading huge attachments.  Right now, this is not possible.  If
the attachment is downloaded in 1MB parts, C-g would cancel the
operation faster.

> I'll watch for the CVS commit.

Don't hold your breath, but I'll try to get around to it.

Re: fetch only a portion of an article? (nnimap mainly)

[Ted Zlatanov]

On Tue, 08 Mar 2005, jas@extundo.com wrote:

> Chunked downloading?  Another feature request has been to cancel
> downloading huge attachments.  Right now, this is not possible.  If
> the attachment is downloaded in 1MB parts, C-g would cancel the
> operation faster.

I promise that if you get this in the backend, I'll write the frontend
code to do chunked downloads :)

Ted