Time for partial download of nnimap messages again?

Time for partial download of nnimap messages again?

[Steinar Bang]

Now that agent is always used by nnimap as a local cache, it may be
time to once more start looking at partial downloads of MIME messages?

Eg. downloading the text part, but leave that big videoclip on the
server, until we decide to look at it.

A quick revisit of the idea:
 - in the agent cache, replace the delayed parts with a
   message/external-body URL part
	<http://www.faqs.org/rfcs/rfc2017.html>
   where the URL is an IMAP URL:
	<http://www.faqs.org/rfcs/rfc2192.html>
 - when the user wishes to view or save the delayed part, it is
   downloaded from the server, and the message/external-body is
   replaced with the actual part

What Simon didn't like about this, IIRC, was that you changed the
message from the original.

But you don't, really.  The original message is kept in its priestine
state on the server.  What happens to the cached message is just an
internal Gnus implementation issue, IMO.

Re: Time for partial download of nnimap messages again?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> What Simon didn't like about this, IIRC, was that you changed the
> message from the original.

What about generality?  Your suggestion works for nnimap, but WIBNI
there was a scheme that could be made to work generally?  Something
like an augmented backend interface.  Some backends could support a
function to retrieve the MIME structure of a message.  (Like a TOC,
also similar to what C-d does on a message.  But a datastructure.)
Then additional functions should be added to retrieve certain parts
of a message.

Then Gnus would look whether the backend supports the new functions.
If so, it would ask for the TOC and retrieve all text/* (or whatever)
parts, or all small parts, or all inline parts, or something, and
replaces the other ones with a button.

Would this be more difficult to do?

A new backend could then store messages on disk like that.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Time for partial download of nnimap messages again?

[Steinar Bang]

>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):

> Steinar Bang <sb@dod.no> writes:
>> What Simon didn't like about this, IIRC, was that you changed the
>> message from the original.

> What about generality?  Your suggestion works for nnimap, but WIBNI
> there was a scheme that could be made to work generally?

Perhaps.  But I don't think any other protocols support multipart MIME
messages the way IMAP does?  Is it possible to do the same thing in,
say, NNTP?  

Re: Time for partial download of nnimap messages again?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> Perhaps.  But I don't think any other protocols support multipart MIME
> messages the way IMAP does?  Is it possible to do the same thing in,
> say, NNTP?  

Like I said before, a new backend could store articles on disk like
that.  Of the existing backends, it's probably only suitable for
nnimap, but why restrict ourselves?

For example, nnml could be augmented to auto-split viewed messages
into several files, I guess.  Then, slowly, all the big attachments
would migrate out of the main file and the Gnus article display would
be a lot faster because it didn't have to filter out all that base64
stuff.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Time for partial download of nnimap messages again?

[Steinar Bang]

>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):

> For example, nnml could be augmented to auto-split viewed messages
> into several files, I guess.  Then, slowly, all the big attachments
> would migrate out of the main file and the Gnus article display
> would be a lot faster because it didn't have to filter out all that
> base64 stuff.

Not having to parse attachments every time, would make sense, I guess.
It would make the nnml groups incompatible with older versions of
Gnus, and different from newsspools.  I do not know if this would be a
problem?

Re: Time for partial download of nnimap messages again?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> Not having to parse attachments every time, would make sense, I guess.
> It would make the nnml groups incompatible with older versions of
> Gnus, and different from newsspools.  I do not know if this would be a
> problem?

It was only an example.  I'm sure it would also work to create
nnmk.el or whatever.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Time for partial download of nnimap messages again?

[Steinar Bang]

>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):

> What about generality?  Your suggestion works for nnimap, but WIBNI
> there was a scheme that could be made to work generally?  Something
> like an augmented backend interface.  Some backends could support a
> function to retrieve the MIME structure of a message.  (Like a TOC,
> also similar to what C-d does on a message.  But a datastructure.)

Btw the C-d result is a good fit to what you get from IMAP.

> Then additional functions should be added to retrieve certain parts
> of a message.

> Then Gnus would look whether the backend supports the new functions.
> If so, it would ask for the TOC and retrieve all text/* (or whatever)
> parts, or all small parts, or all inline parts, or something, and
> replaces the other ones with a button.

> Would this be more difficult to do?

I don't know.  My initial proposal should be possible to do with a
handler for the MIME type message/external-body, and changes to the
nnimap backend.  What do the lisp gurus say?  What's the preferred
approach?

There seems to be quite a bit of conservativism wrt. changing the
backend interface.

_If_ there is to be a backend change I would like it to go hand in
hand with other changes, like making the backend interface
asynchronous. 

Re: Time for partial download of nnimap messages again?

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> There seems to be quite a bit of conservativism wrt. changing the
> backend interface.

My proposed change would be backward-compatible, so no problem there,
I think.

Maybe the conservatism comes from the thing being hard to understand
and not modularized as well as it could be...

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: Time for partial download of nnimap messages again?

[Steinar Bang]

>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):

> Maybe the conservatism comes from the thing being hard to understand
> and not modularized as well as it could be...

I really think it should be redesigned to become asynchronous.

Re: Time for partial download of nnimap messages again?

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):
>
>> Maybe the conservatism comes from the thing being hard to understand
>> and not modularized as well as it could be...
>
> I really think it should be redesigned to become asynchronous.

IMHO using a framework like the URL package for backend communication
would be good.  Perhaps even extended further to be more thread
friendly.

Another good thing would be remove all uses of Gnus functions and Gnus
variables from backends, it would be a first step into making them
more modular.

Re: Time for partial download of nnimap messages again?

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> Steinar Bang <sb@dod.no> writes:
>>>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):

>>> Maybe the conservatism comes from the thing being hard to
>>> understand and not modularized as well as it could be...

>> I really think it should be redesigned to become asynchronous.

> IMHO using a framework like the URL package for backend
> communication would be good.  Perhaps even extended further to be
> more thread friendly.

With asynchronous backends, the idea is that you don't need threads.
The network traffic is powered by the event loop checking for waiting
input on sockets.

But the backend interface would need a complete restructuring: from
"request/response and then use the response to draw summary and
article buffers" to "send a callback with the request, and have the
callback draw summary and article buffers, as data arrives".

There is of course intermediates, where only the requests that
potentially take a long time are made non-blocking, and the callbacks
build up local data structures, and notify when finished, rather than
do the actual drawing.

Re: Time for partial download of nnimap messages again?

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Simon Josefsson <jas@extundo.com>:
>
>> Steinar Bang <sb@dod.no> writes:
>>>>>>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann):
>
>>>> Maybe the conservatism comes from the thing being hard to
>>>> understand and not modularized as well as it could be...
>
>>> I really think it should be redesigned to become asynchronous.
>
>> IMHO using a framework like the URL package for backend
>> communication would be good.  Perhaps even extended further to be
>> more thread friendly.
>
> With asynchronous backends, the idea is that you don't need threads.
> The network traffic is powered by the event loop checking for waiting
> input on sockets.

I don't think an asychnronous backend interface is enough -- at least
nnimap spend an non-neglible amount of its time processing, not
waiting for IO.  So it would still lock up Emacs when doing things.
And Gnus, where most of the time is spent when entering groups,
pressing g, etc, is not waiting for IO most of its time either.  (Try
elp-instrument-package on gnus-, imap-, and nnimap-. Only when doing
very big operations will imap- consume much of the time, and the IO
waiting is there.)

> But the backend interface would need a complete restructuring: from
> "request/response and then use the response to draw summary and
> article buffers" to "send a callback with the request, and have the
> callback draw summary and article buffers, as data arrives".

Yup.  Having incrementally built summary buffers (when you scroll the
emacs buffer, with some intelligent prefetching) would be quite nice.

Re: Time for partial download of nnimap messages again?

[Wes Hardaker]

>>>>> On Sat, 27 Jul 2002 16:09:18 +0200, Simon Josefsson <jas@extundo.com> said:

Simon> Yup.  Having incrementally built summary buffers (when you scroll the
Simon> emacs buffer, with some intelligent prefetching) would be quite nice.

I submitted a patch a while ago that puts a simple (sit-for 0) in the
processing loop to do just that.  However, you still have to wait for
it to complete before you can actually use the summary buffer, but at
least you can wait for it.

Now, if only my schedule would clear up a bit so I can get the
assignment done with my current employer (likely Sept).

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: Time for partial download of nnimap messages again?

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Steinar Bang <sb@dod.no> writes:
>
>> What Simon didn't like about this, IIRC, was that you changed the
>> message from the original.
>
> What about generality?  Your suggestion works for nnimap, but WIBNI
> there was a scheme that could be made to work generally?  Something
> like an augmented backend interface.  Some backends could support a
> function to retrieve the MIME structure of a message.  (Like a TOC,
> also similar to what C-d does on a message.  But a datastructure.)
> Then additional functions should be added to retrieve certain parts
> of a message.
>
> Then Gnus would look whether the backend supports the new functions.
> If so, it would ask for the TOC and retrieve all text/* (or whatever)
> parts, or all small parts, or all inline parts, or something, and
> replaces the other ones with a button.
>
> Would this be more difficult to do?

Depends on how general you want to make it, but I think even achieving
the same amount of functionality would be more difficult.  Having a
more detailed analysis on what needs to be done is probably useful,
like the one I did for Steinar's idea, or preferrably even more
detailed.

Re: Time for partial download of nnimap messages again?

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

> Now that agent is always used by nnimap as a local cache

I'm not convinced the change is good, there are at least some known
problems we need to solve:

1) The daemon stopped working for someone, probably related by this
   change.

2) Expiring the agent directory doesn't happen automatically.  This
   might be what some people want, but not others.

3) Unagentized nnimap (and other) servers do not get local caching at
   all.  Which means people that don't agentize their nnimap server
   actually get no overview cache at all, since the nnimap cache has
   been disabled. Bad bad!  I think non-local servers should be
   agentized by default, but I'm not sure how to change it in a
   backwards compatible way.

4) Slowness in expiring?  Fixed?

5) Any mark problems still occuring?

If those who disabled the agent re-enabled it and reported the
problems they see, it would be very useful.  If there are more
undiscovered problems that looks difficult to solve, reverting the
default value change should be considered.

> it may be time to once more start looking at partial downloads of
> MIME messages?

Feel free to do so. :-)

> Eg. downloading the text part, but leave that big videoclip on the
> server, until we decide to look at it.
>
> A quick revisit of the idea:
>  - in the agent cache, replace the delayed parts with a
>    message/external-body URL part
> 	<http://www.faqs.org/rfcs/rfc2017.html>
>    where the URL is an IMAP URL:
> 	<http://www.faqs.org/rfcs/rfc2192.html>
>  - when the user wishes to view or save the delayed part, it is
>    downloaded from the server, and the message/external-body is
>    replaced with the actual part
>
> What Simon didn't like about this, IIRC, was that you changed the
> message from the original.
>
> But you don't, really.  The original message is kept in its priestine
> state on the server.  What happens to the cached message is just an
> internal Gnus implementation issue, IMO.

Yup.  I have troubles visualising how the UI for this would work
though.  It should handle disconnected support too.  But I guess it
isn't impossible to do.  Let's see.

1) Have a nnimap-undownload-part variable that can contain predicates
   such as (and (string= content-type "audio/mp3") (> size 123456))
   that says which parts to not download.

2) In nnimap-request-article (possibly with help from
   nnimap-request-headers for simple predicates) evaluate the
   predicate and download all other MIME parts, and replace the
   offending part with the multipart/external IMAP URL stuff.

3) In gnus, somehow add a mark saying that an article is not fully
   downloaded.  Maybe a agent specific mark?

4) In gnus, mark the MIME button for replaced entities somehow, to
   make it clear that Gnus replaced it.

5) In gnus, have a command to re-download the whole message.  It
   should work even when disconnected (compare the @ download mark).

6) In gnus, somehow fix C-d to reflect the external property as well?

As I personally don't really need the feature, I don't think I'll do
much more than this though. :-)

Re: Time for partial download of nnimap messages again?

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> As I personally don't really need the feature, I don't think I'll do
> much more than this though. :-)

Oh, but you do need it.  You just don't know it yet...;-)
(Now _where_ did I put that 8MB MPEG of Max Biaggi doing his victory
wheelie at Chech GP in Brno in 1998...;-) )