Partial article download

Partial article download

[Lars Magne Ingebrigtsen]

Since nnimap now works perfectly (or so I'll assume until the next bug
report), perhaps it's time to think about partial article downloads.

So, Gnus would download the message structure and the first bit (if the
user flipped a variable).  But would there be a new Gnus Summary command
for "fetch complete article", or would there be buttons in the article
and...  stuff?

The "fetch complete article" method sounds way easier to implement.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Since nnimap now works perfectly (or so I'll assume until the next bug
> report), 


I still have problems entering my Agent-backed nnimap groups:

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p (nil nil))
  nnheader-insert-nov([634 "Debian package updates on stoner.mydomain.se " "root <root@stoner.mydomain.se> " "Fri, 17 Sep 2010 21:52:25 +0100 " "<E1Owhuz-0002st-DO@stoner.mydomain.se>" nil -1 43 nil ((Id nil nil) (Ip nil) (Queue))])
  mapc(nnheader-insert-nov ([634 "Debian package updates on stoner.mydomain.se " "root <root@stoner.mydomain.se> " "Fri, 17 Sep 2010 21:52:25 +0100 " "<E1Owhuz-0002st-DO@stoner.mydomain.se>" nil -1 43 nil (... ... ...)]))
  nnvirtual-convert-headers()
  gnus-agent-retrieve-headers((551 552 553 554 555 556 557 561 562 564 565 566 567 570 572 584 585 609 633 634) "nnimap+privat:INBOX" nil)
  gnus-retrieve-headers((551 552 553 554 555 556 557 561 562 564 565 566 567 570 572 584 585 609 633 634) "nnimap+privat:INBOX" nil)
  gnus-cache-retrieve-headers((551 552 553 554 555 556 557 561 562 564 565 566 567 570 572 584 585 609 633 634) "nnimap+privat:INBOX" nil)
  gnus-retrieve-headers((551 552 553 554 555 556 557 561 562 564 565 566 567 570 572 584 585 609 633 634) "nnimap+privat:INBOX" nil)
  gnus-fetch-headers((551 552 553 554 555 556 557 561 562 564 565 566 567 570 572 584 585 609 633 634))
  gnus-select-newsgroup("nnimap+privat:INBOX" nil nil)
  gnus-summary-read-group-1("nnimap+privat:INBOX" nil t nil nil nil)
  gnus-summary-read-group("nnimap+privat:INBOX" nil t nil nil nil nil)
  gnus-group-read-group(nil t)
  gnus-group-select-group(nil)
  gnus-topic-select-group(nil)
  call-interactively(gnus-topic-select-group)

Re: Partial article download

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> I still have problems entering my Agent-backed nnimap groups:

Oh, darn.  I forgot about the Agent again.  I'll switch agenting on on
one of the nnimap servers and get everything working.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Since nnimap now works perfectly (or so I'll assume until the next bug
> report), perhaps it's time to think about partial article downloads.

> So, Gnus would download the message structure and the first bit

Do you know that the text/plain part is always the first part
of a message?  Wouldn't it be better to just pull down all text/plain
parts of a message?

> (if the user flipped a variable).  But would there be a new Gnus
> Summary command for "fetch complete article", or would there be
> buttons in the article and...  stuff?

The old postings I linked back to, suggested a buttons in the artile
approach, but...

> The "fetch complete article" method sounds way easier to implement.

Yes it does, doesn't it?

But whatever way you do it, partial downloading should cooperate nicely
with the agent.

Re: Partial article download

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Do you know that the text/plain part is always the first part
> of a message?  Wouldn't it be better to just pull down all text/plain
> parts of a message?

Hm.  Yes... when requesting the headers on group entry, I can also ask
for the article structure.  So when it comes time to request the
article, it should be easy enough to ask for all the text/plain parts.
Or perhaps all text/* parts?  Well, that can be user-configurable. 

> But whatever way you do it, partial downloading should cooperate nicely
> with the agent.

If we have a "fetch complete article" message, it could just bind the
agent thing to nil and then request it straight from the IMAP server?
Then things should Just Work.  That is, the partial articles would be
downloaded by the agent and stored.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Debugger entered--Lisp error: (wrong-type-argument char-or-string-p (nil nil))
>   nnheader-insert-nov([634 "Debian package updates on stoner.mydomain.se " "root <root@stoner.mydomain.se> " "Fri, 17 Sep 2010 21:52:25 +0100 " "<E1Owhuz-0002st-DO@stoner.mydomain.se>" nil -1 43 nil ((Id nil nil) (Ip nil) (Queue))])

I've now gotten nnimap + Agent to a state where I can download group and
visit the articles and stuff.  There were some mighty obscure bugs deep
in the Agent code.  Mighty deep and frustrating.

Anyway, I'm not sure what your bug means.  You seem to have an entry for
mail-header-extra containing `((Id nil nil) (Ip nil) (Queue))', which I
don't think is altogether valid, which is why it bugs out.

I can't find the string "Queue" in the source.  Do you know where these
come from?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Steinar Bang <sb@dod.no> writes:
>
>> Do you know that the text/plain part is always the first part
>> of a message?  Wouldn't it be better to just pull down all text/plain
>> parts of a message?
>
> Hm.  Yes... when requesting the headers on group entry, I can also ask for
> the article structure.  So when it comes time to request the article, it
> should be easy enough to ask for all the text/plain parts.  Or perhaps all
> text/* parts?  Well, that can be user-configurable.

Generally, I would expect all you to fetch all the structural (multipart/*)
and inline (*/*) parts, other than the ones that were not going to be
displayed because they were the lower priority part of a multipart/alternative
chunk of main.  OTOH ...

>> But whatever way you do it, partial downloading should cooperate nicely
>> with the agent.
>
> If we have a "fetch complete article" message, it could just bind the agent
> thing to nil and then request it straight from the IMAP server?  Then things
> should Just Work.  That is, the partial articles would be downloaded by the
> agent and stored.

... I very strongly favour this being an "implementation detail" that I have
absolutely no visibility of as an end user.  (Well, save for offline operation
or whatever.)

Way back when I vaguely thought about this I envisioned rewriting the MIME
bits so the interface between the backend and the rest of Gnus was based on
fetching the MIME structure, then fetching individual parts while rendering.

That would allow the IMAP implementation to delegate that to the server, but
the NNTP (etc) implementation to use a common function that basically emulated
the same stuff on top of the entire article.

        Daniel

I never wrote working code, though, so I don't know it counts.
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Partial article download

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Anyway, I'm not sure what your bug means.  You seem to have an entry for
> mail-header-extra containing `((Id nil nil) (Ip nil) (Queue))', which I
> don't think is altogether valid, which is why it bugs out.

> I can't find the string "Queue" in the source.  Do you know where
> these come from?

Grepping through my home directory, I have the string "Queue" as part of
some agentized nntp articles.  In most of them, the match is on the form
of an X-Queue header (whatever that is).

But some look interesting:
./News/agent/nntp/news.gmane.org/gmane/emacs/gnus/general/70938:90:nil nil) (Ip nil) (Queue))])
./News/agent/nntp/news.gmane.org/gmane/emacs/gnus/general/71037:79:  nnheader-insert-nov([634 "Debian package updates on stoner.mydomain.se " "root <root@stoner.mydomain.se> " "Fri, 17 Sep 2010 21:52:25 +0100 " "<E1Owhuz-0002st-DO@stoner.mydomain.se>" nil -1 43 nil ((Id nil nil) (Ip nil) (Queue))])
...
./News/agent/nntp/news.gmane.org/gmane/emacs/gnus/general/71049:87:>   nnheader-insert-nov([634 "Debian package updates on stoner.mydomain.se " "root <root@stoner.mydomain.se> " "Fri, 17 Sep 2010 21:52:25 +0100 " "<E1Owhuz-0002st-DO@stoner.mydomain.se>" nil -1 43 nil ((Id nil nil) (Ip nil) (Queue))])
./News/agent/nntp/news.gmane.org/gmane/emacs/gnus/general/71049:94:mail-header-extra containing `((Id nil nil) (Ip nil) (Queue))', which I

But nothing agentized for nnimap contains this string.

Re: Partial article download

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> Generally, I would expect all you to fetch all the structural (multipart/*)
> and inline (*/*) parts, other than the ones that were not going to be
> displayed because they were the lower priority part of a multipart/alternative
> chunk of main.

That sounds like work.  :-)

My guess is that this feature is only interesting for people with low
bandwidth (for instance, on a phone).  So even if they're getting
text/html rendered, they don't want the images that would normally be
displayed in the article (referred to with cid:).

So I think "just download text/*" is good enough.

>> If we have a "fetch complete article" message, it could just bind the agent
>> thing to nil and then request it straight from the IMAP server?  Then things
>> should Just Work.  That is, the partial articles would be downloaded by the
>> agent and stored.
>
> ... I very strongly favour this being an "implementation detail" that I have
> absolutely no visibility of as an end user.  (Well, save for offline operation
> or whatever.)

Well, it'll be a new command, so it'll be visible.

> Way back when I vaguely thought about this I envisioned rewriting the MIME
> bits so the interface between the backend and the rest of Gnus was based on
> fetching the MIME structure, then fetching individual parts while rendering.
>
> That would allow the IMAP implementation to delegate that to the server, but
> the NNTP (etc) implementation to use a common function that basically emulated
> the same stuff on top of the entire article.

I'm not sure how useful that would be.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Grepping through my home directory, I have the string "Queue" as part of
> some agentized nntp articles.  In most of them, the match is on the form
> of an X-Queue header (whatever that is).

The `Queue' thing sounds really vaguely familiar to me, but I can't
place it...

Anyway, I'll just fix the function to not bug out in the presence of
these things.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Anyway, I'll just fix the function to not bug out in the presence of
> these things.

Seems to be working.  I can enter agentized nnimap groups when online,
and I can enter agentized nnimap groups when offline, and I see the
articles I read when online... I have this setting in my ~/.emacs:

;; Store article in agent when reading it while online
(add-hook 'gnus-select-article-hook 'gnus-agent-fetch-selected-article)

(which is a non-standard setting that also worked with nnimap+agent)

And it seems to be faster.  Especially `g', and initial startup.  I ran
the pre-new-nnimap version of gnus a little this morning, and it uses
forever to start up compared to the new one.

Re: Partial article download

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

>> The "fetch complete article" method sounds way easier to implement.

> Yes it does, doesn't it?

It will break for some: dude with a low bandwidt connection talking to
his employer's Exchange server, and receiving an email with a text part,
a small attachment that he wants, and a 500MB PowerPoint that he doesn't
want.

Re: Partial article download

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> It will break for some: dude with a low bandwidt connection talking to
> his employer's Exchange server, and receiving an email with a text part,
> a small attachment that he wants, and a 500MB PowerPoint that he doesn't
> want.

It would be nicer (and more elegant) to be able to fetch stuff bit by
bit, but I'm not sure it's worth the effort.

I think I'll just implement the "fetch the entire thing" and partial
downloads, and we can see later whether fetching single bits is a
something in great demand.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Partial article download

[Lars Magne Ingebrigtsen]

I've started looking at implementing partial downloads.  It's way more
fiddly than I thought -- I had imagined that you'd just say what parts
you wanted, and then you'd get a complete article back, only missing the
bits you didn't request.

Instead, it looks like you have to recreate the message from the
BODYSTRUCTURE and the parts you fetch.

I'll fiddle a bit more...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen