Greg's Perennial Plea For an Asynchronous Backend Interface

Greg's Perennial Plea For an Asynchronous Backend Interface

[Greg Stark]

So I go through this about every other year or so, I think this is my third
time. Periodically I get really frustrated that some gnus has tied up my emacs
doing some long network operation like downloading a few thousand news
headers, or expunging an imap mailbox, etc. For someone who lives entirely
within Emacs such things can be terribly frustrating. And it's completely
unnecessary.

No you don't need a threaded Emacs. 

Imagine if could pull down the complete summary of some huge newsgroup to do
searches while editing your files and composing replies in another group.

No you do *not* need a threaded Emacs. Really.

Imagine if when starting Gnus it initiated the check of the active file of all
20 slow news servers simultaneously and let you read from your main server
until they all respond or time out five minutes later.

No you do *not* need a threaded Emacs. Go away.

All you need is an interface between Gnus and the backends that allows Gnus to
pass a callback with a closure to the backend when an asynchronous call is
permitted. And of course you want to do this in a backwards compatible way, so
if the backend doesn't implement the asynchronous call Gnus just calls the
normal backend function. 

The backend of course needs to be written cleverly to make this possible. The
good news is that many of them already are. Because it's the most memory
efficient way to code process handling code in modern Emacsen many of the
backends implement their process handling code using after-change-hooks hooks
on the process buffer. These hooks get called asynchronously when the response
comes in, all they have to do is arrange to call Gnus' callback.

If anyone still doesn't believe me that these things do not require a threaded
Emacs consider this: I've already done it, back in 1996 when I originally
brought this up. 

My implementation uses a short cut(*) that wouldn't make sense in more than one
backend at a time. But the concept is proven. Emacs is perfectly responsive
while doing a large slow network operation. I can edit, compose, do all the
million other things I do within Emacs every day without getting blocked by
Gnus.

Doing it properly requires the new interface between backends and Gnus. And it
require Gnus be aware of when it's using the asynchronous interface to avoid
conflicts. But it would be much more powerful than my kludge.

(*) In case anyone's curious, the magic words are: recursive-edit :)

-- 
greg

Re: Greg's Perennial Plea For an Asynchronous Backend Interface

[Kai Großjohann]

I like your idea.  Alas, I have neither time nor expertise to tackle
this.  :-|

My thesis is more important, right now...

kai
-- 
~/.signature

Re: Greg's Perennial Plea For an Asynchronous Backend Interface

[Lloyd Zusman]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> I like your idea.  Alas, I have neither time nor expertise to tackle
> this.  :-|

I know that "me too" posts are evil on usenet, but in this case, it's
an evil that's well worth it, because your suggestion is excellent,
and I'd like to add my vote for this long overdue feature.  And so ...

Me too!


> My thesis is more important, right now...

As for me, I don't have tons of spare time, but I might be able to
make some on-again-off-again contributions of programming time towards
this cause.

Would you (Greg) be willing to post your hack?


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: Greg's Perennial Plea For an Asynchronous Backend Interface

[Steinar Bang]

>>>>> Greg Stark <gsstark@mit.edu>:

> All you need is an interface between Gnus and the backends that
> allows Gnus to pass a callback with a closure to the backend when an
> asynchronous call is permitted. And of course you want to do this in
> a backwards compatible way, so if the backend doesn't implement the
> asynchronous call Gnus just calls the normal backend function.

[snip!]
> Doing it properly requires the new interface between backends and
> Gnus. And it require Gnus be aware of when it's using the
> asynchronous interface to avoid conflicts. But it would be much more
> powerful than my kludge.

Is this basically the same thing that nnimap needs for its performance 
enhancements?

Re: Greg's Perennial Plea For an Asynchronous Backend Interface

[Simon Josefsson]

Steinar Bang <sb@metis.no> writes:

> > Doing it properly requires the new interface between backends and
> > Gnus. And it require Gnus be aware of when it's using the
> > asynchronous interface to avoid conflicts. But it would be much more
> > powerful than my kludge.
> 
> Is this basically the same thing that nnimap needs for its performance 
> enhancements?

Sort of, primarily nnimap need a more fine-grained backend interface
to enable Gnus to take advantage of IMAP features, and that would
speed things up considerably.

An asynchronous backend interface would probably make things "feel"
fast enough, though, because it wouldn't lock up emacs while the
backends "thinks".

If the more fine-grained backend interface happen (which is lots of
work, I think), it might as well be asynchronous too.

Re: Greg's Perennial Plea For an Asynchronous Backend Interface

[Steinar Bang]

>>>>> Simon Josefsson <sj@extundo.com>:

> Sort of, primarily nnimap need a more fine-grained backend interface
> to enable Gnus to take advantage of IMAP features, and that would
> speed things up considerably.

> An asynchronous backend interface would probably make things "feel"
> fast enough, though, because it wouldn't lock up emacs while the
> backends "thinks".

> If the more fine-grained backend interface happen (which is lots of
> work, I think), it might as well be asynchronous too.

I think it would be well worth the work.  I'm using Gnus agent with
nnimap, mainly because using it online is too slow.