Re: is there a possibility for gnus to download data without blocking?

[Wayne Harris <wharris1@protonmail.com>]

Wayne Harris <wharris1@protonmail.com> writes:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

>> Let's see how Dick is handling it! :)
>
> Yes, let's do that.  I bet it won't be easy for me, but I'll give it a
> try after dinner today, that is, within 6 hours.  Expect my report.

A quick attempt at compiling it will likely not work because I'm
running a Windows system without a resourceful toolbox for all kinds
of software.

--8<---------------cut here---------------start------------->8---
%./autogen.sh
Checking whether you have the necessary tools...
(Read INSTALL.REPO for more details on building Emacs)
Checking for autoconf (need at least version 2.65) ... missing
[...]
--8<---------------cut here---------------end--------------->8---

Of course.  Won't even install such tools because I'm sure there's
many more needed.  Let's read the source code without running it.  We
surely should get to interesting places if we start at Lisp's function
gnus-group-list-active or perhaps gnus-group-get-new-news.  Indeed,
we find docstring

--8<---------------cut here---------------start------------->8---
(defun gnus-group-get-new-news (&optional arg one-level background)
  "Get newly arrived articles.
[...]
If BACKGROUND then run `gnus-get-unread-articles' in a separate thread.
--8<---------------cut here---------------end--------------->8---

GNU Emacs 24 has no such BACKGROUND option.  We can see in dickmao's
source code that there are changes to gnus-get-unread-articles too
because it's also taking the BACKGROUND argument, just like the
docstring suggests.  That seems to be the only change dickmao added.
So let's read gnus-get-unread-articles.

It's a long procedure.  As a quick side note, reading dickmao's
gnus-start.el we learn that the variable

  gnus-threaded-get-unread-articles

is required for the threaded modus operandi, just like his README
discreetly mentioned with 
 
  (custom-set-variables '(gnus-threaded-get-unread-articles t)).

The docstring for the variable confirms with

--8<---------------cut here---------------start------------->8---
  "Instantiate parallel threads for `gnus-get-unread-articles' which encapsulates
most of the network retrieval when `gnus-group-get-new-news' is run."
--8<---------------cut here---------------end--------------->8---

and denounces dickmao's line with 81 characters, not counting the
newline!  He's a longliner and we'll use that against him in court for
sure.  ``We're gonna nail his ass.''

The new function has the following signature:

--8<---------------cut here---------------start------------->8---
(cl-defun gnus-get-unread-articles (&optional requested-level dont-connect
                                              one-level background
                                    &aux (level (gnus-group-default-level
                                                 requested-level t)))
  "Go through `gnus-newsrc-alist' and compare with `gnus-active-hashtb'
  and compute how many unread articles there are in each group."
[...]
--8<---------------cut here---------------end--------------->8---

Dickmao forked GNU Emacs 27.0.50.  It will be useful to compare his code
against a version that's close such as emacs-27.0.90.  In any case, I am
not able to understand much.  

I believe he builds a list of THINGS to do in a list called COMMANDS.
These THINGS are procedures he builds with applications of
apply-partially.  If BACKGROUND was asked, then all THINGS to do, which
are stored in COMMANDS, will suffer the following action:

   (apply #'gnus-run-thread
          gnus-mutex-get-unread-articles
          thread-group
          commands)

At this point I advance that each server will be handled by a separate
thread and each thread will all the THINGS contained in COMMANDS in the
order they are.  That's why he uses gnus-push-end.  He's ordering each
task of each thread.  What happens in parallel, therefore, is just the
conversation with each separate server.

Once all threads are done, a procedure of name CODA will be run.  If
BACKGROUND is not true, then there is no thread for anything and CODA
will do it all.  CODA is not a procedure defined in gnus-start.el and
it's not clear what it is or what it does.

What each thread does sequentially is

  gnus-open-server
  gnus-retrieve-group-data-early
  gnus-read-active-for-groups

et cetera.  A buffer called

  *gnus-thread <THREAD-NAME>*

is created for each thread, so it's likely these buffers collect the
data produced by each server.  Now we would have to see how the data is
used and Gnus buffers are updated.

At this point, we should compile and run the software to confirm the
theory exposed here.

The question was how does dickmao does it.  In summary, he added
procedures like make-thread and now he's just using it. :-) But I think
the real thing you wanted to know was how he would handle, for example,
the situation the intermixing of different servers returning data and
perhaps not finishing and all the complications of network talk and also
updating the buffers.  I would expect that when one server delivers all
the information stored in likely buffer named

  *gnus-thread <THREAD-NAME>*

then dickmao updates Gnus buffers with that information.  If a server
does not produce sensible information, that buffer is discarded and it's
as if we never spoke to that server at all.  The updating of Gnus
buffers would not happen in parallel, of course.  It's only the talk to
the network that does.

One question comes up now.  Dickmao seems to have parallelized the
network conversations, which is perhaps possible to do with external
processes too.  For instance, spawn an external process for each server,
collect up all the data in a separate buffer.  Same strategy as
dickmao's, but using processes, not threads.  

Perhaps Gnus currently does things sequentially and blocks on the
network talk?  If so, it would be way better if it just wouldn't block.
Maybe we don't need dickmao's threads after all.  But, yes, we're back
to speculation.  I guess we like it. :-)