From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, FREEMAIL_FROM,RCVD_IN_DNSWL_MED autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 7421 invoked from network); 22 Aug 2020 02:29:36 -0000 Received: from lists1.math.uh.edu (129.7.128.208) by inbox.vuxu.org with ESMTPUTF8; 22 Aug 2020 02:29:36 -0000 Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.94) (envelope-from ) id 1k9JH5-008z5c-Px; Fri, 21 Aug 2020 21:28:55 -0500 Received: from mx2.math.uh.edu ([129.7.128.33]) by lists1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1k9JH0-008z3e-Gc for ding@lists.math.uh.edu; Fri, 21 Aug 2020 21:28:50 -0500 Received: from quimby.gnus.org ([95.216.78.240]) by mx2.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1k9JGx-00FQFB-7V for ding@lists.math.uh.edu; Fri, 21 Aug 2020 21:28:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:Mime-Version:References:Message-ID:Date:Subject: From:To:Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=ndgLR63s052xVYF+4ZJ6Uw769+7Dl8Zdzfkq/5vLhQM=; b=ekkA7KB9iw3njG5Nsbs6JtxC8o CK1cI8A+0WIROB8if5mHGU2iO7hEoMcHs665NZ1Khjxhu2LOZVRI9GEWrjX4OO63/6j3NAbiI5dA0 xhhDo6KaoWmInasQHnihN2RTf/lEbp7BNYTAmL5nwRjnTwDK9uQgGSF0j13kN2OTz/EM=; Received: from static.214.254.202.116.clients.your-server.de ([116.202.254.214] helo=ciao.gmane.io) by quimby with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1k9JGi-0001Wp-MP for ding@gnus.org; Sat, 22 Aug 2020 04:28:42 +0200 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1k9JGh-00040P-LF for ding@gnus.org; Sat, 22 Aug 2020 04:28:31 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: ding@gnus.org From: Wayne Harris Subject: Re: is there a possibility for gnus to download data without blocking? Date: Fri, 21 Aug 2020 23:27:39 -0300 Message-ID: <86o8n3bejo.fsf@protonmail.com> References: <86y2m8hccw.fsf@protonmail.com> <87imdcvchr.fsf@ericabrahamsen.net> <868se8gjzj.fsf@protonmail.com> <875z9bvs13.fsf@ericabrahamsen.net> <86eenzeo8f.fsf@protonmail.com> Mime-Version: 1.0 Content-Type: text/plain Cancel-Lock: sha1:McbmGtQw4qD949hCOaaVAfTEGW4= List-ID: Precedence: bulk Wayne Harris writes: > Eric Abrahamsen writes: [...] >> Let's see how Dick is handling it! :) > > Yes, let's do that. I bet it won't be easy for me, but I'll give it a > try after dinner today, that is, within 6 hours. Expect my report. A quick attempt at compiling it will likely not work because I'm running a Windows system without a resourceful toolbox for all kinds of software. --8<---------------cut here---------------start------------->8--- %./autogen.sh Checking whether you have the necessary tools... (Read INSTALL.REPO for more details on building Emacs) Checking for autoconf (need at least version 2.65) ... missing [...] --8<---------------cut here---------------end--------------->8--- Of course. Won't even install such tools because I'm sure there's many more needed. Let's read the source code without running it. We surely should get to interesting places if we start at Lisp's function gnus-group-list-active or perhaps gnus-group-get-new-news. Indeed, we find docstring --8<---------------cut here---------------start------------->8--- (defun gnus-group-get-new-news (&optional arg one-level background) "Get newly arrived articles. [...] If BACKGROUND then run `gnus-get-unread-articles' in a separate thread. --8<---------------cut here---------------end--------------->8--- GNU Emacs 24 has no such BACKGROUND option. We can see in dickmao's source code that there are changes to gnus-get-unread-articles too because it's also taking the BACKGROUND argument, just like the docstring suggests. That seems to be the only change dickmao added. So let's read gnus-get-unread-articles. It's a long procedure. As a quick side note, reading dickmao's gnus-start.el we learn that the variable gnus-threaded-get-unread-articles is required for the threaded modus operandi, just like his README discreetly mentioned with (custom-set-variables '(gnus-threaded-get-unread-articles t)). The docstring for the variable confirms with --8<---------------cut here---------------start------------->8--- "Instantiate parallel threads for `gnus-get-unread-articles' which encapsulates most of the network retrieval when `gnus-group-get-new-news' is run." --8<---------------cut here---------------end--------------->8--- and denounces dickmao's line with 81 characters, not counting the newline! He's a longliner and we'll use that against him in court for sure. ``We're gonna nail his ass.'' The new function has the following signature: --8<---------------cut here---------------start------------->8--- (cl-defun gnus-get-unread-articles (&optional requested-level dont-connect one-level background &aux (level (gnus-group-default-level requested-level t))) "Go through `gnus-newsrc-alist' and compare with `gnus-active-hashtb' and compute how many unread articles there are in each group." [...] --8<---------------cut here---------------end--------------->8--- Dickmao forked GNU Emacs 27.0.50. It will be useful to compare his code against a version that's close such as emacs-27.0.90. In any case, I am not able to understand much. I believe he builds a list of THINGS to do in a list called COMMANDS. These THINGS are procedures he builds with applications of apply-partially. If BACKGROUND was asked, then all THINGS to do, which are stored in COMMANDS, will suffer the following action: (apply #'gnus-run-thread gnus-mutex-get-unread-articles thread-group commands) At this point I advance that each server will be handled by a separate thread and each thread will all the THINGS contained in COMMANDS in the order they are. That's why he uses gnus-push-end. He's ordering each task of each thread. What happens in parallel, therefore, is just the conversation with each separate server. Once all threads are done, a procedure of name CODA will be run. If BACKGROUND is not true, then there is no thread for anything and CODA will do it all. CODA is not a procedure defined in gnus-start.el and it's not clear what it is or what it does. What each thread does sequentially is gnus-open-server gnus-retrieve-group-data-early gnus-read-active-for-groups et cetera. A buffer called *gnus-thread * is created for each thread, so it's likely these buffers collect the data produced by each server. Now we would have to see how the data is used and Gnus buffers are updated. At this point, we should compile and run the software to confirm the theory exposed here. The question was how does dickmao does it. In summary, he added procedures like make-thread and now he's just using it. :-) But I think the real thing you wanted to know was how he would handle, for example, the situation the intermixing of different servers returning data and perhaps not finishing and all the complications of network talk and also updating the buffers. I would expect that when one server delivers all the information stored in likely buffer named *gnus-thread * then dickmao updates Gnus buffers with that information. If a server does not produce sensible information, that buffer is discarded and it's as if we never spoke to that server at all. The updating of Gnus buffers would not happen in parallel, of course. It's only the talk to the network that does. One question comes up now. Dickmao seems to have parallelized the network conversations, which is perhaps possible to do with external processes too. For instance, spawn an external process for each server, collect up all the data in a separate buffer. Same strategy as dickmao's, but using processes, not threads. Perhaps Gnus currently does things sequentially and blocks on the network talk? If so, it would be way better if it just wouldn't block. Maybe we don't need dickmao's threads after all. But, yes, we're back to speculation. I guess we like it. :-)