From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 1259 invoked by alias); 11 Feb 2017 20:14:37 -0000 Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Workers List List-Post: List-Help: X-Seq: 40529 Received: (qmail 4362 invoked from network); 11 Feb 2017 20:14:37 -0000 X-Qmail-Scanner-Diagnostics: from new1-smtp.messagingengine.com by f.primenet.com.au (envelope-from , uid 7791) with qmail-scanner-2.11 (clamdscan: 0.99.2/21882. spamassassin: 3.4.1. Clear:RC:0(66.111.4.221):SA:0(-0.4/5.0):. Processed in 0.930449 secs); 11 Feb 2017 20:14:37 -0000 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-0.4 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.1 X-Envelope-From: psprint2@fastmail.com X-Qmail-Scanner-Mime-Attachments: | X-Qmail-Scanner-Zip-Files: | Received-SPF: pass (ns1.primenet.com.au: SPF record at spf.messagingengine.com designates 66.111.4.221 as permitted sender) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=fastmail.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=mesmtp; bh=CZHa6TZviybnFBTnQYj4etlHDx U=; b=OQ/8vZzxvDqHTbruqY7VzJNj70E8t0A7TeJBrl86WreCxDhGNydXk35P6t QzSipEJ3EqTRQs8xdtDQe++CbBput3wtjffWmjTPFjoq9CKIQPNxGOAzARuPliCP EGK3j0na42aj3tBNVfnth+43EGy7OI6O3k4KHNQcPfhoBXb1s= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=smtpout; bh=CZ Ha6TZviybnFBTnQYj4etlHDxU=; b=Fe5w0wwVVtEfSGe9aMF3D3lc0wbZo4/rjA 8CGHEQYQCxTP7U9EaiRjxteAQtFG49NBzYDoNliIWfT5VYOpAwcRO8ULpKIsX/+b xUB6qj6KD3IuPUa4W54PN+s59MMDUpqtBO1vBm7FyiP1L6kcegvaEUYfdmf8Jztc 8rQo6G44o= X-ME-Sender: Message-Id: <1486844071.2825373.877945608.5B8686FF@webmail.messagingengine.com> From: Sebastian Gniazdowski To: zsh-workers@zsh.org Cc: Bart Schaefer MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-4a450d19 References: <1486817916.2745591.877718432.2CE18595@webmail.messagingengine.com> <170211114632.ZM665@torch.brasslantern.com> In-Reply-To: <170211114632.ZM665@torch.brasslantern.com> Subject: Re: Multithreading support in pluggable module, but with problems Date: Sat, 11 Feb 2017 12:14:31 -0800 On Sat, Feb 11, 2017, at 11:46 AM, Bart Schaefer wrote: > So if I understand this correctly, you're attempting to use POSIX threads > to share the my_hash variable between the main shell and a 2nd thread of > which the main shell is not aware, so that the 2nd thread can stuff a > value into my_hash while the main thread goes on about its business? Yes. 2nd thread has it's forked pipe data provider =E2=80=93 "zpin" command= that does eval =E2=80=93 and can eval freely because it's forked =E2=80=93=C2=A0= Zsh forks first components of pipeline, last one isn't forked =E2=80=93=C2=A0that's the tig= ht window for this solution. > I can almost promise you there is no way to make that work, at least not > in a crash-proof manner. Even making my_hash PM_READONLY -- which is I > presume what yesterday's question was about -- what is there to stop > the thread-unaware main shell from accessing it at a time when the 2nd > thread is changing it in some way that will send the main thread off > into uncharted memory? I know it might seem dirty, but what if it's rock-solid done. Also, for me it might be interesting, for someone outside it might be repelling =E2= =80=93 I think I can tell from my own reactions to creativity of others, plugins created by others =E2=80=93=C2=A0sadly I'm miles away from trying t= hem. But maybe they're just not interesting, it's possible. I think I got code close to working state. Can consecutively spawn 32 threads one hundred times. Have functions without queue_signals() etc. copied from Zsh. What is to stop conflicts are helper variables. Got two of them casually added, readonly: =E2=80=93 $workers_count =E2=80=93 holds number of active threads. User is = to wait for 0. =E2=80=93 $worker_finished =E2=80=93 array of 32 ones. If a thread is activ= e, there is "0" for him. If in syntax highlighting I would spawn "check_path()" with worker number "X", output to variable "output_X", then I would check for $worker_finished[X] to become "1" at main loop one or few times, join the thread (TODO), use the variable. This might seem complex and maybe it is, I will verify. > In particular you *might* need to call either movefd() or redup() > rather than call dup2() directly, and you should otherwise be using > addmodulefd() to protect the fd from exec.c:closeallelse(). Examples > in Src/Modules/tcp.c. Good to know, thanks. I might reserve 32 file descriptors. Also I think I found some very real proof. In my message I wrote I guarded fclose() with fcntl() that is checking if FD is valid: if ( -1 !=3D fcntl( fileno( oconf->stream ), F_GETFD ) ) { if ( 0 !=3D fclose( oconf->stream ) ) { This summoned away almost all FD problems, literaly 99.5%. Once in a while fclose() would however fail. I thought: "It's a race condition, between fcntl() and fclose() something closes the FD". So I modified: if ( -1 !=3D fcntl( fileno( oconf->stream ), F_GETFD ) ) { sleep( 1 ); if ( 0 !=3D fclose( oconf->stream ) ) { And now I get many "Bad descriptor" errors each time. So something is really managing the FD behind the curtains, either is like I originally suspected that closing on one end invalidates on other (there are problems in this thinking..), or it's Zsh that does the management, which might be a good thing. It seems I can just forget about fclose(), it's not needed apparently. Will check for any memory leak tomorrow to be sure. --=20 Sebastian Gniazdowski