Re: Multithreading support in pluggable module, but with problems

[Sebastian Gniazdowski <psprint2@fastmail.com>]

On Sat, Feb 11, 2017, at 11:46 AM, Bart Schaefer wrote:
> So if I understand this correctly, you're attempting to use POSIX threads
> to share the my_hash variable between the main shell and a 2nd thread of
> which the main shell is not aware, so that the 2nd thread can stuff a
> value into my_hash while the main thread goes on about its business?

Yes. 2nd thread has it's forked pipe data provider – "zpin" command that
does eval – and can eval freely because it's forked – Zsh forks first
components of pipeline, last one isn't forked – that's the tight window
for this solution.

> I can almost promise you there is no way to make that work, at least not
> in a crash-proof manner.  Even making my_hash PM_READONLY -- which is I
> presume what yesterday's question was about -- what is there to stop
> the thread-unaware main shell from accessing it at a time when the 2nd
> thread is changing it in some way that will send the main thread off
> into uncharted memory?

I know it might seem dirty, but what if it's rock-solid done. Also, for
me it might be interesting, for someone outside it might be repelling –
I think I can tell from my own reactions to creativity of others,
plugins created by others – sadly I'm miles away from trying them. But
maybe they're just not interesting, it's possible.

I think I got code close to working state. Can consecutively spawn 32
threads one hundred times. Have functions without queue_signals() etc.
copied from Zsh. What is to stop conflicts are helper variables. Got two
of them casually added, readonly:

– $workers_count – holds number of active threads. User is to wait for
0.
– $worker_finished – array of 32 ones. If a thread is active, there is
"0" for him.

If in syntax highlighting I would spawn "check_path()" with worker
number "X", output to variable "output_X", then I would check for
$worker_finished[X] to become "1" at main loop one or few times, join
the thread (TODO), use the variable. This might seem complex and maybe
it is, I will verify.

> In particular you *might* need to call either movefd() or redup()
> rather than call dup2() directly, and you should otherwise be using
> addmodulefd() to protect the fd from exec.c:closeallelse().  Examples
> in Src/Modules/tcp.c.

Good to know, thanks. I might reserve 32 file descriptors.

Also I think I found some very real proof. In my message I wrote I
guarded fclose() with fcntl() that is checking if FD is valid:

            if ( -1 != fcntl( fileno( oconf->stream ), F_GETFD ) ) {
                if ( 0 != fclose( oconf->stream ) ) {

This summoned away almost all FD problems, literaly 99.5%. Once in a
while fclose() would however fail. I thought: "It's a race condition,
between fcntl() and fclose() something closes the FD". So I modified:

            if ( -1 != fcntl( fileno( oconf->stream ), F_GETFD ) ) {
                sleep( 1 );
                if ( 0 != fclose( oconf->stream ) ) {

And now I get many "Bad descriptor" errors each time. So something is
really managing the FD behind the curtains, either is like I originally
suspected that closing on one end invalidates on other (there are
problems in this thinking..), or it's Zsh that does the management,
which might be a good thing. It seems I can just forget about fclose(),
it's not needed apparently. Will check for any memory leak tomorrow to
be sure.

-- 
Sebastian Gniazdowski