Loading Eprog in other thread doesn't work

Loading Eprog in other thread doesn't work

[Sebastian Gniazdowski]

Hello,
I do quite isolated thing:

    Eprog prog = pn->prog;
    execode(prog, 1, 0, "filecode");

pn (PrepareNode) is passed as (void*) to pthread_create. I use conditional variables to make "source_load" wait for background loading to finish.

But it looks like state of Zsh gets disrupted. Single run of source_prepare / source_load often works, the code is executed fine. However the longer I repeat this, the weirder things happen. Example backtraces at the end.

Does execode() expect some setup, and without it, it disrupts state? I've run Valgrind and get no memory errors. But the state can be disrupted logically? Is there alternative to run Eprog?

Full code:
https://github.com/zdharma/zplugin/blob/master/zmodules/Src/zdharma/zplugin.c

--  
Sebastian Gniazdowski
psprint /at/ zdharma.org



* thread #1: joinlists(first=0x000000010ecf0ef0, second=0x000000010ecf0e80) + 68 at linklist.c:366, stop reason = signal SIGSTOP
  * frame #0: joinlists(first=0x000000010ecf0ef0, second=0x000000010ecf0e80) + 68 at linklist.c:366
    frame #1: execcmd_exec(state=0x00007fff512539b0, eparams=0x00007fff51252b50, input=0, output=0, how=18, last1=2) + 4270 at exec.c:3028


* thread #1: libsystem_platform.dylib`_platform_strcmp + 19, stop reason = signal SIGSTOP
  * frame #0: libsystem_platform.dylib`_platform_strcmp + 19
    frame #1: execcmd_exec(state=0x00007fff5168f9b0, eparams=0x00007fff5168eb50, input=0, output=0, how=18, last1=2) + 6790 at exec.c:3189


* thread #1: countlinknodes(list=0x000000010e2b7f18) + 46 at linklist.c:309, stop reason = signal SIGSTOP
  * frame #0: countlinknodes(list=0x000000010e2b7f18) + 46 at linklist.c:309
    frame #1: execbuiltin(args=0x000000010e2b7f18, assigns=0x0000000000000000, bn=0x000000010e387220) + 239 at builtin.c:274

Re: Loading Eprog in other thread doesn't work

[Bart Schaefer]

On Mon, Jun 19, 2017 at 7:37 AM, Sebastian Gniazdowski
<psprint@zdharma.org> wrote:
>
> But it looks like state of Zsh gets disrupted. Single run of source_prepare / source_load often works, the code is executed fine. However the longer I repeat this, the weirder things happen. Example backtraces at the end.
>

Neither signal handling (which includes reaping child processes when
they exit) nor memory management is thread-safe, and there may be
shared memory issues with threads and traditional process forking as
well.  The more threads you try to have the more likely you're going
to be to wander into one of these areas.  I'd be very skeptical that
you can just plop threading into one part of zsh without at least some
serious semaphore work on the rest of it.

Re: Loading Eprog in other thread doesn't work

[Sebastian Gniazdowski]

PS. Found the reason of instability – the background-loading code did zhalloc. After replacing with zalloc things are stable. Although after each 1000 iterations of source_prepare / source_load shell slows down, so some pthread issue is apparently still there.

--  
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Loading Eprog in other thread doesn't work

[Bart Schaefer]

On Mon, Jun 19, 2017 at 9:37 PM, Sebastian Gniazdowski
<psprint@zdharma.org> wrote:
> PS. Found the reason of instability – the background-loading code did zhalloc. After replacing with zalloc things are stable.

Even zalloc() is only signal-safe, not really thread-safe.  I wouldn't
be surprised to find there's still some command you might run in the
main thread that messes this up.

Re: Loading Eprog in other thread doesn't work

[Sebastian Gniazdowski]

On 19.06.2017 at 22:12:53, Bart Schaefer (schaefer@brasslantern.com) wrote:
> Neither signal handling (which includes reaping child processes when
> they exit) nor memory management is thread-safe, and there may be
> shared memory issues with threads and traditional process forking as
> well. The more threads you try to have the more likely you're going
> to be to wander into one of these areas. I'd be very skeptical that
> you can just plop threading into one part of zsh without at least some
> serious semaphore work on the rest of it.

But the only thing that I do concurrently is loading of Eprog. So, open(), read(), that's all. When I invoke execode(), it is the main thread. "source_prepare" does the open, read, while "source_load" takes Eprog that is ready to use.

-- 
Sebastian Gniazdowski
psprint /at/ zdharma.org

Re: Loading Eprog in other thread doesn't work

[Bart Schaefer]

On Mon, Jun 19, 2017 at 7:42 PM, Sebastian Gniazdowski
<psprint@zdharma.org> wrote:
>
> But the only thing that I do concurrently is loading of Eprog.

It's not necessarily what that thread is doing, what may matter is
what any other thread is doing at the same time.