[Caml-list] Bug when printing from signal handlers?

[Caml-list] Bug when printing from signal handlers?

[Anders Peter Fugmann]

Hi,

I'm experiencing a live lock while printing from a signal handler.
It seems to be triggered when the main is interrupted while printing.

The following program exposes the problem:
(tested on 4.00.1 and 4.01.0)

let rec loop () =
   print_endline "In main thread";
   loop ()

let _ =
   Sys.(set_signal sigalrm
          (Signal_handle
             (fun _ -> print_endline "In signal handler")));
   let _ = Unix.(setitimer ITIMER_REAL
                   { it_interval = 0.1; it_value = 0.1 })
   in
   loop ()

For some reason, in caml_flush_parial channel->buff becomes greater than 
channel->curr which means caml_flush never completes, as 
caml_flush_partial never returns true.

Is printing from a signal handler illegal?
I tries searching for bug on this matter but only found examples that 
printed from signal handlers.

Regards
Anders

Re: [Caml-list] Bug when printing from signal handlers?

[Jesper Louis Andersen]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

On Wed, Mar 19, 2014 at 10:21 PM, Anders Peter Fugmann
<anders@fugmann.net>wrote:

> Is printing from a signal handler illegal?


FWIW, that is illegal in some operating systems. I have seen it in one case
at least:

* Linux with threads. printf in C is protected by a mutex. Suppose we are
inside the printf, with the mutex locked and then we get a signal. We run
the handler, and it wants to print out debug information. Welcome deadlock!
* Many system calls are not safe in the signal handler. FreeBSD has a list
in sigaction(2) for instance. See


http://www.freebsd.org/cgi/man.cgi?query=sigaction&apropos=0&sektion=0&manpath=FreeBSD+10.0-RELEASE&arch=default&format=html

In practice, you should probably just set a flag in the signal handler and
then handle that in a main loop of the program. Doing advanced stuff in a
signal handler is usually dangerous non-compatible territory.


-- 
J.

[-- Attachment #2: Type: text/html, Size: 1711 bytes --]

Re: [Caml-list] Bug when printing from signal handlers?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1994 bytes --]

OCaml printing isn't printf in C - it calls directly write() and is
always possible. OCaml signal handlers aren't signal handlers from the C
viewpoint: When the signal is caught, a flag in the OCaml runtime is
set, and this flag is regularly checked by the running code. (I.e. what
you suggest is already done in the runtime.)

So, I'd say this is a bug in the OCaml runtime. The bug goes away when
you print to a different channel from the signal handler, so it looks
like channels and signal handlers have some unwanted effect on each
other.

Gerd

Am Mittwoch, den 19.03.2014, 23:19 +0100 schrieb Jesper Louis Andersen:
> 
> On Wed, Mar 19, 2014 at 10:21 PM, Anders Peter Fugmann
> <anders@fugmann.net> wrote:
>         Is printing from a signal handler illegal?
> 
> FWIW, that is illegal in some operating systems. I have seen it in one
> case at least:
> 
> 
> * Linux with threads. printf in C is protected by a mutex. Suppose we
> are inside the printf, with the mutex locked and then we get a signal.
> We run the handler, and it wants to print out debug information.
> Welcome deadlock!
> * Many system calls are not safe in the signal handler. FreeBSD has a
> list in sigaction(2) for instance. See
> 
> 
> 
>  http://www.freebsd.org/cgi/man.cgi?query=sigaction&apropos=0&sektion=0&manpath=FreeBSD+10.0-RELEASE&arch=default&format=html
> 
> 
> In practice, you should probably just set a flag in the signal handler
> and then handle that in a main loop of the program. Doing advanced
> stuff in a signal handler is usually dangerous non-compatible
> territory.
> 
> 
> 
> -- 
> J.

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] Bug when printing from signal handlers?

[ygrek]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

On Thu, 20 Mar 2014 00:21:36 +0100
Gerd Stolpmann <info@gerd-stolpmann.de> wrote:

> OCaml printing isn't printf in C - it calls directly write() and is
> always possible. OCaml signal handlers aren't signal handlers from the C
> viewpoint: When the signal is caught, a flag in the OCaml runtime is
> set, and this flag is regularly checked by the running code. (I.e. what
> you suggest is already done in the runtime.)
> 
> So, I'd say this is a bug in the OCaml runtime. The bug goes away when
> you print to a different channel from the signal handler, so it looks
> like channels and signal handlers have some unwanted effect on each
> other.

stdlib channels are protected with non-recursive mutex, so the deadlock on re-entrant invocation is guaranteed. 
AFAICS runtime system tries to execute signal immediately (see signal_handle in asmrun/signals_asm.c)
and if that is not possible - records signal for later execution.
Anyway doing complex stuff in signal handler is a bad idea, because even with delayed processing (when
things are safe from the libc point of view) the points of invocation of OCaml signal handler are scattered
all around the program (allocation sites) and any OCaml resource that doesn't support reentrant usage will break
the program.

-- 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Caml-list] Bug when printing from signal handlers?

[Anders Fugmann]

On 03/20/2014 06:29 AM, ygrek wrote:
>
> stdlib channels are protected with non-recursive mutex, so the deadlock on re-entrant invocation is guaranteed.
I also assumed when I first saw the behavior that it was some deadlock. 
But it seems that that is not the case as the system enters a live lock 
and just sits there burning off cpu time.

> AFAICS runtime system tries to execute signal immediately (see signal_handle in asmrun/signals_asm.c)
> and if that is not possible - records signal for later execution.
> Anyway doing complex stuff in signal handler is a bad idea, because even with delayed processing (when
> things are safe from the libc point of view) the points of invocation of OCaml signal handler are scattered
> all around the program (allocation sites) and any OCaml resource that doesn't support reentrant usage will break
> the program.

Maybe the documentation should be updated to explain how signal handling 
works and what is allowed in signal handers. Possibly also ideas on how 
to cope with these limitations.

I circumvented the problem by using unbuffered writes (Unix.write), but 
it makes me wonder if I have code erroneous code lying around.

/Anders

Re: [Caml-list] Bug when printing from signal handlers?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 3028 bytes --]

Am Donnerstag, den 20.03.2014, 13:29 +0800 schrieb ygrek:
> On Thu, 20 Mar 2014 00:21:36 +0100
> Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> 
> > OCaml printing isn't printf in C - it calls directly write() and is
> > always possible. OCaml signal handlers aren't signal handlers from the C
> > viewpoint: When the signal is caught, a flag in the OCaml runtime is
> > set, and this flag is regularly checked by the running code. (I.e. what
> > you suggest is already done in the runtime.)
> > 
> > So, I'd say this is a bug in the OCaml runtime. The bug goes away when
> > you print to a different channel from the signal handler, so it looks
> > like channels and signal handlers have some unwanted effect on each
> > other.
> 
> stdlib channels are protected with non-recursive mutex, so the deadlock on re-entrant invocation is guaranteed. 

No, there aren't any mutexes here involved - the failing program is
single-threaded.

> AFAICS runtime system tries to execute signal immediately (see signal_handle in asmrun/signals_asm.c)
> and if that is not possible - records signal for later execution.

Right, this handles the case that the current thread is doing a blocking
system call (sorry, forgot this case). Apparently, the OCaml code is
then run directly from the signal handler.

> Anyway doing complex stuff in signal handler is a bad idea, because even with delayed processing (when
> things are safe from the libc point of view) the points of invocation of OCaml signal handler are scattered
> all around the program (allocation sites) and any OCaml resource that doesn't support reentrant usage will break
> the program.

Anyway, I don't think this has anything to do with calling
non-signal-safe libc functions (so far I can see the only called
function is write()). It is most likely because flush isn't reentrant:

caml_flush() calls caml_flush_partial() which in turn calls do_write().
The signal arrives during the write() syscall, and in the signal handler
another flush is invoked, for the same channel. The effect is that (so
far I see it) channel->offset and channel->curr are set to illegal
values.

I don't see how this can be fixed properly. You probably can avoid the
livelock by doing nothing when flush is invoked for the second time, but
(a) this changes the semantics of flushing, and (b) doesn't fix the
other potential problems (when a flush is interrupted and one of the
other channel functions is called from the signal handler).

Nevertheless it's a bit surprising that innocent-looking OCaml code
turns out as unsafe. The current state is a bit unsatisfactory, at
least.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] Bug when printing from signal handlers?

[ygrek]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]


On Thu, 20 Mar 2014 08:42:47 +0100
Anders Fugmann <anders@fugmann.net> wrote:

> I circumvented the problem by using unbuffered writes (Unix.write), but 
> it makes me wonder if I have code erroneous code lying around.

I personally prefer to use signalfd for non-trivial signal handling, which means that
handler is invoked with specific and controlled program state.


On Thu, 20 Mar 2014 21:07:22 +0100
Gerd Stolpmann <info@gerd-stolpmann.de> wrote:

> No, there aren't any mutexes here involved - the failing program is
> single-threaded.

Indeed, the program is compiled with single-threaded runtime, so that's an additional failure scenario.

> Nevertheless it's a bit surprising that innocent-looking OCaml code
> turns out as unsafe. The current state is a bit unsatisfactory, at
> least.

Agreed, that's the point - OCaml signal handler though safe on the libc side, is nevertheless
subject to asynchronous execution at arbitrary program spots and, unless the user code
and runtime are fully reentrant, various unexpected program failures are inevitable..  

-- 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]