[Caml-list] ANN: exn-source - exception backtraces with source code printing

[Caml-list] ANN: exn-source - exception backtraces with source code printing

[John Whitington]

Hi,

OCaml 4.02 has new facilities for installing a new backtrace handler, 
and processing the back trace symbolically. As a little proof of 
concept, I've built a package which you can add to any project to get 
backtrace with source code printed and highlighted:

https://github.com/johnwhitington/exn-source

Here's the start of a backtrace:

http://www.coherentpdf.com/exnsource.png

By default, it prints five lines either side, and looks the current 
directory for source code. But as one can see from the screenshot, one 
can add /usr/local/lib/ocaml to get stdlib source and so on...

It's somewhat difficult to test this kind of thing (we can't use 
exceptions at all inside a backtrace handler, for example), so I'm 
appealing for help. Attach exn-source to your code and let me know how 
you get on...

Thanks,

-- 
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

That looks like a nice project ! Another trick I use personally is to run
the faulting program from inside Emacs's compile-mode (using "build && run"
instead of just "build" as a compilatoin command), and piggy-back on its
parsing of OCaml error locations to ask Emacs to drive me around the
relevant files.
I'm sure there are scenarios where your simultaneous view of all fragments
at once can be very helpful, and I'm also interested in the uses of the new
backtrace/callstack inspection API.

> (we can't use exceptions at all inside a backtrace handler, for example)

I'm a bit surprised by this. Part of the point of reifying traces into
datatypes (raw_backtrace, backtrace_slot) was to make them persistent.
set_uncaught_exception_handler is passed a raw trace that should not be
mutated by raising new exceptions. What breaks if you use exceptions inside
a backtrace handler?

On Sun, Oct 19, 2014 at 8:03 PM, John Whitington <
john@coherentgraphics.co.uk> wrote:

> Hi,
>
> OCaml 4.02 has new facilities for installing a new backtrace handler, and
> processing the back trace symbolically. As a little proof of concept, I've
> built a package which you can add to any project to get backtrace with
> source code printed and highlighted:
>
> https://github.com/johnwhitington/exn-source
>
> Here's the start of a backtrace:
>
> http://www.coherentpdf.com/exnsource.png
>
> By default, it prints five lines either side, and looks the current
> directory for source code. But as one can see from the screenshot, one can
> add /usr/local/lib/ocaml to get stdlib source and so on...
>
> It's somewhat difficult to test this kind of thing (we can't use
> exceptions at all inside a backtrace handler, for example), so I'm
> appealing for help. Attach exn-source to your code and let me know how you
> get on...
>
> Thanks,
>
> --
> John Whitington
> Director, Coherent Graphics Ltd
> http://www.coherentpdf.com/
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 3165 bytes --]

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[John Whitington]

Hi Gabriel,

Gabriel Scherer wrote:
>  > (we can't use exceptions at all inside a backtrace handler, for example)
>
> I'm a bit surprised by this. Part of the point of reifying traces into
> datatypes (raw_backtrace, backtrace_slot) was to make them persistent.
> set_uncaught_exception_handler is passed a raw trace that should not be
> mutated by raising new exceptions. What breaks if you use exceptions
> inside a backtrace handler?

To quote the documentation for set_uncaught_exception_handler: "If fn 
raises an exception, it is ignored."

This is a bit ambiguous -- it might mean

a) The uncaught exception handler you register won't get called from 
inside itself in the case of an uncaught exception. Nor will any such 
exceptions be handled by the standard exception handler; or

b) Exceptions in the handler literally have no effect.

I've done a little test just now, and it seems to be (a), so perhaps 
it's just in need of clarification in the documentation.

(It might be nice to have, for development, an option to have the 
default exception handler remain active, so it reports any exceptions 
escaping from the user-installed exception handler).

Thanks,

-- 
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2356 bytes --]

> perhaps it's just in need of clarification in the documentation.

My understanding is that uncaught exception raised by the handler are
dropped/ignored.
Would you mind creating a mantis issue ( http://caml.inria.fr/mantis/ ) so
that we can discuss improving the documentation there?

Note that a way to side-step this issue entirely would be for your API to
provide something in the style of Printexc.print
  handle : ('a -> 'b) -> 'a -> 'b
that handles any exception raised by the function application. Users would
be able to call this explicitly around their main processing loop (which is
the place where they often already handle exception-handling) instead of
delegating to a possibly-fragile mutable final handler.

My understanding is that exn-source currently operates by a side-effect at
link-time. I would rather have the choice between an explicit (unit ->
unit) registration function, and a side-effect-free handling function as
above.

On Sun, Oct 19, 2014 at 8:55 PM, John Whitington <
john@coherentgraphics.co.uk> wrote:

> Hi Gabriel,
>
> Gabriel Scherer wrote:
>
>>  > (we can't use exceptions at all inside a backtrace handler, for
>> example)
>>
>> I'm a bit surprised by this. Part of the point of reifying traces into
>> datatypes (raw_backtrace, backtrace_slot) was to make them persistent.
>> set_uncaught_exception_handler is passed a raw trace that should not be
>> mutated by raising new exceptions. What breaks if you use exceptions
>> inside a backtrace handler?
>>
>
> To quote the documentation for set_uncaught_exception_handler: "If fn
> raises an exception, it is ignored."
>
> This is a bit ambiguous -- it might mean
>
> a) The uncaught exception handler you register won't get called from
> inside itself in the case of an uncaught exception. Nor will any such
> exceptions be handled by the standard exception handler; or
>
> b) Exceptions in the handler literally have no effect.
>
> I've done a little test just now, and it seems to be (a), so perhaps it's
> just in need of clarification in the documentation.
>
> (It might be nice to have, for development, an option to have the default
> exception handler remain active, so it reports any exceptions escaping from
> the user-installed exception handler).
>
>
> Thanks,
>
> --
> John Whitington
> Director, Coherent Graphics Ltd
> http://www.coherentpdf.com/
>
>

[-- Attachment #2: Type: text/html, Size: 3183 bytes --]

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2567 bytes --]

>
> Done. http://caml.inria.fr/mantis/view.php?id=6619
>

Thanks! Having an issue really helps tracking that things get done.

there is one advantage to the current approach: no source code needs to be
> changed in the simplest case (just searching '.') - we just link exn-source
> in.
> [...]
> So perhaps I should make it build exn-source.cm(x)a and exn-source-easy.cm(x)a,
> and have two choices.
>

Indeed. exn-source-easy (or -autolink) could simply depend on the "clean
API" in the exn-source library, by being implemented as just
  let () = Exn_source.register ()

Then simply making sure to pass the flag -linkall when building that second
cma means that explicitly requiring it (in the build system) will enable
the feature.

On Sun, Oct 19, 2014 at 9:38 PM, John Whitington <
john@coherentgraphics.co.uk> wrote:

> Hi,
>
> Gabriel Scherer wrote:
>
>>  > perhaps it's just in need of clarification in the documentation.
>>
>> My understanding is that uncaught exception raised by the handler are
>> dropped/ignored.
>> Would you mind creating a mantis issue ( http://caml.inria.fr/mantis/ )
>> so that we can discuss improving the documentation there?
>>
>
> Done. http://caml.inria.fr/mantis/view.php?id=6619
>
>  Note that a way to side-step this issue entirely would be for your API
>> to provide something in the style of Printexc.print
>>    handle : ('a -> 'b) -> 'a -> 'b
>> that handles any exception raised by the function application. Users
>> would be able to call this explicitly around their main processing loop
>> (which is the place where they often already handle exception-handling)
>> instead of delegating to a possibly-fragile mutable final handler.
>>
>> My understanding is that exn-source currently operates by a side-effect
>> at link-time. I would rather have the choice between an explicit (unit
>> -> unit) registration function, and a side-effect-free handling function
>> as above.
>>
>
> Correct. Your suggestion would be much cleaner, but there is one advantage
> to the current approach: no source code needs to be changed in the simplest
> case (just searching '.') - we just link exn-source in.
>
> People won't want to necessarily ship code with exn-source linked in, and
> it's not nice to have to modify code between debug and release -- I like to
> try to have those differences all in the build system.
>
> So perhaps I should make it build exn-source.cm(x)a and exn-source-easy.cm(x)a,
> and have two choices.
>
> Thanks.
>
>
> --
> John Whitington
> Director, Coherent Graphics Ltd
> http://www.coherentpdf.com/
>
>

[-- Attachment #2: Type: text/html, Size: 4323 bytes --]

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Nicolas Boulay]

[-- Attachment #1: Type: text/plain, Size: 1705 bytes --]

Very nive project !

May a suggest a different result presentation ? Most of the time, only name
of the function called are usefull, its more compact than 5 lines arround
the call site.

One other very important feature will be the parameter value of the
function call. It's value could be serialised on fixe size (<20 characters
by default). On a tricky bug, the next step is to print such value, so you
save time to print it with the function call.

2014-10-19 20:03 GMT+02:00 John Whitington <john@coherentgraphics.co.uk>:

> Hi,
>
> OCaml 4.02 has new facilities for installing a new backtrace handler, and
> processing the back trace symbolically. As a little proof of concept, I've
> built a package which you can add to any project to get backtrace with
> source code printed and highlighted:
>
> https://github.com/johnwhitington/exn-source
>
> Here's the start of a backtrace:
>
> http://www.coherentpdf.com/exnsource.png
>
> By default, it prints five lines either side, and looks the current
> directory for source code. But as one can see from the screenshot, one can
> add /usr/local/lib/ocaml to get stdlib source and so on...
>
> It's somewhat difficult to test this kind of thing (we can't use
> exceptions at all inside a backtrace handler, for example), so I'm
> appealing for help. Attach exn-source to your code and let me know how you
> get on...
>
> Thanks,
>
> --
> John Whitington
> Director, Coherent Graphics Ltd
> http://www.coherentpdf.com/
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2695 bytes --]

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Peter Zotov]

On 2014-10-19 22:03, John Whitington wrote:
> Hi,
> 
> OCaml 4.02 has new facilities for installing a new backtrace handler,
> and processing the back trace symbolically. As a little proof of
> concept, I've built a package which you can add to any project to get
> backtrace with source code printed and highlighted:
> 
> [snip]

Very nice project! However, it looks like the user needs to explicitly
specify the paths to the sources. I have some code to perform a similar
task[1]; the paths are actually contained inside the bytecode.
I think that for native-code you can fetch them in a similar way, though
it is a little harder to extract the debug info.

[1]: 
https://github.com/whitequark/pry.ml/blob/master/src/pry_bytecode.ml,
see di_paths.

-- 
Peter Zotov

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[John Whitington]

Hi,

Peter Zotov wrote:
> On 2014-10-19 22:03, John Whitington wrote:
>> Hi,
>>
>> OCaml 4.02 has new facilities for installing a new backtrace handler,
>> and processing the back trace symbolically. As a little proof of
>> concept, I've built a package which you can add to any project to get
>> backtrace with source code printed and highlighted:
>>
>> [snip]
>
> Very nice project! However, it looks like the user needs to explicitly
> specify the paths to the sources. I have some code to perform a similar

Yes, except for '.' which is hardcoded. So, in the simplest case, it works.

> task[1]; the paths are actually contained inside the bytecode.
> I think that for native-code you can fetch them in a similar way, though
> it is a little harder to extract the debug info.
>
> [1]: https://github.com/whitequark/pry.ml/blob/master/src/pry_bytecode.ml,
> see di_paths.

Noted. I'd like to add functionality like this. The first thing I'm 
going to add is automatic finding of the stdlib, since this just 
requires parsing the output of "ocamlc -config".

Unfortunately, I don't believe most OPAM packages install their source, 
just the .mli files. I wonder how much extra space installing source 
would take on average, and if people think it's worth it, for this or 
other reasons.

Thanks,

-- 
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Peter Zotov]

On 2014-10-20 15:52, John Whitington wrote:
> Noted. I'd like to add functionality like this. The first thing I'm
> going to add is automatic finding of the stdlib, since this just
> requires parsing the output of "ocamlc -config".

I would take this directly from OCaml configuration (module Config,
-package compiler-libs.common). No shelling out needed.

> Unfortunately, I don't believe most OPAM packages install their
> source, just the .mli files. I wonder how much extra space installing
> source would take on average, and if people think it's worth it, for
> this or other reasons.

I think it's mainly due to lack of infrastructure that could install
sources (eg OASIS couldn't easily be coerced into doing that). You could
probably make opam do the grunt work for you; no need to have the 
sources
installed in the same places as mli or binaries!

-- 
Peter Zotov

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Francois Berenger]

On 10/20/2014 02:06 PM, Peter Zotov wrote:
> On 2014-10-20 15:52, John Whitington wrote:
>> Noted. I'd like to add functionality like this. The first thing I'm
>> going to add is automatic finding of the stdlib, since this just
>> requires parsing the output of "ocamlc -config".
>
> I would take this directly from OCaml configuration (module Config,
> -package compiler-libs.common). No shelling out needed.
>
>> Unfortunately, I don't believe most OPAM packages install their
>> source, just the .mli files. I wonder how much extra space installing
>> source would take on average, and if people think it's worth it, for
>> this or other reasons.
>
> I think it's mainly due to lack of infrastructure that could install
> sources (eg OASIS couldn't easily be coerced into doing that). You could
> probably make opam do the grunt work for you; no need to have the sources
> installed in the same places as mli or binaries!

Isn't opam -b or --keep-build-dir supposed to keep the sources?

-- 
Regards,
Francois.

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Török Edwin]

On 10/20/2014 03:15 PM, Francois Berenger wrote:
> On 10/20/2014 02:06 PM, Peter Zotov wrote:
>> On 2014-10-20 15:52, John Whitington wrote:
>>> Noted. I'd like to add functionality like this. The first thing I'm
>>> going to add is automatic finding of the stdlib, since this just
>>> requires parsing the output of "ocamlc -config".
>>
>> I would take this directly from OCaml configuration (module Config,
>> -package compiler-libs.common). No shelling out needed.
>>
>>> Unfortunately, I don't believe most OPAM packages install their
>>> source, just the .mli files. I wonder how much extra space installing
>>> source would take on average, and if people think it's worth it, for
>>> this or other reasons.

Other reasons for wanting things from the build directory are 'cmt/cmti' files for merlin/ocamlspot/ocp-index (or their equivalent file that only stores documentation used by opam-doc).
Do these require the original source files to extract&format the documentation strings?

>>
>> I think it's mainly due to lack of infrastructure that could install
>> sources (eg OASIS couldn't easily be coerced into doing that). You could
>> probably make opam do the grunt work for you; no need to have the sources
>> installed in the same places as mli or binaries!
> 
> Isn't opam -b or --keep-build-dir supposed to keep the sources?
> 

It does, but it also requires more disk space by keeping all the binaries and intermediate files around.

Perhaps this tool could be extended to install the source files too, or something equivalent provided at the opam level:
https://bitbucket.org/camlspotter/spotinstall

Then you wouldn't need to modify the build system of every package to install its sources / its documentation /its annotation files.

Best regards,
--Edwin

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[John Whitington]

Hi,

John Whitington wrote:
> Hi,
>
> OCaml 4.02 has new facilities for installing a new backtrace handler,
> and processing the back trace symbolically. As a little proof of
> concept, I've built a package which you can add to any project to get
> backtrace with source code printed and highlighted:
>
> https://github.com/johnwhitington/exn-source
>
> Here's the start of a backtrace:
>
> http://www.coherentpdf.com/exnsource.png
>
> By default, it prints five lines either side, and looks the current
> directory for source code.

Have cut a v0.1, now in OPAM "opam install exn-source".

a) Now finds the stdlib automatically;
b) Fixed to underline multiple-line errors;
c) Compiles with -linkall so activated merely by linking.

Thanks,

-- 
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Sébastien Hinderer]

Hi,

John Whitington (2014/10/23 09:32 +0100):
> Hi,
> 
> John Whitington wrote:
> >Hi,
> >
> >OCaml 4.02 has new facilities for installing a new backtrace handler,
> >and processing the back trace symbolically. As a little proof of
> >concept, I've built a package which you can add to any project to get
> >backtrace with source code printed and highlighted:
> >
> >https://github.com/johnwhitington/exn-source

This seems very interesting, thanks!

> >Here's the start of a backtrace:
> >
> >http://www.coherentpdf.com/exnsource.png

Just wanted to mention that not everybody can look at pictures, so it
would be great to show the text itself whenever it'spossible.

Thanks,
Sébastien.

Re: [Caml-list] ANN: exn-source - exception backtraces with source code printing

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

exn-source produces an output using ANSI escape codes. Here is a text
version (ansifilter --text):
  http://gallium.inria.fr/~scherer/tmp/tmp/exnsource-example.text
and a HTML version
  http://gallium.inria.fr/~scherer/tmp/tmp/exnsource-example.html

To (re)produce this output I ran the command
  OCAMLRUNPARAM="b" ./cpdf -debug foo.pdf
from the the tag v1.7 of the repo
  https://github.com/johnwhitington/cpdf-source/
(after installing both camlpdf (from OPAM) and exnsource (from github))
with the following patch applied:

diff --git a/Makefile b/Makefile
index d706771..c4f9523 100644
--- a/Makefile
+++ b/Makefile
@@ -5,7 +5,7 @@ SOURCES = $(foreach x,$(MODS),$(x).ml $(x).mli)
cpdfcommandrun.ml

 RESULT = cpdf

-PACKS = camlpdf
+PACKS = camlpdf exnsource

 OCAMLNCFLAGS = -g
 OCAMLBCFLAGS = -g
diff --git a/cpdfcommandrun.ml b/cpdfcommandrun.ml
index 2a43a43..bb0af72 100644
--- a/cpdfcommandrun.ml
+++ b/cpdfcommandrun.ml
@@ -1,2 +1,4 @@
+let _ = Exnsource.add_dir ""
+
 let _ = Cpdfcommand.go ()



On Thu, Oct 23, 2014 at 1:48 PM, Sébastien Hinderer <
Sebastien.Hinderer@inria.fr> wrote:

> Hi,
>
> John Whitington (2014/10/23 09:32 +0100):
> > Hi,
> >
> > John Whitington wrote:
> > >Hi,
> > >
> > >OCaml 4.02 has new facilities for installing a new backtrace handler,
> > >and processing the back trace symbolically. As a little proof of
> > >concept, I've built a package which you can add to any project to get
> > >backtrace with source code printed and highlighted:
> > >
> > >https://github.com/johnwhitington/exn-source
>
> This seems very interesting, thanks!
>
> > >Here's the start of a backtrace:
> > >
> > >http://www.coherentpdf.com/exnsource.png
>
> Just wanted to mention that not everybody can look at pictures, so it
> would be great to show the text itself whenever it'spossible.
>
> Thanks,
> Sébastien.
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 3626 bytes --]