[Caml-list] A question about Ocaml logging

[Caml-list] A question about Ocaml logging

[Chet Murthy]

All,

I hope this is the right place to ask this question.  I've been
writing a nontrivial distributed system (well, a number of them over
the last few years) and have had need of a robust and flexible logging
framework.  Specifically, I've been using "bolt" and its descendant,
"volt", which provide camlp4 syntax extensions.  These extensions make
the syntax of the logging statements significantly less verbose, and
that in itself ia a valuable thing.

With the arrival of ppx rewriters, I realize that the camlp4/camlp5
method of adding syntax to ocaml is deprecated.  So I wonder: is there
some really good logging toolkit out there, that I've overlooked.

I'm aware of a number of different packages, but only bolt/volt have
syntax extensions, and it's my belief that they're essential to making
effortless pervasive log-line instrumentation.

But perhaps I just haven't looked hard enough ....  So .... before I
go write my own, I figured I'd ask the list if there were such a
thing.

For concreteness, the basic thing I'm looking for, is a syntax
extension that somewhat resembles

  LOG "%s" appid LEVEL DEBUG ;

which expands out to code something like

begin
  if log_enabled <current-module/function-path-as-string> DEBUG then
    log (Printf.sprintf "%s" appid)
end

I hate writing "slight variations on somebody else's good idea", so
would much prefer to "improve" somebody else's library/tool, than come
up with my own.

Thanks in advance,
--chet--

Re: [Caml-list] A question about Ocaml logging

[Daniil Baturin]

[-- Attachment #1.1: Type: text/plain, Size: 2172 bytes --]

Hi Chet,

Have you looked into the logging modules from Lwt? While they work best
for those who already use Lwt,
and they definitely have their shortcoming, on the plus side the log
statements are very concise,
you can set the log level and add logging rules at runtime, and it has
file and syslog logging out of the box.


On 01/10/2017 01:52 AM, Chet Murthy wrote:
> All,
>
> I hope this is the right place to ask this question.  I've been
> writing a nontrivial distributed system (well, a number of them over
> the last few years) and have had need of a robust and flexible logging
> framework.  Specifically, I've been using "bolt" and its descendant,
> "volt", which provide camlp4 syntax extensions.  These extensions make
> the syntax of the logging statements significantly less verbose, and
> that in itself ia a valuable thing.
>
> With the arrival of ppx rewriters, I realize that the camlp4/camlp5
> method of adding syntax to ocaml is deprecated.  So I wonder: is there
> some really good logging toolkit out there, that I've overlooked.
>
> I'm aware of a number of different packages, but only bolt/volt have
> syntax extensions, and it's my belief that they're essential to making
> effortless pervasive log-line instrumentation.
>
> But perhaps I just haven't looked hard enough ....  So .... before I
> go write my own, I figured I'd ask the list if there were such a
> thing.
>
> For concreteness, the basic thing I'm looking for, is a syntax
> extension that somewhat resembles
>
>   LOG "%s" appid LEVEL DEBUG ;
>
> which expands out to code something like
>
> begin
>   if log_enabled <current-module/function-path-as-string> DEBUG then
>     log (Printf.sprintf "%s" appid)
> end
>
> I hate writing "slight variations on somebody else's good idea", so
> would much prefer to "improve" somebody else's library/tool, than come
> up with my own.
>
> Thanks in advance,
> --chet--
>
>


-- 
#!/usr/bin/env perl
@a=split(//, "daniil @ baturin  .  org" );# Daniil Baturin
@b=split(//,q/Px%!+o0Q6lh*7dp$.@8#%|y{/);while($i<24){$_.=
chr((ord(@b[$i])-ord(@a[$i])+62)%94+32);$i++};print"$_\n"#



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 847 bytes --]

Re: [Caml-list] A question about Ocaml logging

[Hugo Herbelin]

Hi Chet,

> With the arrival of ppx rewriters, I realize that the camlp4/camlp5
> method of adding syntax to ocaml is deprecated.  So I wonder: is there
> some really good logging toolkit out there, that I've overlooked.

I don't know if OCaml developers recommend a deprecation of camlp4,
but as far as camlp5 is concerned, I don't believe there is an
intention from his current main developer (who actually works in the
office next door to mine) to deprecate camlp5. My understanding is
even the opposite, that he would be happy that campl5 continues to
exist after him. What is not exclusive at all of using ppx of course!
There is a room for different approaches and different objectives!

Best,

Hugo

Re: [Caml-list] A question about Ocaml logging

[Anil Madhavapeddy]

> On 9 Jan 2017, at 18:52, Chet Murthy <chetsky@gmail.com> wrote:
> 
> All,
> 
> I hope this is the right place to ask this question.  I've been
> writing a nontrivial distributed system (well, a number of them over
> the last few years) and have had need of a robust and flexible logging
> framework.  Specifically, I've been using "bolt" and its descendant,
> "volt", which provide camlp4 syntax extensions.  These extensions make
> the syntax of the logging statements significantly less verbose, and
> that in itself ia a valuable thing.
> 
> With the arrival of ppx rewriters, I realize that the camlp4/camlp5
> method of adding syntax to ocaml is deprecated.  So I wonder: is there
> some really good logging toolkit out there, that I've overlooked.
> 
> I'm aware of a number of different packages, but only bolt/volt have
> syntax extensions, and it's my belief that they're essential to making
> effortless pervasive log-line instrumentation.
> 
> But perhaps I just haven't looked hard enough ....  So .... before I
> go write my own, I figured I'd ask the list if there were such a
> thing.

Dear Chet,

In MirageOS, we've been moving away from syntax extensions and
towards placing the logging directives as closures directly within the
code.  This is slightly slower in the case of debug logging, but in
practise for distributed systems we are finding that having "permanent"
logging at different levels is more valuable than the "recompile with
debug logging" that we used to use.

The basis library for this that our libraries are mostly using now in the
forthcoming MirageOS3 is Daniel Buenzli's Logs library:
  http://erratique.ch/software/logs

It does not have a syntax extension out of the box, but it does provide
flexible support for multiple backends, and has an Lwt module included.

Hope that helps!

regards,
Anil

Re: [Caml-list] A question about Ocaml logging

[Jesper Louis Andersen]

[-- Attachment #1: Type: text/plain, Size: 4415 bytes --]

Another point from the trenches of the real world:

I've had great success with metrics over logging in many situations.

Logs are really nice for knowing what went wrong in a system. And if the
logs have some kind of structure (JSON, S-exps, Erlang Terms, ...) you can
also dump all context information relevant to the log entry. Modern systems
such as Kibana can read such structure and build a search index over them.
It is very useful when trying to figure out what went wrong. In a
distributed system, you keep a unique request id in the structural log so
you can join log entries from multiple subsystems easily in the centralized
logging platform.

Likewise, whenever a given event occurs, you bump a counter or add an entry
to a histogram (HdrHistogram comes to mind). This allows you to export
counts to a foreign system for plotting of how the system fares internally.
It is much cheaper than rendering a log line in the system, and it is
almost as informative if you don't need the log line, but rather its
occurrence. Live timing histograms also has the advantage that problems
tend to show up in the small before a catastrophe. So you can alter the
operating point of the system dynamically long before the catastrophe occur
in the real world.

I almost never have debug-style logging lines in my code anymore. I much
prefer adding a live tracing to the system instead (Erlang has tracing
facilities, DTrace is useful on Illumos and FreeBSD, etc). Granted, you
can't do this posthumously of an error, but on the other hand, you can
tailor it to the situation you have. It also takes care of the need to
recompile and redeploy with debug logging (in which case you can't
posthumously handle an error).

The right way to handle system failure is to take a snapshot of the systems
state (or of part of the system to which the problem pertains). Store this
snapshot somewhere (in the cloud) and index them, so you can go back with a
debugger, attach to the core dump, and inspect what went wrong. Post-mortem
debugging is needed because many of the errors which occur lies outside the
imagination of the programmers: people abuse systems in ways nobody thought
possible.


On Mon, Jan 16, 2017 at 4:15 PM Anil Madhavapeddy <anil@recoil.org> wrote:

> > On 9 Jan 2017, at 18:52, Chet Murthy <chetsky@gmail.com> wrote:
> >
> > All,
> >
> > I hope this is the right place to ask this question.  I've been
> > writing a nontrivial distributed system (well, a number of them over
> > the last few years) and have had need of a robust and flexible logging
> > framework.  Specifically, I've been using "bolt" and its descendant,
> > "volt", which provide camlp4 syntax extensions.  These extensions make
> > the syntax of the logging statements significantly less verbose, and
> > that in itself ia a valuable thing.
> >
> > With the arrival of ppx rewriters, I realize that the camlp4/camlp5
> > method of adding syntax to ocaml is deprecated.  So I wonder: is there
> > some really good logging toolkit out there, that I've overlooked.
> >
> > I'm aware of a number of different packages, but only bolt/volt have
> > syntax extensions, and it's my belief that they're essential to making
> > effortless pervasive log-line instrumentation.
> >
> > But perhaps I just haven't looked hard enough ....  So .... before I
> > go write my own, I figured I'd ask the list if there were such a
> > thing.
>
> Dear Chet,
>
> In MirageOS, we've been moving away from syntax extensions and
> towards placing the logging directives as closures directly within the
> code.  This is slightly slower in the case of debug logging, but in
> practise for distributed systems we are finding that having "permanent"
> logging at different levels is more valuable than the "recompile with
> debug logging" that we used to use.
>
> The basis library for this that our libraries are mostly using now in the
> forthcoming MirageOS3 is Daniel Buenzli's Logs library:
>   http://erratique.ch/software/logs
>
> It does not have a syntax extension out of the box, but it does provide
> flexible support for multiple backends, and has an Lwt module included.
>
> Hope that helps!
>
> regards,
> Anil
>
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 6528 bytes --]

Re: [Caml-list] A question about Ocaml logging

[Malcolm Matalka]

Jesper Louis Andersen <jesper.louis.andersen@gmail.com> writes:

> Another point from the trenches of the real world:
>
> I've had great success with metrics over logging in many situations.
>
> Logs are really nice for knowing what went wrong in a system. And if the
> logs have some kind of structure (JSON, S-exps, Erlang Terms, ...) you can
> also dump all context information relevant to the log entry. Modern systems
> such as Kibana can read such structure and build a search index over them.
> It is very useful when trying to figure out what went wrong. In a
> distributed system, you keep a unique request id in the structural log so
> you can join log entries from multiple subsystems easily in the centralized
> logging platform.
>
> Likewise, whenever a given event occurs, you bump a counter or add an entry
> to a histogram (HdrHistogram comes to mind). This allows you to export
> counts to a foreign system for plotting of how the system fares internally.
> It is much cheaper than rendering a log line in the system, and it is
> almost as informative if you don't need the log line, but rather its
> occurrence. Live timing histograms also has the advantage that problems
> tend to show up in the small before a catastrophe. So you can alter the
> operating point of the system dynamically long before the catastrophe occur
> in the real world.
>
> I almost never have debug-style logging lines in my code anymore. I much
> prefer adding a live tracing to the system instead (Erlang has tracing
> facilities, DTrace is useful on Illumos and FreeBSD, etc). Granted,
> you

Is it possible to create dtrace probes in Ocaml code?


> can't do this posthumously of an error, but on the other hand, you can
> tailor it to the situation you have. It also takes care of the need to
> recompile and redeploy with debug logging (in which case you can't
> posthumously handle an error).
>
> The right way to handle system failure is to take a snapshot of the systems
> state (or of part of the system to which the problem pertains). Store this
> snapshot somewhere (in the cloud) and index them, so you can go back with a
> debugger, attach to the core dump, and inspect what went wrong. Post-mortem
> debugging is needed because many of the errors which occur lies outside the
> imagination of the programmers: people abuse systems in ways nobody thought
> possible.
>
>
> On Mon, Jan 16, 2017 at 4:15 PM Anil Madhavapeddy <anil@recoil.org> wrote:
>
>> > On 9 Jan 2017, at 18:52, Chet Murthy <chetsky@gmail.com> wrote:
>> >
>> > All,
>> >
>> > I hope this is the right place to ask this question.  I've been
>> > writing a nontrivial distributed system (well, a number of them over
>> > the last few years) and have had need of a robust and flexible logging
>> > framework.  Specifically, I've been using "bolt" and its descendant,
>> > "volt", which provide camlp4 syntax extensions.  These extensions make
>> > the syntax of the logging statements significantly less verbose, and
>> > that in itself ia a valuable thing.
>> >
>> > With the arrival of ppx rewriters, I realize that the camlp4/camlp5
>> > method of adding syntax to ocaml is deprecated.  So I wonder: is there
>> > some really good logging toolkit out there, that I've overlooked.
>> >
>> > I'm aware of a number of different packages, but only bolt/volt have
>> > syntax extensions, and it's my belief that they're essential to making
>> > effortless pervasive log-line instrumentation.
>> >
>> > But perhaps I just haven't looked hard enough ....  So .... before I
>> > go write my own, I figured I'd ask the list if there were such a
>> > thing.
>>
>> Dear Chet,
>>
>> In MirageOS, we've been moving away from syntax extensions and
>> towards placing the logging directives as closures directly within the
>> code.  This is slightly slower in the case of debug logging, but in
>> practise for distributed systems we are finding that having "permanent"
>> logging at different levels is more valuable than the "recompile with
>> debug logging" that we used to use.
>>
>> The basis library for this that our libraries are mostly using now in the
>> forthcoming MirageOS3 is Daniel Buenzli's Logs library:
>>   http://erratique.ch/software/logs
>>
>> It does not have a syntax extension out of the box, but it does provide
>> flexible support for multiple backends, and has an Lwt module included.
>>
>> Hope that helps!
>>
>> regards,
>> Anil
>>
>>
>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>