[ANN] camlish: a simple module for shell scripting in OCaml

[ANN] camlish: a simple module for shell scripting in OCaml

[Zheng Li]

Hi,

I'd like to announce the availability of a small module for shell 
scripting: camlish.

The project URL is: http://zheng.li/projects/ocaml/camlish, where you 
can get all the code and docs. A source repo is going to provide later.

It's rather experimental, bug reports are always welcome.

Have fun!


FYI, the rest parts are directly copy&pasted from the README file:
-----------------------------------------------------------------------


===== Motivation =====

I was thinking about using OCaml for shell scripting. I once looked at
Cash, but quickly went away: I didn't really want to read 100+ pages of
manual just for my little scripting purposes. Recently there came
Caml-Shcaml, it also successfully stopped me from exploring any further
with its ~700 APIs (Sure I believe learning a small portion of them can
be enough to get me a quick start, but still...)

I believe they are both excellent and full-fledged shell
implementations, but they are really overkilled for me. Vanilla OCaml is
already a full-functional language which I can do anything with, include
scripting. What I want is just a little more convenience on this aspect,
better with minimal learning cost, not to learn another "language".
Otherwise I could just resort to plain ocaml because I'm lazy. Moreover,
I don't want to patch my compiler source or use Camlp4 magic to have a
more shell-like syntax: it's probably nice-looking but alien.


==== Basics ====

So I wrote camlish, a plain OCaml module to ease shell scripting in
OCaml, especially for those thinking the same.

Camlish follows simple principles:

   * It deals with simple coordination and interaction with outside
     commands and shells. It provides a very thin layer of glues to
     ease that. That's all it does.

   * OCaml itself has far better language features than any shell
     script languages out there, so when camlish is used for scripting,
     it encourages you to stay on the OCaml side as much as possible,
     call external and push/pull the inputs/outputs back/forth as OCaml
     values, and sometime resort to /bin/sh if necessary.

==== Features ====

Here are some features of camlish:

   * Code base is small. The core is just 500+ loc, 30+ definitions
     (except operator sugar).

   * Interface is simple. It deals and only deals with the follows:

     - command construction: wrap a string or a OCaml function to a
       typed command

       <code ocaml>
       let c1 = cmd "seq 1 20"                    (* (unit, unit) cmd *)

       let c2 = func (List.map float_of_int)
                                        (* (int list, float list) cmd *)
       </code>

     - command redirection: redirect input/output/error to any files,
       file descriptors and OCaml values, via plain record syntax. We
       call all these direction adapters as "ports". Besides a number
       of primitive ports, we can define and compose our own "ports"
       for any data type.

       <code ocaml>
       let c3 = { c1 with pout = pout_lines }
       	       	    	      		  (* (unit, string list) cmd *)
       let c4 = { c2 with pin = int_of_string =| c2.pin }
                                     (* (string list, float list) cmd *)
       </code>

     - command composition: pipeline(>>>), sequence (&&&) and
       subshell(~$) and more.

       <code ocaml>
       let c5 = c3 >>> c4                    (* (unit, float list) cmd *)
       let c6 = {(cmd "pwd" &&& cmd "date") with pout = pout_file "log"}
                                                   (* (unit, unit) cmd *)
       </code>
       						
     - command execution: input/return values according to command's
       redirection. There are different execution schemes depending on
       various factors (e.g. having input value or not, running in
       background or not etc.)

       <code ocaml>
       !! c5                          (* get float list [1.; ...; 20.] *)
       </code>

       The example is made up for illustration. In practice you'd
       instead write

       <code ocaml>
       !! { (cmd "seq 1 20") with pout = pout_lines |= float_of_string }
       </code>

       and commands without redirection look normal:

       <code ocaml>
       !! (cmd "ls -l" >>> cmd "wc -l")
       !$ "ls -l | wc -l"                    (* resort to /bin/sh pipe *)
       </code>

       Here is an example collaborating commands and functions to count
       the numbers x between 1 .. n where x mod 3 or 5 is 0.

       <code ocaml>
       let count n=
         { (cmd "seq 1 %d" n) with pout = pout_lines }
         >>= int_of_string
         >>- List.filter (fun x -> x mod 3 = 0 || x mod 5 = 0)
         >>> {( cmd "wc -l") with
                pin = string_of_int =| pin_lines;
                pout = pout_string |- int_of_string }
                                          (* int -> (unit, int) cmd *)

       !! (count 1000)                     (* return OCaml int: 466 *)
       </code>

       Please read the manual for more details.

   * The interaction between external commands and OCaml values are
     implemented asynchronously inside (but with a synchronous
     interface). So it won't get stuck unnecessarily when dealing with
     large chunk of input/output.

     Moreover, a pair of stream ports are provided, so that you can
     read/write "not fully available" or infinite string/data on
     the borders of OCaml and commands. E.g.

     <code ocaml>
     let s = !!. { (cmd "yes") with pout = pout_stream "\n" }
                                  (* the endless output of "yes" is
				    redirected to a stream value. *)

     let _ = Stream.iter print_endline s     (* print infinite "y" *)
     </code>

   * It's tempting to use Camlp4 or to modify the compiler in order to
     achieving terse shell-like syntax, but we choose not. Camlish is a
     plain OCaml module.

   * It's also tempting to implement simple versions of common unix
     tools such as "ls" and "grep" as OCaml functions to achieve better
     composability. We opt not for the core module and not for now.

   * Camlish supports RC file written in OCaml, i.e. when you load
     camlish module from toplevel or run customized toplevel with
     Camlish embedded, it will try to read a file named ".camlishrc"
     from the current working directory firstly and the home directory
     of current user secondly. The RC file can be any plain ocaml file,
     e.g.

     <code ocaml>
     open Camlish
     open List
     open Prelude
     module A = Array
     module S = String
     module U = Unix
     </code>

   * When used interactively, camlish can additionally execute shell
     commands directly. So we can now use the toplevel as a shell,
     feeding it ocaml phrases and shell commands at the same time.

     Here is a trace of record:

     <code ocaml>
     $ rlwrap camlish   (* Or rlwrap ocaml unix.cma camlish.cma *)

            Objective Caml version 3.10.2
	
     [1] ls -l | wc -l;;
     53
     - : unit = ()
     [2] date;;
     Sat Oct 25 12:06:51 CEST 2008
     - : unit = ()
     [3] let f x = x;;
     val f : 'a -> 'a = <fun>
     [4] f 100;;
     - : int = 100
     [5] for i in `seq 1 3`; do echo "Now: "$i; done;;;
     Now: 1
     Now: 2
     Now: 3
     - : unit = ()
     [6] emacs -nw;;
     </code>

     In case there is some ambiguity, e.g. a OCaml value shadows a
     command name, a ";;;" ending can be used to enforce it is a
     command instead of OCaml toplevel phrase.


Please refer to the manual for more details.

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Sylvain Le Gall]

Hello,

On 27-10-2008, Zheng Li <zheng_li@users.sourceforge.net> wrote:
> Hi,
>
> I'd like to announce the availability of a small module for shell 
> scripting: camlish.
>
> The project URL is: http://zheng.li/projects/ocaml/camlish, where you 
> can get all the code and docs. A source repo is going to provide later.
>

If you want to setup a source code repository, you should consider
http://forge.ocamlcore.org where you can have bug report, mailing list
and source repository (git, darcs, hg, svn and cvs).

>
>    * It's also tempting to implement simple versions of common unix
>      tools such as "ls" and "grep" as OCaml functions to achieve better
>      composability. We opt not for the core module and not for now.
>

You should take a look at 
http://le-gall.net/sylvain+violaine/documentation/ocaml-fileutils/html/api/FileUtil.StrUtil.html

Where you already have a basic implementation for ls, find, touch, rm,
cp, du, cmp... These are pure OCaml functions, that you can call directly
from OCaml (not external call).

Regards,
Sylvain Le Gall

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Zheng Li]

Hi,

Sylvain Le Gall wrote:
> On 27-10-2008, Zheng Li <zheng_li@users.sourceforge.net> wrote:
>>    * It's also tempting to implement simple versions of common unix
>>      tools such as "ls" and "grep" as OCaml functions to achieve better
>>      composability. We opt not for the core module and not for now.
>>
> 
> You should take a look at 
> http://le-gall.net/sylvain+violaine/documentation/ocaml-fileutils/html/api/FileUtil.StrUtil.html
> 
> Where you already have a basic implementation for ls, find, touch, rm,
> cp, du, cmp... These are pure OCaml functions, that you can call directly
> from OCaml (not external call).

Thanks for the info. I remember I once saw a smaller lib with similar
name for this purpose, maybe it was the same one but you upgraded it?

That's the exact reason I'd leave camlish as small as possible and only
concentrate on the glue layer. Your lib just gave this another vote.
I'll add a reference to FileUtil in the README. Camlish users can just
customize their working environments by importing various
modules/functions in scope via their .camlishrc file.

--
Zheng

Re: [Caml-list] [ANN] camlish: a simple module for shell scripting in OCaml

[Mikkel Fahnøe Jørgensen]

2008/10/27 Zheng Li <zheng_li@users.sourceforge.net>:
> Hi,
>
> I'd like to announce the availability of a small module for shell scripting:
> camlish.
>
> The project URL is: http://zheng.li/projects/ocaml/camlish, where you can
> get all the code and docs. A source repo is going to provide later.
>
> It's rather experimental, bug reports are always welcome.
>
> Have fun!

Great!, but bummer, I wrote one too last night....

http://git.dvide.com/pub/ocaml-shell-utils/tree/
http://git.dvide.com/pub/ocaml-shell-utils/plain/README.txt

Also looked at FileUtils and ShCaml, but wanted something light.

Anyway - it seems orthogonal to mine.

I don't do any pipeline processing, I just make it easy to cp, mv,
etc. using ocaml toplevel scripts as an alternative to install
scripts.
Perhaps this could be combined ?
I'm currently adding a simple file globber.


Regards, Mikkel

Re: [Caml-list] [ANN] camlish: a simple module for shell scripting in OCaml

[Andre Nathan]

On Mon, 2008-10-27 at 14:11 +0100, Mikkel Fahnøe Jørgensen wrote:
> I'm currently adding a simple file globber.

FWIW,

http://github.com/andrenth/ocaml-glob/


Andre

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Sylvain Le Gall]

On 27-10-2008, Andre Nathan <andre@digirati.com.br> wrote:
> On Mon, 2008-10-27 at 14:11 +0100, Mikkel Fahnøe Jørgensen wrote:
>> I'm currently adding a simple file globber.
>
> FWIW,
>
> http://github.com/andrenth/ocaml-glob/
>

Why don't use forge.ocamlcore.org ?

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Andre Nathan]

On Mon, 2008-10-27 at 13:18 +0000, Sylvain Le Gall wrote:
> Why don't use forge.ocamlcore.org ?

Good question :)  I'll create a project there later today.

Andre

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Sylvain Le Gall]

On 27-10-2008, Andre Nathan <andre@digirati.com.br> wrote:
> On Mon, 2008-10-27 at 13:18 +0000, Sylvain Le Gall wrote:
>> Why don't use forge.ocamlcore.org ?
>
> Good question :)  I'll create a project there later today.
>

FYI, you can ask also for a git repository there.

Regards,
Sylvain Le Gall

Re: [Caml-list] [ANN] camlish: a simple module for shell scripting in OCaml

[Mikkel Fahnøe Jørgensen]

2008/10/27 Andre Nathan <andre@digirati.com.br>:
> On Mon, 2008-10-27 at 14:11 +0100, Mikkel Fahnøe Jørgensen wrote:
>> I'm currently adding a simple file globber.
>
> FWIW,
>
> http://github.com/andrenth/ocaml-glob/
>

This is nice.

I wanted a simple ocaml based globber that can be embedded directly in
the shell_utils.ml source so there isn't any cross platform and
installation issues. But my globber only takes a single directory for
now and is probably not fast.

Users could install your globber and use it together with shell_utils.ml.

I need to think about the 'cp' interface a bit more.
Currently it takes a string. Should there be a variant taking a glob,
or an expanded list, or should cp take a single string of multiple
files that could originate from a glob?

Or should cp always take a glob pattern and have a way to plug in
other glob functions?

Mikkel

Re: [Caml-list] [ANN] camlish: a simple module for shell scripting in OCaml

[Mikkel Fahnøe Jørgensen]

>  * The interaction between external commands and OCaml values are
>    implemented asynchronously inside (but with a synchronous
>    interface). So it won't get stuck unnecessarily when dealing with
>    large chunk of input/output.

Would it be possible to combine this with the LWT thread library?

http://www.ocsigen.org/lwt

Mikkel

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Zheng Li]

Mikkel Fahnøe Jørgensen wrote:
> Great!, but bummer, I wrote one too last night....
> 
> http://git.dvide.com/pub/ocaml-shell-utils/tree/
> http://git.dvide.com/pub/ocaml-shell-utils/plain/README.txt
> 
> Also looked at FileUtils and ShCaml, but wanted something light.
> 
> Anyway - it seems orthogonal to mine.
> 
> I don't do any pipeline processing, I just make it easy to cp, mv,
> etc. using ocaml toplevel scripts as an alternative to install
> scripts.

I just had a visit to your project, and yes, I agree they are mostly
orthogonal. Camlish only concentrates on the interaction, redirection,
composition and coordination of _external_ commands, pushing/pulling the
input/output as OCaml values. It doesn't, and probably won't, define
functions as common shell commands by itself, which, I believe, are
better left to other libraries.

As a compensation, camlish allows one to execute shell commands directly
from toplevel, if (s)he doesn't care about the interaction with OCaml
world. So instead of writing

   # !! cmd "ls -l";;

which is plain OCaml function calling outside command "ls", one can
simply write

   # ls -l ;;

which is a shell command, not a function named "ls". I myself am
interested in using OCaml toplevel as a shell environment.

Still, it's very nice to see others confronting similar problems and
trying to solve them in different approaches.

--
Zheng

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Zheng Li]

Mikkel Fahnøe Jørgensen wrote:
>>  * The interaction between external commands and OCaml values are
>>    implemented asynchronously inside (but with a synchronous
>>    interface). So it won't get stuck unnecessarily when dealing with
>>    large chunk of input/output.
> 
> Would it be possible to combine this with the LWT thread library?
> http://www.ocsigen.org/lwt

I'm not clear what kind of "combine" you had in mind.

   * If you meant to use them together, I think that's fine. They
are both user level libraries, Lwt has an asynchronous interface,
camlish has a synchronous one, so you can just use camlish API as
common functions application everywhere.

  * If you meant to implement the inside asynchronous mechanics of
camlish on top of Lwt, I did thought of that. Actually, I have
another library called Async does similar thing as Lwt. It's
possible, but not necessary. Besides, the CPS-based approach
always brings some syntactic burdens, which I prefer to avoid.
So finally, the inside implementation of camlish is asynchronous,
while we only expose the synchronous interface to the outside world.

  * Maybe you were though about parallel execution? Pipeline is
parallel already, plus there are two parallel combinators have
been planed (opposite to "&&&", the sequential combinator). I
haven't seen this kind of stuff in other shells, but I think that's
reasonable.

HTH.

--
Zheng

Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Zheng Li]

Hi Mikkel,

I didn't see your post appearing on list, but I think maybe there are 
other also interested in this topic, so I reply to the list directly.

Mikkel Fahnøe Jørgensen wrote:
> I'm sorry for not being clear
> 
> I did _not_ mean: would it be smart for camlish to take advantage of LWT ...
> 
> I did mean: for users buying into asynchronous programming using the
> LWT interface, it is critical to avoid blocking, or everything stops.
> The usual read / write functions have been wrapped asynchronously.
> 
> Say you are on a web server handling various backend processes. You
> want to pipeline some image transformation and compression using a
> shell pipeline.
> 
> Now it would be sad to block the web server while the image is completed.
> But it would be nice to use a powerful shell scripting interface to
> manipulate images.
> And it would be nice to take advantage of parallelism for performance,
> not just to avoid blocking.

Now I understand what you meant.

Yes, asynchronous lib such as Lwt requires every possibly blocking
operation to be asynchronous, i.e. return a value of Lwt.t. For
OCaml's type system, it's not obliged, but then you are not
asynchronous any more. It's like the common problem of monad, once you
get one, it has to spread everywhere.

However, I think it's not practical to ask every library developer
to adopt the asynchronous style and bind their type definition to
a specific lib.

> More comments below:
Me too :)

>>  * If you meant to use them together, I think that's fine. They
>> are both user level libraries, Lwt has an asynchronous interface,
>> camlish has a synchronous one, so you can just use camlish API as
>> common functions application everywhere.
> 
> So this is possible, but will apparently block the LWT thread system ...
> LWT does have an option to park external operations in a system thread

I didn't know about that. When I looked at it, it didn't have that, but 
maybe they added it later.

> so that might be a way to go?

Yes, I think so. It should be the asynchronous lib to deal with that, 
otherwise we have to ask every library author to be aware of it. In 
Async, we provide a few functions to help wrapping synchronous
functions into asynchronous ones. So other libraries don't have to 
change, but still this is not transparent.

>>  * If you meant to implement the inside asynchronous mechanics of
>> camlish on top of Lwt, I did thought of that. Actually, I have
>> another library called Async does similar thing as Lwt. It's
>> possible, but not necessary. Besides, the CPS-based approach
>> always brings some syntactic burdens, which I prefer to avoid.
>> So finally, the inside implementation of camlish is asynchronous,
>> while we only expose the synchronous interface to the outside world.
> 
> Async could be an alternative to LWT, I have not looked at it, but the
> it is the same issue. You would not want to block the threads while
> calling shell commands.

It has been used internally, but not released yet. I haven't got the 
time to clean up the code.

>>  * Maybe you were though about parallel execution? Pipeline is
>> parallel already, plus there are two parallel combinators have
>> been planed (opposite to "&&&", the sequential combinator). I
>> haven't seen this kind of stuff in other shells, but I think that's
>> reasonable.
> 
> Yes that too. Fire off all the shell commands and have an LWT thread
> monitor them and feed the result in the LWT thread chain.

Sure. What you want in you post is concurrency not parallelism.

--
Zheng