Re: [ANN] camlish: a simple module for shell scripting in OCaml

[Zheng Li <zheng_li@users.sourceforge.net>]

Hi Mikkel,

I didn't see your post appearing on list, but I think maybe there are 
other also interested in this topic, so I reply to the list directly.

Mikkel Fahnøe Jørgensen wrote:
> I'm sorry for not being clear
> 
> I did _not_ mean: would it be smart for camlish to take advantage of LWT ...
> 
> I did mean: for users buying into asynchronous programming using the
> LWT interface, it is critical to avoid blocking, or everything stops.
> The usual read / write functions have been wrapped asynchronously.
> 
> Say you are on a web server handling various backend processes. You
> want to pipeline some image transformation and compression using a
> shell pipeline.
> 
> Now it would be sad to block the web server while the image is completed.
> But it would be nice to use a powerful shell scripting interface to
> manipulate images.
> And it would be nice to take advantage of parallelism for performance,
> not just to avoid blocking.

Now I understand what you meant.

Yes, asynchronous lib such as Lwt requires every possibly blocking
operation to be asynchronous, i.e. return a value of Lwt.t. For
OCaml's type system, it's not obliged, but then you are not
asynchronous any more. It's like the common problem of monad, once you
get one, it has to spread everywhere.

However, I think it's not practical to ask every library developer
to adopt the asynchronous style and bind their type definition to
a specific lib.

> More comments below:
Me too :)

>>  * If you meant to use them together, I think that's fine. They
>> are both user level libraries, Lwt has an asynchronous interface,
>> camlish has a synchronous one, so you can just use camlish API as
>> common functions application everywhere.
> 
> So this is possible, but will apparently block the LWT thread system ...
> LWT does have an option to park external operations in a system thread

I didn't know about that. When I looked at it, it didn't have that, but 
maybe they added it later.

> so that might be a way to go?

Yes, I think so. It should be the asynchronous lib to deal with that, 
otherwise we have to ask every library author to be aware of it. In 
Async, we provide a few functions to help wrapping synchronous
functions into asynchronous ones. So other libraries don't have to 
change, but still this is not transparent.

>>  * If you meant to implement the inside asynchronous mechanics of
>> camlish on top of Lwt, I did thought of that. Actually, I have
>> another library called Async does similar thing as Lwt. It's
>> possible, but not necessary. Besides, the CPS-based approach
>> always brings some syntactic burdens, which I prefer to avoid.
>> So finally, the inside implementation of camlish is asynchronous,
>> while we only expose the synchronous interface to the outside world.
> 
> Async could be an alternative to LWT, I have not looked at it, but the
> it is the same issue. You would not want to block the threads while
> calling shell commands.

It has been used internally, but not released yet. I haven't got the 
time to clean up the code.

>>  * Maybe you were though about parallel execution? Pipeline is
>> parallel already, plus there are two parallel combinators have
>> been planed (opposite to "&&&", the sequential combinator). I
>> haven't seen this kind of stuff in other shells, but I think that's
>> reasonable.
> 
> Yes that too. Fire off all the shell commands and have an LWT thread
> monitor them and feed the result in the LWT thread chain.

Sure. What you want in you post is concurrency not parallelism.

--
Zheng