select (or polling) on in_channel?

select (or polling) on in_channel?

[Eijiro Sumii]

Hi again...

Is there any easy way to _correctly_ do a "select" (as in the UNIX
system call) on Pervasives.in_channel?  My naive approach

  Unix.select [Unix.descr_of_in_channel ic] [] [] 0.

seems to fail (and it indeed does fail) because of the buffering
inside Pervasives.in_channel, as long as I checked the source code of
ocaml-3.08.3.  Pervasives.in_channel_length doesn't help in my case
since the input channel is not a file.  Should I use only the
low-level functions in the Unix module if I _ever_ want "select"?

Thanks,

--
Eijiro Sumii (http://www.cis.upenn.edu/~sumii/)
Department of Computer and Information Science, University of Pennsylvania

Re: [Caml-list] select (or polling) on in_channel?

[Gerd Stolpmann]

Am Donnerstag, den 31.03.2005, 18:28 -0500 schrieb Eijiro Sumii:
> Hi again...
> 
> Is there any easy way to _correctly_ do a "select" (as in the UNIX
> system call) on Pervasives.in_channel?  My naive approach
> 
>   Unix.select [Unix.descr_of_in_channel ic] [] [] 0.
> 
> seems to fail (and it indeed does fail) because of the buffering
> inside Pervasives.in_channel, as long as I checked the source code of
> ocaml-3.08.3.  Pervasives.in_channel_length doesn't help in my case
> since the input channel is not a file.  Should I use only the
> low-level functions in the Unix module if I _ever_ want "select"?

Yes. The point is that you cannot interpret the result of Unix.select in
a reasonable manner. When Unix.select says the descriptor has no data,
it might still the case that there are data in the buffer. If the
descriptor has data, you don't know when they are exactly read into the
buffer.

I don't understand why you mention in_channel_length here. What is your
real problem?

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------

Re: [Caml-list] select (or polling) on in_channel?

[Eijiro Sumii]

Thanks for the reply!

From: "Gerd Stolpmann" <gerd@gerd-stolpmann.de>
> Yes. The point is that you cannot interpret the result of Unix.select in
> a reasonable manner. When Unix.select says the descriptor has no data,
> it might still the case that there are data in the buffer.

Yes, this is what I thought.

> If the descriptor has data, you don't know when they are exactly
> read into the buffer.

Here I don't quite understand what you mean - does this issue per se
prevent us from polling an in_channel?

> I don't understand why you mention in_channel_length here.

If the channel were a (binary) file, I could use in_channel_length to
know how many bytes I should read (by Pervasives.really_input, for
example).

> What is your real problem?

In my program, I need to wait _only for 60 seconds_ for data from a
channel created by Unix.open_connection.  For this purpose, something
like Unix.select would be the best if it worked.

Assuming that no such function like Unix.select exists for
Pervasives.in_channel, I've written the following ad hoc external
function, which can be used (in combination with Unix.select) to
achieve similar effects.  Of course, it might be unsafe if the channel
structure is different in other versions of OCaml.  Any better idea,
anyone...?

	Eijiro

----------------------------------------------------------------------
#include <sys/types.h>
#include <caml/mlvalues.h>

/* adapted from ocaml-3.08.3/byterun/io.h */
struct channel {
  int fd; off_t offset;
  char * end; char * curr; char * max;
  void * mutex; struct channel * next;
  int revealed; int old_revealed; int refcount;
  char buff[0];
};
#define Channel(v) (*((struct channel **) (Data_custom_val(v))))

value in_channel_is_empty(value vc) {
  struct channel *c = Channel(vc);
  return Val_int((c->curr >= c->max) ? 1 : 0);
}

Re: [Caml-list] select (or polling) on in_channel?

[Christoph Bauer]

Hi,

here is a solution that seems to work. The function Stdinbuf.lines
returns a list of available lines. It works with or without
   Unix.select [Unix.stdin] [] [] delay

Christoph Bauer


(* Stdinbuf.ml *)

let () = Unix.set_nonblock Unix.stdin 

let buf = ref (String.create 256) 
let pos = ref 0

let rec lines stdin = 
  let len = String.length !buf in
  let r = 
    try Unix.read stdin !buf !pos (len - !pos) 
    with Unix.Unix_error( Unix.EAGAIN, _ , _ ) -> 0 in 
    pos := r + !pos;
    if !pos >= len then
      let s' = String.create (len*2) in
	String.blit !buf 0 s' 0 len;
	buf := s';
	lines stdin
    else
      try 
	let e = String.rindex_from !buf !pos '\n' in
	let c = String.sub !buf 0 e in
	  String.blit !buf e !buf 0 (!pos-e);
	  pos := !pos - e;
	  StringUtils.split ~sep:'\n' c 
      with Not_found -> []
(* eof *)

For completeness:

(* stringUtils.ml *)
(* ... * )

let split ?(sep = ' ') ?(empty = false) s =
  let rec loop acc i =
    let s, idx =
      try
        let idx = String.rindex_from s i sep in
          String.sub s (idx+1) (i-idx), (idx-1)
      with Not_found ->
        String.sub s 0 (i+1), ~-1
    in let acc' =
        if empty || s <> "" then s::acc
        else acc
      in
        if idx = ~-1 then acc'
        else loop acc' idx 
  in loop [] (String.length s -1)




-- 
let () = let rec f a w i j = Printf.printf "%.20f\r" a; let a1 = a *. i /. j in
if w then f a1 false (i +. 2.0) j else f a1 true i (j +. 2.0) in f 2.0 false 2.0 1.0

Re: [Caml-list] select (or polling) on in_channel?

[Eijiro Sumii]

Thanks to everyone for ideas on how to "select" Pervasives.in_channel.
Since there seems to be no better solution than

  (1) hacking the low-level structure by using external C, or

  (2) reinventing the high-level library on top of Unix I/O,

I'm thinking of submitting a feature wish at
http://pauillac.inria.fr/bin/caml-bugs.  It should be easy enough for
the developers to implement, I hope...  (I could contribute my own
implementation if necessary, but I'm sure they will do better work.)

--
Eijiro Sumii (http://www.cis.upenn.edu/~sumii/)
Department of Computer and Information Science, University of Pennsylvania

Re: [Caml-list] select (or polling) on in_channel?

[Eijiro Sumii]

P.S.  I've also received an e-mail informing me that Cash

  http://pauillac.inria.fr/cash/latest/doc/Cash.html

has such a function.  It looks nice, but having it in pure/core OCaml
would be even nicer.

From: "Eijiro Sumii" <eijiro_sumii@anet.ne.jp>
> Thanks to everyone for ideas on how to "select" Pervasives.in_channel.
> Since there seems to be no better solution than
> 
>   (1) hacking the low-level structure by using external C, or
> 
>   (2) reinventing the high-level library on top of Unix I/O,
> 
> I'm thinking of submitting a feature wish at
> http://pauillac.inria.fr/bin/caml-bugs.  It should be easy enough for
> the developers to implement, I hope...  (I could contribute my own
> implementation if necessary, but I'm sure they will do better work.)

Re: [Caml-list] select (or polling) on in_channel?

[Alex Baretta]

Eijiro Sumii wrote:
> Thanks to everyone for ideas on how to "select" Pervasives.in_channel.
> Since there seems to be no better solution than
> 
>   (1) hacking the low-level structure by using external C, or
> 
>   (2) reinventing the high-level library on top of Unix I/O,
> ...

I have not been following this thread closely, so I did not answer 
earlier. My impression is that select and buffered IO are concepts that 
do not live well together. I This is probably the reason the Caml team 
has for not implementing such a feature in the Pervasives channels API.

I would attempt to model the problem in a multithreaded paradigm, where 
the the "selecting" is implicit in the thread scheduling mechanism. Of 
course, if you allow the use of threads, the selecting on channels 
becomes fairly trivial.

let (++) x f = f x

let input_string =
   let buf =  String.create 1024 in
     fun ic ->
       let bytes = input ic buf 0 1024 in
         String.sub buf 0 bytes

module IO_manager = struct
   type tio_manager = string Event.channel
   let wrap_in_channel (evch : io_manager) (ic : input_channel)=
     let rec cycle () =
       ic ++ input_string ++ Event.send ++ Event.sync;
       cycle ()
     in Thread.create cycle ()
end

Of course, this is a very rough sketch of message_passing API based on 
the Event module, encapsulating the select facility on top of an 
input_channel. Actually, ic could be any resource the program might need 
to synchronize on.

Let me add that Baretta DE&IT is using this paradigm to multiplex IO 
from/to different field buses in its Ocaml based, SoftPLC / SoftCNC kernel.

Alex

-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. +39 02 370 111 55
fax. +39 02 370 111 54

Our technology:

The Application System/Xcaml (AS/Xcaml)
<http://www.asxcaml.org/>

The FreerP Project
<http://www.freerp.org/>

Re: [Caml-list] select (or polling) on in_channel?

[Eijiro Sumii]

From: "Alex Baretta" <alex@barettadeit.com>
> My impression is that select and buffered IO are concepts that do
> not live well together.

Actually, I don't quite think so - buffering can often be regarded an
optimization and select could be implemented just by checking the
buffer in advance (as others have suggested).  I also suppose there is
at least some kind of buffering even inside UNIX OS kernels with a
"select" (or "poll" in Solaris) system call.

> I would attempt to model the problem in a multithreaded paradigm, where 
> the the "selecting" is implicit in the thread scheduling mechanism.

I rather want to avoid multi-threading if select could be easy enough,
because the former is often too tricky due to the possibility of
deadlocks and races.

	Eijiro

Re: [Caml-list] select (or polling) on in_channel?

[Alex Baretta]

Eijiro Sumii wrote:
> From: "Alex Baretta" <alex@barettadeit.com>

> I rather want to avoid multi-threading if select could be easy enough,
> because the former is often too tricky due to the possibility of
> deadlocks and races.
> 
> 	Eijiro

Static determination of potential deadlocks in a message passing 
environment like that implemented in Event amounts to proving that a 
graph is cyclic.

Selecting amounts to manual threading: it gives you more control, but 
not necessarily any more safety.

Alex

-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. +39 02 370 111 55
fax. +39 02 370 111 54

Our technology:

The Application System/Xcaml (AS/Xcaml)
<http://www.asxcaml.org/>

The FreerP Project
<http://www.freerp.org/>

Re: [Caml-list] select (or polling) on in_channel?

[Richard Jones]

On Tue, Apr 05, 2005 at 09:34:39AM -0400, Eijiro Sumii wrote:
> I rather want to avoid multi-threading if select could be easy enough,
> because the former is often too tricky due to the possibility of
> deadlocks and races.

You can implement threads on top of select / poll semantics.  You also
need to have a stack switching operation (such as makecontext /
swapcontext, setjmp / longjmp, or a bit of asm).  Proof by example:
http://www.annexia.org/freeware/pthrlib

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

Re: [Caml-list] select (or polling) on in_channel?

[Eijiro Sumii]

From: "Alex Baretta" <alex@barettadeit.com>
> Static determination of potential deadlocks in a message passing 
> environment like that implemented in Event amounts to proving that a 
> graph is cyclic.

Er, I believe I know it very well...:-) You might want to read our
papers

  - An Implicitly-Typed Deadlock-Free Process Calculus (CONCUR 2000)

and

  - A Generalized Deadlock-Free Process Calculus (HLCL 98)

linked from:

  http://www.cis.upenn.edu/~sumii/pub/

> Selecting amounts to manual threading: it gives you more control, but 
> not necessarily any more safety.

I'm afraid I disagree here: select _can_ be used for multi-threading,
but can also be used for things that are much easier than
multi-threading (which is my case as explained before).

Regards,

--
Eijiro Sumii (http://www.cis.upenn.edu/~sumii/)
Department of Computer and Information Science, University of Pennsylvania

Re: [Caml-list] select (or polling) on in_channel?

[Eijiro Sumii]

From: "Richard Jones" <rich@annexia.org>
> > I rather want to avoid multi-threading if select could be easy enough,
> > because the former is often too tricky due to the possibility of
> > deadlocks and races.
> 
> You can implement threads on top of select / poll semantics.  You also
> need to have a stack switching operation (such as makecontext /
> swapcontext, setjmp / longjmp, or a bit of asm).  Proof by example:
> http://www.annexia.org/freeware/pthrlib

Yes, I think I know - my colleagues (as well as I myself) have even
_taught_ how to implement threads by using select in our classes.

But how is that relevant in this context?  It doesn't mean
multi-threading is always easier than select (in my case, for
instance)...?

	Eijiro