[Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Richard Jones]

Dear OCaml users,

I'm pretty convinced that having the input* functions and readdir
throw End_of_file when they run out of data is wrong. Instead
'readdir' (for example) should be prototyped as:

readdir : dir_handle -> string option

and the functions should return None when they run out of data.
Either that or alternative forms which don't throw exceptions should
be available.

The reason is it makes input* and readdir very hard to use since they
are throwing exceptions in what is essentially an un-exceptional case
(it's _normal_, not exceptional, to keep reading data until the end of
the stream).

Take a look at the attached code. I've wrapped up 'readdir' in a
function which hides the annoying exception (or course, this is
inefficient, but I don't have much choice). The code is reasonably
easy to understand. Please someone show me the elegant solution to
this problem which doesn't involve hiding the exception ...

An alternative would be, like Perl, to provide alternative forms
of input_line and readdir (in particular) which return a list of
lines/names in a single go, rather like:

  my @files = readdir DIR;

in Perl.

(Another minor point - do I need to call closedir, or will the garbage
collector do that for me?)

Rich.

----------------------------------------------------------------------
open Unix

type filesystem = File of string | Directory of filesystem list;;

let readdir_no_ex dirh =
  try
    Some (readdir dirh)
  with
    End_of_file -> None
  ;;

let rec read_directory path =
  let dirh = opendir path in
  let rec loop () =
    let filename = readdir_no_ex dirh in
    match filename with
      None -> []
    | Some "." -> loop ()
    | Some ".." -> loop ()
    | Some filename ->
	let pathname = path ^ "/" ^ filename in
	let stat = lstat pathname in
	let this = if stat.st_kind = S_DIR then
	  Directory (read_directory pathname)
	else
	  File pathname in
	this :: loop ()
  in
  loop ()
  ;;

let path = Sys.argv.(1);;
let fs = read_directory path;;


-- 
Richard Jones, Red Hat Inc. (London) and Merjis Ltd. http://www.merjis.com/
http://www.annexia.org/ Freshmeat projects: http://freshmeat.net/users/rwmj
NET::FTPSERVER is a full-featured, secure, configurable, database-backed
FTP server written in Perl: http://www.annexia.org/freeware/netftpserver/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Michal Moskal]

On Fri, Jun 06, 2003 at 06:01:32PM +0100, Richard Jones wrote:
> (Another minor point - do I need to call closedir, or will the garbage
> collector do that for me?)

Even if it does, I wouldn't relay on it -- you never know when it runs,
and if your objects are actually GCed. So if there is risk of running
out of file descriptors -- close it.

-- 
: Michal Moskal :: http://www.kernel.pl/~malekith : GCS {C,UL}++++$ a? !tv
: PLD Linux ::::::::: Wroclaw University, CS Dept : {E-,w}-- {b++,e}>+++ h

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Ville-Pertti Keinonen]

> I'm pretty convinced that having the input* functions and readdir
> throw End_of_file when they run out of data is wrong. Instead
> 'readdir' (for example) should be prototyped as:
>
> readdir : dir_handle -> string option
>
> and the functions should return None when they run out of data.
> Either that or alternative forms which don't throw exceptions should
> be available.

I don't think it's necessarily such a clear requirement, unless you are 
assuming the interface for some particular language.

Since exceptions are reasonably inexpensive in O'Caml, and used for 
things such as Not_found in basic collection types, using exceptions to 
indicate the end of some particular set of values seems reasonable, to 
me.

Writing a try ... with statement isn't really that different from a 
match statement except for the fact that you can handle exceptional 
circumstances generated by several expressions in the with ... part of 
a try ... with statement - if anything, that's more permissive.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Richard Jones]

On Fri, Jun 06, 2003 at 09:03:38PM +0300, Ville-Pertti Keinonen wrote:
> Writing a try ... with statement isn't really that different from a 
> match statement except for the fact that you can handle exceptional 
> circumstances generated by several expressions in the with ... part of 
> a try ... with statement - if anything, that's more permissive.

The problem is that there doesn't seem to be a way to write the
loop function using readdir. eg:

  let rec loop () =
    let filename = readdir dirh in
    match filename with
    | "." -> loop ()
    | ".." -> loop ()
    | filename ->
	let pathname = path ^ "/" ^ filename in
	let stat = lstat pathname in
	let this = if stat.st_kind = S_DIR then
	  read_directory pathname
	else
	  File pathname in
	this :: loop ()
  in
  try
    Directory (loop ())
  with
    End_of_file ->  XXX what?

Because the exception is always raised (it's not an exception at all)
there's no way to return the result of the call to loop ().

Rich.

-- 
Richard Jones, Red Hat Inc. (London) and Merjis Ltd. http://www.merjis.com/
http://www.annexia.org/ Freshmeat projects: http://freshmeat.net/users/rwmj
C2LIB is a library of basic Perl/STL-like types for C. Vectors, hashes,
trees, string funcs, pool allocator: http://www.annexia.org/freeware/c2lib/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Brian Hurt]

On Fri, 6 Jun 2003, Ville-Pertti Keinonen wrote:

> Since exceptions are reasonably inexpensive in O'Caml, and used for 
> things such as Not_found in basic collection types, using exceptions to 
> indicate the end of some particular set of values seems reasonable, to 
> me.
> 
> Writing a try ... with statement isn't really that different from a 
> match statement except for the fact that you can handle exceptional 
> circumstances generated by several expressions in the with ... part of 
> a try ... with statement - if anything, that's more permissive.
> 

The problem with try/with is that it's way to easy to break tail recursion 
using try/with.  About every other week someone comes to the Ocaml 
beginners list where they are doing something like:

let rec read_all_lines chan accum =
   try
       let line = input_line chan in
       read_all_lines chan (line :: accum)
   with
      End_of_file -> List.rev accum

Unfortunately, this is *not* tail recursive.  Works fine on small files,
but when they try to run it on large files they stack overflow.  Which
means you need to contort to take the recursion out of the try/with block.
Several different approaches to this are possible- but none with the 
obviousness of the above code.

Personally, I'd like the following to be an efficient way to read all the 
lines from a file into a list:

let rec read_all_lines chan =
     try
         let line = input_line chan in
         line :: (read_all_lines chan)
     with
         End_of_file -> []

and have the compiler contort the code into an efficient, tail-recursive 
form, with a combination of data structures with a hole and catch removal.

Brian


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Richard Jones]

On Fri, Jun 06, 2003 at 01:49:55PM -0500, Brian Hurt wrote:
> The problem with try/with is that it's way to easy to break tail recursion 
> using try/with.  About every other week someone comes to the Ocaml 
> beginners list where they are doing something like:

What's a good way to read all the lines of a file into a list? I
can put it into the OCaml tutorial I'm writing.

Rich.

-- 
Richard Jones, Red Hat Inc. (London) and Merjis Ltd. http://www.merjis.com/
http://www.annexia.org/ Freshmeat projects: http://freshmeat.net/users/rwmj
NET::FTPSERVER is a full-featured, secure, configurable, database-backed
FTP server written in Perl: http://www.annexia.org/freeware/netftpserver/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Re: Why do input* and readdir throw End_of_file ... annoying!

[Alan Post]

In article <20030606185111.GA3545@redhat.com>, Richard Jones wrote:
> On Fri, Jun 06, 2003 at 01:49:55PM -0500, Brian Hurt wrote:
>> The problem with try/with is that it's way to easy to break tail recursion 
>> using try/with.  About every other week someone comes to the Ocaml 
>> beginners list where they are doing something like:
> 
> What's a good way to read all the lines of a file into a list? I
> can put it into the OCaml tutorial I'm writing.

Use ExtLib, of course.  :)

List.of_enum (input_enum chan)

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Brian Hurt]

On Fri, 6 Jun 2003, Richard Jones wrote:

> On Fri, Jun 06, 2003 at 01:49:55PM -0500, Brian Hurt wrote:
> > The problem with try/with is that it's way to easy to break tail recursion 
> > using try/with.  About every other week someone comes to the Ocaml 
> > beginners list where they are doing something like:
> 
> What's a good way to read all the lines of a file into a list? I
> can put it into the OCaml tutorial I'm writing.

The way I'd recommend doing it:

let read_all_lines chan =
    let rec loop accum =
        let line, eof = try (input_line chan), true
                        with End_of_file -> "", false
        in
        if eof then
            accum
        else
            loop (line :: accum)
    in
    List.rev (loop [])

The original poster's idea of a function:
   let my_input_line chan = 
      try Some(input_line chan)
      with End_of_file -> None

has the advantage of enlisting the type checker to make sure you handle 
the eof condition correctly everywhere.

For those seeking best performance, using the ExtLib would allow you to do 
the following:

let read_all_lines chan
    let e = Enum.from (fun () -> try input_line chan
                                 with End_of_file -> 
                                     raise No_more_elements)
    in
    ExtList.of_enum e

This has the effect of eliminating the List.rev.  Although at this point 
you should be very tempted to just keep it an enum, which are in some ways 
easier to work with than lists, and you don't read the data until you need 
it.  For example, to read in all the lines of one file, make them upper 
case, and write them out to another file, you might write:

let cap_all_lines inchan outchan
    let e = Enum.from (fun () -> try input_line inchan
                                 with End_of_file -> 
                                     raise No_more_elements)
    in
    let e' = Enum.map String.capitalize e
    in
    Enum.iter (fun x -> output_string outchan x; output_char '\n') e'

The advatange of this method is that only one line at a time needs to be 
in memory.

Brian


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Lauri Alanko]

On Fri, Jun 06, 2003 at 01:49:55PM -0500, Brian Hurt wrote:
> The problem with try/with is that it's way to easy to break tail recursion 
> using try/with.  About every other week someone comes to the Ocaml 
> beginners list where they are doing something like:
> 
> let rec read_all_lines chan accum =
>    try
>        let line = input_line chan in
>        read_all_lines chan (line :: accum)
>    with
>       End_of_file -> List.rev accum

This is a good argument for the alternative exception handling syntax
proposed by in "Exceptional syntax", JFP 11(4):395-410,
<http://research.microsoft.com/~nick/except.ps>. They suggest a form

try x <= e1 in e2 unless E => e3

which acts like "let x = e1 in e2" when e1 causes no exceptions, but
otherwise goes directly to the handler and skips e2 entirely.

So the function above could be written:

let rec read_all_lines chan accum =
   try
       line <= input_line chan 
   in
       read_all_lines chan (line :: accum)
   unless
       End_of_file => List.rev accum

It looks almost the same, but here the scope of the handler does _not_
extend over the recursive call, only the binding

Actually, as pointed out by the article, this thing has been suggested
on caml-list before: <http://caml.inria.fr/archives/199912/msg00078.html>.


Lauri Alanko
la@iki.fi

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Eric C. Cooper]

On Fri, Jun 06, 2003 at 07:43:04PM +0100, Richard Jones wrote:
> The problem is that there doesn't seem to be a way to write the
> loop function using readdir. eg:
> 
>   let rec loop () =
>     let filename = readdir dirh in
>     match filename with
>     | "." -> loop ()
>     | ".." -> loop ()
>     | filename ->
> 	let pathname = path ^ "/" ^ filename in
> 	let stat = lstat pathname in
> 	let this = if stat.st_kind = S_DIR then
> 	  read_directory pathname
> 	else
> 	  File pathname in
> 	this :: loop ()
>   in
>   try
>     Directory (loop ())
>   with
>     End_of_file ->  XXX what?
> 
> Because the exception is always raised (it's not an exception at all)
> there's no way to return the result of the call to loop ().

Define your loop to take an accumulator (a list of the results so far):

let rec read_filesystem path =
  if (lstat path).st_kind = S_DIR then
    Directory (read_directory pathname)
  else
    File path

and read_directory path =
  let dirh = opendir path in
  let rec loop entries =
    try match readdir dirh with
    | "." | ".." -> loop entries
    | filename -> loop (read_filesystem filename :: entries)
    with End_of_file -> entries
  in
  let list = loop [] in
  closedir dirh;
  list

-- 
Eric C. Cooper          e c c @ c m u . e d u

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[David Brown]

On Sat, Jun 07, 2003 at 11:08:54AM -0400, Eric C. Cooper wrote:

> and read_directory path =
>   let dirh = opendir path in
>   let rec loop entries =
>     try match readdir dirh with
>     | "." | ".." -> loop entries
>     | filename -> loop (read_filesystem filename :: entries)
>     with End_of_file -> entries
>   in
>   let list = loop [] in
>   closedir dirh;
>   list

But, this isn't tail recursive, so you might as well not pass the
argument and build up the list upon returning.  Is there a way of making
it really tail recursive, while using the exception?  It isn't too hard
with a list ref to accumulate the results.

Dave

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[art yerkes]

On Sat, 7 Jun 2003 13:27:11 -0700
David Brown <caml-list@davidb.org> wrote:

> On Sat, Jun 07, 2003 at 11:08:54AM -0400, Eric C. Cooper wrote:
> 
> > and read_directory path =
> >   let dirh = opendir path in
> >   let rec loop entries =
> >     try match readdir dirh with
> >     | "." | ".." -> loop entries
> >     | filename -> loop (read_filesystem filename :: entries)
> >     with End_of_file -> entries
> >   in
> >   let list = loop [] in
> >   closedir dirh;
> >   list
> 
> But, this isn't tail recursive, so you might as well not pass the
> argument and build up the list upon returning.  Is there a way of making
> it really tail recursive, while using the exception?  It isn't too hard
> with a list ref to accumulate the results.
> 
> Dave



how about:

open Unix

let read_fs starting_dir =
  let rec go_until_exception list_ref f =
    list_ref := (f ()) :: !list_ref ; go_until_exception list_ref f
  and dirs_ref = ref [] in
  let the_dir = opendir starting_dir in
  try
    go_until_exception dirs_ref (fun () -> readdir the_dir)
  with End_of_file -> !dirs_ref
 
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners


-- 
`No, you don't understand,' the Knight said, looking a little vexed. 
`That's what the name is called. The name really is "The Aged Aged 
Man."'
-- Lewis Carroll

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Fabrice Le Fessant]

>  On Fri, Jun 06, 2003 at 07:43:04PM +0100, Richard Jones wrote:
>  > The problem is that there doesn't seem to be a way to write the
>  > loop function using readdir. eg:

Why don't you use the while ... do ... done loop instead with an
external reference ?

let list = ref [] in
let dir = opendir "......." in
try
  while true do
    match readdir dir with
      "." | ".." -> ()
    | filename -> list := filename :: !list
  done
with End_of_file -> !list

Recursive functions are elegant to use, but sometimes, it is simpler
to use simple loops instead.

- Fabrice


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Why do input* and readdir throw End_of_file ... annoying!

[Brian Hurt]

On Sat, 7 Jun 2003, David Brown wrote:

> On Sat, Jun 07, 2003 at 11:08:54AM -0400, Eric C. Cooper wrote:
> 
> > and read_directory path =
> >   let dirh = opendir path in
> >   let rec loop entries =
> >     try match readdir dirh with
> >     | "." | ".." -> loop entries
> >     | filename -> loop (read_filesystem filename :: entries)
> >     with End_of_file -> entries
> >   in
> >   let list = loop [] in
> >   closedir dirh;
> >   list
> 
> But, this isn't tail recursive, so you might as well not pass the
> argument and build up the list upon returning.  Is there a way of making
> it really tail recursive, while using the exception?  It isn't too hard
> with a list ref to accumulate the results.

let read_directory path =
    let dirh = opendir path in
    let rec loop entries =
        let fname, eof = try (readdir dirh), false
                         with End_of_file -> "", true
        in
        if eof then List.rev entries else
        match fname with
            | "." | ".." -> loop entries
            | _ -> loop (fname :: entries)
    in
    loop []

I will also take this opportunity to plug ExtLib.  Using ExtLib you can 
remove the List.rev call, and do something like:

let read_directory path =
    let dirh = opendir path in
    let rec f () = try 
                   match readdir dirh with
                       | "." | ".." -> f()
                       | s -> s
                   with End_of_file -> raise No_more_elements
    in
    ExtList.of_enum (Enum.from f)

Here f is not tail recursive- but it only recurses to get past "." and 
".." files, so it'll never go more than 3 levels deep.  If it bugs you, 
rewrite f in a manner to my first example.

Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners