Re: [Caml-list] best and fastest way to read lines from a file?

[kirillkh <kirillkh@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3083 bytes --]

Replying to a private mail from Brian:

2007/10/2, Brian Hurt <bhurt@janestcapital.com>:
>
> kirillkh wrote:
>
>
>  This should be a FAQ.
> >
>
> Since we're talking of 10+ lines of code and only one case among many
> possible (you might also want to do something fairly similar, but not quite
> the same, as iterating over all words or characters in a file, doing
> something else than counting, etc.), I would rather see it implemented in a
> library as combinator. What I have in mind is a function that goes over a
> file and invokes some user code on each block of
> bytes/characters/lines/words/... The points of customization would be:
> * how to detect the start and end of block
> * routine to pass the blocks to
>
> Then, on top of this combinator, build block-specific ones: for byte,
> char, line, word blocks. Also make it possible to customize buffering
> behavior.
>
> Being new to OCaml, I'm interested in comments - is what I suggest a good
> idea? If yes, why hasn't anyone implemented it yet? One could argue that
> it's trivial and can be implemented in each use case anew, but among 5
> different solutions posted so far, each has its own problems, besides the
> supposedly ideal 5-th -- I'd take this as an indication that, although
> simple, it's not really trivial to write this thing.
>
> Note that the fifth version is only perfect because the author knew that
> he didn't need to keep the line around.  Doing something even moderately
> more complicated would almost certainly require an allocation in the main
> loop- not that this is that big of a problem (Ocaml allocation is lightning
> fast).
>
> The problem with this library is when to stop.  A real simple "fold over
> the lines of a file" interface might be nice, but there'll always be
> pressure to add just one more feature, deal with just one more slightly more
> complex case- at the end of which you get a badly specified and badly
> implemented parser combinator library.
>
> Brian
>

OK, so I'll give up the parsing/buffering part and only leave efficient
exception handling. This should leave the user free to do anything with it,
but prevent performance pitfalls. The following is based on Mattias
Engdegard's code:

(* I couldn't figure out, how to declare a polymorphic exception properly *)
exception Done of 'a

let fold_file (file: in_channel)
              (read_func: in_channel->'a)
              (elem_func: 'a->'b->'b)
              (seed: 'b) =
  let rec loop prev_val =
    let input =
      try read_func file
      with End_of_file -> raise (Done prev_val)
    in
      let combined_val = elem_func input prev_val in
      loop combined_val
  in
    try loop seed with Done x -> x

And the usage for line counting:

let line_count filename =
   let f = open_in filename in
   let counter _ count = count + 1 in
   fold_file f readline counter 0

Since it's library code, we can tolerate the little annoyance of the second
try-catch. As far as I can tell, this code has the same performance
characteristics as yours: no consing + tail recursion. Any other problems
with it?

[-- Attachment #2: Type: text/html, Size: 4600 bytes --]