Interestingly, this probably has nothing to do with the size of the buffer. input_char actually acquires and releases a lock for every single call, whether or not an underlying system call is required to fill the buffer. This has always struck me as an odd aspect of the in/out channel implementation, and means that IO is a lot more expensive in a threaded context than it should be.
At Jane Street, performance-sensitive code tends to use other libraries that we've built directly on top of file descriptors that batches the IO and doesn't require constant lock acquisition.
yOn Tue, Feb 17, 2009 at 5:07 AM, Sylvain Le Gall <sylvain@le-gall.net> wrote:On 17-02-2009, Rémi Dewitte <remi@gide.net> wrote:
>
> test.csv is a 21mo file with ~13k rows and a thousands of columns on a 15rp=
> m
> disk.You are using input_char and standard IO channel. This is a good choice
>
> ocaml version : 3.11.0
>
for non-threaded program. But in your case, I will use Unix.read with a
big buffer (32KB to 4MB) and change your program to use it. As
benchmarked by John Harrop, you are spending most of your time in
caml_enter|leave_blocking section. I think it comes from reading using
std IO channel which use 4k buffer. Using a bigger buffer will allow
less call to this two functions (but you won't win time at the end, I
think you will just reduce the difference between non-threaded and
threaded code).
Regards
Sylvain Le Gall
_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs
_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs