From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.6 required=5.0 tests=HTML_00_10,HTML_MESSAGE autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by yquem.inria.fr (Postfix) with ESMTP id 41CD7BBCA for ; Tue, 18 Mar 2008 00:30:01 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhEBACOb3kdC+Vqzhmdsb2JhbACCOjiODAEBAQgEBAkIGpIJhH0 X-IronPort-AV: E=Sophos;i="4.25,514,1199660400"; d="scan'208";a="23883786" Received: from ik-out-1112.google.com ([66.249.90.179]) by mail4-smtp-sop.national.inria.fr with ESMTP; 18 Mar 2008 00:30:00 +0100 Received: by ik-out-1112.google.com with SMTP id b35so2175439ika.3 for ; Mon, 17 Mar 2008 16:30:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=9fblSfcULEYjsbxXlK4hjjkkohKpeMjpt/jtNiQvdmA=; b=E9jB/eBCFmTOvEWv2DMyGLbdZcIdgwM419FLqB9vpzjLZX4/YjsVWzOSA7aeh4wCou8o9tZChbfyq8ePK2USyi8JbPDve1aGUudUfEg6F3/b6Q9vHtVMpL0Jnyt4NArDbOWET1V8gu93iw919magSt66CtQrLrk7mqBcpkgO/78= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=uMMFCCQ4uOWI5NXdxcxvkfmzfP3EDRgDLWH/zG6xbfPp/NDZCXuibDoxnUpO17jIbvBuQ0yJhl4/g1rr6Iz2iuyB3ngnrrWjvKMeTbvgYOWuFUn22riG89Kcv/sIejT48JQlkB+ehBCDV3v8ByPKtfGxZerhEPvKmCnTg42nasg= Received: by 10.150.123.16 with SMTP id v16mr492663ybc.156.1205796598971; Mon, 17 Mar 2008 16:29:58 -0700 (PDT) Received: by 10.150.216.4 with HTTP; Mon, 17 Mar 2008 16:29:58 -0700 (PDT) Message-ID: <8cc3d8520803171629o497c4838q4ed03534bc3becfc@mail.gmail.com> Date: Mon, 17 Mar 2008 19:29:58 -0400 From: "viktor tron" To: caml-list Subject: copy the rest of a file after scanning MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_7959_1454377.1205796598964" X-Spam: no; 0.00; edits:01 buffer:01 chunks:01 printf:01 fprintf:01 chunks:01 buf:01 buf:01 printf:01 fprintf:01 incorrectly:01 crlf:01 edits:01 buffer:01 incorrectly:01 ------=_Part_7959_1454377.1205796598964 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Dear list, I have the funniest problem. I use Scan to scan a file, and output an edited variant, when edits are done I just need to copy the remainder of the file. This ludicrious task is proving more elusive to handle than the whole project. The problem is that when I finish editing, the scanning buffer is active and not empty and my reading position in the input channel is not where I am currently at in scanning. I don't want to use scanner to consume the remaining 3 terrabytes digesting it line by line. I can't seem to be able to scan bigger chunks than lines either unless I can name a character that certainly does not appear in the text I am reading. If there is a character like this, say @, then I am ok with 1) scan "%s@@%!" (fun s -> Printf.fprintf corpus_out "%s" s) this one reads till the next @ character which is ignored or the end of the input, which is checked with putting %! explicitly. This passes my tests, but horribly ugly, since there is no character that I can guarantee this way. Plus I might not have memory for passing this whole chunk as one string if the file is large. So as an alternative I did this: 2) (* we set the input channel reading position to where we are in scanning *) let _ = scan "%n" (fun x -> seek_in corpus_in (x - 1)) in (* and then dump the rest trivially in chunks of buf_size chars *) let buf = String.create buf_size in let rec dump () = let len = input corpus_in buf 0 buf_size in if len > 0 then (output corpus_out buf 0 len; dump () ) in or in one go 3) let end_pos = in_channel_length corpus_in let len = end_pos - pos_in corpus_in in let s = String.create len in let _ = really_input corpus_in s 0 len in Printf.fprintf corpus_out "%s" s; On my mac and linux, all works smoothly, till I used it on windows. 3) does not work on windows, since in_channel_length and seek do not take into account the newline translations that take place at reading. Or in other words, the scan module reports character positions incorrectly since CRLF=\013\010 is counted as one character and matched by \n. But then again, 2 seems to work, but have no clue why seek is not the same as in_channel_length when it comes to counting chars. In which case it should not work either. Any thoughts? Viktor ------=_Part_7959_1454377.1205796598964 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Dear list,
I have the funniest problem.

I use Scan to scan a file, and output an edited variant, when edits are done
I just need to copy the remainder of the file.

This ludicrious task is proving more elusive to handle than the whole project.

The problem is that when I finish editing, the scanning buffer is active and not empty and
my reading position in the input channel is not where I am currently at in scanning.


I don't want to use scanner to consume the remaining 3 terrabytes digesting
it line by line. I can't seem to be able to scan bigger chunks than lines either unless I can name a character
that certainly does not appear in the text I am reading.
If there is a character like this, say @, then I am ok with

1)
scan "%s@@%!" (fun s -> Printf.fprintf corpus_out "%s" s)

this one reads till the next @ character which is ignored or the end of the input, which is checked with putting %! explicitly.
This passes my tests, but horribly ugly, since there is no character that I can guarantee this way.
Plus I might not have memory for passing this whole chunk as one string if the file is large.

So as an alternative I did this:

2)
(* we set the input channel reading position to where we are in scanning *)
let _ = scan "%n" (fun x -> seek_in corpus_in (x - 1)) in
(* and then dump the rest trivially in chunks of buf_size chars *)
let buf = String.create buf_size in
let rec dump () =
let len = input corpus_in buf 0 buf_size in
if len > 0 then (output corpus_out buf 0 len; dump () )
in

or in one go

3)
let end_pos = in_channel_length corpus_in
let len = end_pos - pos_in corpus_in in
let s = String.create len in
let _ = really_input corpus_in s 0 len in
Printf.fprintf corpus_out "%s" s;

On my mac and linux, all works smoothly, till I used it on windows.
3) does not work on windows, since in_channel_length and seek do not take into account the newline translations that take
place at reading.
Or in other words, the scan module reports character positions incorrectly since CRLF=\013\010 is counted as one character and matched by \n.

But then again, 2 seems to work, but have no clue why seek is not the same as in_channel_length when it comes to counting chars.
In which case it should not work either.

Any thoughts?

Viktor

------=_Part_7959_1454377.1205796598964--