From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.9 required=5.0 tests=HTML_10_20,HTML_MESSAGE autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by yquem.inria.fr (Postfix) with ESMTP id A11EBBBCA for ; Mon, 17 Mar 2008 19:00:51 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ai8FADNP3kfAXQIm/2dsb2JhbACCOzegbYRe X-IronPort-AV: E=Sophos;i="4.25,513,1199660400"; d="scan'208";a="8493306" Received: from discorde.inria.fr ([192.93.2.38]) by mail2-smtp-roc.national.inria.fr with ESMTP; 17 Mar 2008 19:00:39 +0100 Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by discorde.inria.fr (8.13.6/8.13.6) with ESMTP id m2HI0cJe023879 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=OK) for ; Mon, 17 Mar 2008 19:00:39 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlABAH5O3kfY7zq4gWdsb2JhbACCOzeODAEBCQQECwYYkiCEXg X-IronPort-AV: E=Sophos;i="4.25,513,1199660400"; d="scan'208";a="10369650" Received: from gv-out-0910.google.com ([216.239.58.184]) by mail3-smtp-sop.national.inria.fr with ESMTP; 17 Mar 2008 19:00:38 +0100 Received: by gv-out-0910.google.com with SMTP id i36so1023968gve.28 for ; Mon, 17 Mar 2008 11:00:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=qFPZLyjJGWLKjaMFyeHD/fXnzdy0J2aTD5mUWY0HQgk=; b=jahL+7qfjom9RwXmY8FukMYnz350x8Ch8WbFZ1X0XefWAfwOhr+BsJbTjcVKg6u19MuI20d0e+NIYy+qJFcrxYuujSk02CvsiaZZhOnx8m+B6soUeDD1bylxrhKs9u6UWOefdr3shwHAIdqXaFLbPNn7ffQnhVsArmxE9LH4c6k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=Q3gO5j8XKK8CJkPbOF34pSpgFyxtI3jjpny8cdThJvpqP6KYYCMSdj8rsvLyGdTNvsX+/eKmsit+A9daACwi74Vb2ZLfBHctT5uW0rk1JEiFbY8vW6II8disiM+tiN9IuuaFG47elfT/y7E+KNav2fjvKD+jz5hfTXD8y3KNek0= Received: by 10.150.49.15 with SMTP id w15mr300588ybw.101.1205776836338; Mon, 17 Mar 2008 11:00:36 -0700 (PDT) Received: by 10.150.216.4 with HTTP; Mon, 17 Mar 2008 11:00:35 -0700 (PDT) Message-ID: <8cc3d8520803171100q55ea2229uec70b0eb92c8173f@mail.gmail.com> Date: Mon, 17 Mar 2008 14:00:35 -0400 From: "viktor tron" To: caml-list@inria.fr Subject: copy the rest of a file after scanning MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_6896_30429301.1205776836329" X-Miltered: at discorde with ID 47DEB1C6.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; edits:01 buffer:01 chunks:01 printf:01 fprintf:01 chunks:01 buf:01 buf:01 printf:01 fprintf:01 incorrectly:01 crlf:01 buffer:01 trivial:01 edits:01 ------=_Part_6896_30429301.1205776836329 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Dear list, I have the funniest problem. I use Scan to scan a file, and output an edited variant, when edits are done I just need to copy the remainder of the file. This ludicrious task is proving more elusive to handle than the whole project. The problem is that when I finish editing, the scanning buffer is active and not empty and my reading position in the input channel is not where I am currently at in scanning. I don't want to use scanner to consume the remaining 3 terrabytes digesting it line by line. I can't seem to be able to scan bigger chunks than lines either unless I can name a character that certainly does not appear in the text I am reading. If there is a character like this, say @, then I am ok with 1) scan "%s@@%!" (fun s -> Printf.fprintf corpus_out "%s" s) this one reads till the next @ character which is ignored or the end of the input, which is checked with putting %! explicitly. This passes my tests, but horribly ugly, since there is no character that I can guarantee this way. Plus I might not have memory for passing this whole chunk as one string if the file is large. So as an alternative I did this: 2) (* we set the input channel reading position to where we are in scanning *) let _ = scan "%n" (fun x -> seek_in corpus_in (x - 1)) in (* and then dump the rest trivially in chunks of buf_size chars *) let buf = String.create buf_size in let rec dump () = let len = input corpus_in buf 0 buf_size in if len > 0 then (output corpus_out buf 0 len; dump () ) in or in one go 3) let end_pos = in_channel_length corpus_in let len = end_pos - pos_in corpus_in in let s = String.create len in let _ = really_input corpus_in s 0 len in Printf.fprintf corpus_out "%s" s; On my mac and linux, all works smoothly, till I used it on windows. These do not work on windows, since in_channel_length and seek do not take into account the newline translations that take place at reading. Or in other words, the scan module reports character positions incorrectly since CRLF=\013\010 is counted as one character and matched by \n. I suspect 2 does not work either, since seek is probably the same as in_channel_length when it comes to counting chars. So there is no way to combine scan positions and in_channel/seek type positions. If there was a way to dump and empty the scanning buffer, then I could then just use (2), since then scanning pos and pos_in would align, but I found no way of doing that. I have no idea how to solve this. Well, I guess I am missing something trivial. Thanks for help Viktor ------=_Part_6896_30429301.1205776836329 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Dear list,
I have the funniest problem.

I use Scan to scan a file, and output an edited variant, when edits are done
I just need to copy the remainder of the file. 

This ludicrious task is proving more elusive to handle than the whole project.

The problem is that when I finish editing, the scanning buffer is active and not empty and
my reading position in the input channel is not where I am currently at in scanning.


I don't want to use scanner to consume the remaining 3 terrabytes digesting 
it line by line. I can't seem to be able to scan bigger chunks than lines either unless I can name a character
that certainly does not appear in the text I am reading.
If there is a character like this, say @, then I am ok with

1)
scan "%s@@%!" (fun s -> Printf.fprintf corpus_out "%s" s)

this one reads till the next @ character which is ignored or the end of the input, which is checked with putting %! explicitly.
This passes my tests, but horribly ugly, since there is no character that I can guarantee this way.
Plus I might not have memory for passing this whole chunk as one string if the file is large.

So as an alternative I did this:

2)
(* we set the input channel reading position to where we are in scanning *)
let _ = scan "%n" (fun x -> seek_in corpus_in (x - 1)) in
(* and then dump the rest trivially in chunks of buf_size chars *)
let buf = String.create buf_size in
let rec dump () =
let len = input corpus_in buf 0 buf_size in
if len > 0 then (output corpus_out buf 0 len; dump () )
in

or in one go

3)
let end_pos = in_channel_length corpus_in
let len = end_pos - pos_in corpus_in in
let s = String.create len in
let _ = really_input corpus_in s 0 len in
Printf.fprintf corpus_out "%s" s;

On my mac and linux, all works smoothly, till I used it on windows.
These do not work on windows, since in_channel_length and seek do not take into account the newline translations that take
place at reading.
Or in other words, the scan module reports character positions incorrectly since CRLF=\013\010 is counted as one character and matched by \n.
I suspect 2 does not work either, since seek is probably the same as in_channel_length when it comes to counting chars.

So there is no way to combine scan positions and in_channel/seek type positions.

If there was a way to dump and empty the scanning buffer, then I could then just use (2), since
then scanning pos and pos_in would align, but I found no way of doing that.

I have no idea how to solve this. Well, I guess I am missing something trivial.

Thanks for help

Viktor



------=_Part_6896_30429301.1205776836329--