From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 3D89E7F946 for ; Fri, 20 Jun 2014 14:29:01 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jean-vincent.loddo@lipn.univ-paris13.fr) identity=pra; client-ip=194.254.163.15; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jean-vincent.loddo@lipn.univ-paris13.fr"; x-sender="jean-vincent.loddo@lipn.univ-paris13.fr"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jean-vincent.loddo@lipn.univ-paris13.fr) identity=mailfrom; client-ip=194.254.163.15; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jean-vincent.loddo@lipn.univ-paris13.fr"; x-sender="jean-vincent.loddo@lipn.univ-paris13.fr"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail.lipn.univ-paris13.fr) identity=helo; client-ip=194.254.163.15; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jean-vincent.loddo@lipn.univ-paris13.fr"; x-sender="postmaster@mail.lipn.univ-paris13.fr"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AoUBAK0npFPC/qMPl2dsb2JhbABZg1+DR8IFDwEBAQEBCBYHPIQtBBFVIQImAl8biD6dRo8jnjoTBIEqkEqBTASaRJceOw X-IPAS-Result: AoUBAK0npFPC/qMPl2dsb2JhbABZg1+DR8IFDwEBAQEBCBYHPIQtBBFVIQImAl8biD6dRo8jnjoTBIEqkEqBTASaRJceOw X-IronPort-AV: E=Sophos;i="5.01,513,1400018400"; d="scan'208";a="81122950" Received: from gw.lipn.univ-paris13.fr (HELO mail.lipn.univ-paris13.fr) ([194.254.163.15]) by mail2-smtp-roc.national.inria.fr with ESMTP; 20 Jun 2014 14:29:00 +0200 Received: from webmail.lipn.univ-paris13.fr (lipn-webmail [10.10.0.93]) by mail.lipn.univ-paris13.fr (Postfix) with ESMTP id 680102613A7 for ; Fri, 20 Jun 2014 14:29:00 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Fri, 20 Jun 2014 14:29:00 +0200 From: jean-vincent.loddo@lipn.univ-paris13.fr To: Message-ID: X-Sender: loddo@lipn.univ-paris13.fr User-Agent: Roundcube Webmail/0.5.2 X-Validation-by: jean-vincent.loddo@lipn.univ-paris13.fr Subject: [Caml-list] Memory leaks generated by Scanf.fscanf? Hi, working on Marionnet (https://launchpad.net/marionnet), I noticed a serious memory leak making the system unusable after a few tens of minutes. After investigation, the problem seems to be related to Scanf.fscanf. This hypothesis is confirmed by the fact that by replacing it with the composition of Pervasives.input_line and Scanf.sscanf, the problem disappears. I tried to write a simple code (I put it at the end of this message) that illustrates the problem with a thread scanning the content of a file repetitively (with fscanf or sscanf). The result is the same with OCaml 3.12.1 or 4.01.0: with sscanf (~with_sscanf:true) the function `start_thread' shows that data are successfully collected: Iteration #01: stat called 256 times: live blocks: 109590 Iteration #02: stat called 256 times: live blocks: 103063 Iteration #03: stat called 256 times: live blocks: 104091 Iteration #04: stat called 256 times: live blocks: 105119 Iteration #05: stat called 256 times: live blocks: 106147 Iteration #06: stat called 256 times: live blocks: 107175 Iteration #07: stat called 256 times: live blocks: 108203 Iteration #08: stat called 256 times: live blocks: 99637 Iteration #09: stat called 256 times: live blocks: 100665 Iteration #10: stat called 256 times: live blocks: 101693 Iteration #11: stat called 256 times: live blocks: 102721 Iteration #12: stat called 256 times: live blocks: 103749 Iteration #13: stat called 256 times: live blocks: 99637 Iteration #14: stat called 256 times: live blocks: 100665 Iteration #15: stat called 256 times: live blocks: 101693 With fscanf however the used memory continues to grow (things are apparently not collected, even if they should): Iteration #01: stat called 256 times: live blocks: 114469 Iteration #02: stat called 256 times: live blocks: 107613 Iteration #03: stat called 256 times: live blocks: 111456 Iteration #04: stat called 256 times: live blocks: 115299 Iteration #05: stat called 256 times: live blocks: 118890 Iteration #06: stat called 256 times: live blocks: 116601 Iteration #07: stat called 256 times: live blocks: 120235 Iteration #08: stat called 256 times: live blocks: 123925 Iteration #09: stat called 256 times: live blocks: 127615 Iteration #10: stat called 256 times: live blocks: 131179 Iteration #11: stat called 256 times: live blocks: 135023 Iteration #12: stat called 256 times: live blocks: 135583 Iteration #13: stat called 256 times: live blocks: 140450 Iteration #14: stat called 256 times: live blocks: 144197 Iteration #15: stat called 256 times: live blocks: 147790 Sorry if the problem is well known or if something is wrong in my analysis, but I can not find anything about it on this list neither on the net. Best regards, Jean-Vincent Loddo --- let stat_with_fscanf pid = let filename = Printf.sprintf "/proc/%d/stat" pid in try let ch = open_in filename in let result = try let obj = Scanf.fscanf ch "%d %s %c %d %d %s@\n" (fun pid comm state ppid pgrp _ -> (pid, comm, state, ppid, pgrp)) in Some obj with Scanf.Scan_failure(msg) -> (Printf.kfprintf flush stderr "failed scanning file %s: %s\n" filename msg; None) in let () = close_in ch in result with _ -> None let stat_with_sscanf pid = let input_line_from_file filename = try let ch = open_in filename in let result = try Some (input_line ch) with _ -> None in let () = close_in ch in result with _ -> None in let filename = Printf.sprintf "/proc/%d/stat" pid in match (input_line_from_file filename) with | None -> None | Some line -> try let obj = Scanf.sscanf line "%d %s %c %d %d %s@\n" (fun pid comm state ppid pgrp _ -> (pid, comm, state, ppid, pgrp)) in Some obj with Scanf.Scan_failure(msg) -> (Printf.kfprintf flush stderr "failed scanning file %s: %s\n" filename msg; None) (* Just for testing: stat 256 times the `init' process (pid 1) *) let get_some_stats ?(with_sscanf=false) () = let stat = if with_sscanf then stat_with_sscanf else stat_with_fscanf in let init_pid = 1 in let zs = Array.create 256 init_pid in Array.map (stat) zs let start_thread ?with_sscanf () = let rec loop i = let xs = get_some_stats ?with_sscanf () in let () = Printf.kfprintf flush stderr "Iteration #%02d: stat called %d times: live blocks: %d\n" i (Array.length xs) (Gc.stat ()).Gc.live_blocks in let () = Thread.delay 2. in loop (i+1) in let _ = Thread.create (loop) 1 in () (* Usage: start_thread ~with_sscanf:true ();; start_thread ();; *)