[Caml-list] howto: recursively iterate over filesystem

[Caml-list] howto: recursively iterate over filesystem

[Ivan]

[-- Attachment #1: Type: text/html, Size: 423 bytes --]

Re: [Caml-list] howto: recursively iterate over filesystem

[Rudi Grinberg]

Take a look here http://pleac.sourceforge.net/pleac_ocaml/directories.html
under the heading: "Processing All Files in a Directory Recursively"

On Mon, Jul 2, 2012 at 11:30 PM, Ivan <ivg@ieee.org> wrote:
> Hello!
>
> Are there any libraries that allow to walk through a filesystem (linux
> filesystem in particular). At best I would like to find some library that
> provides an ``iter'' function that will recursively iterate over files and
> subdir in the given folder. At the very best, a library resembling (by
> functionality) boost.filesystem will be very nice.
>
> Thanks in advance!

Re: [Caml-list] howto: recursively iterate over filesystem

[Stéphane Glondu]

Le 03/07/2012 05:30, Ivan a écrit :
> Are there any libraries that allow to walk through a filesystem (linux
> filesystem in particular). At best I would like to find some library
> that provides an ``iter'' function that will recursively iterate over
> files and subdir in the given folder. At the very best, a library
> resembling (by functionality) boost.filesystem will be very nice.

http://forge.ocamlcore.org/projects/ocaml-fileutils/


Cheers,

-- 
Stéphane

Re: [Caml-list] howto: recursively iterate over filesystem

[Ivan]

Merсi, it looks nice! But I didn't understand what's the issue with links. The FilePath.mli file contains a 
warning, that I've failed to understand:
  32     This module manipulate abstract path that are not bound to a real filesystem.
  33     In particular, it makes the assumption that there is no symbolic link that
  34     should modify the meaning of a path. If you intend to use this module
  35     against a real set of filename, the best solution is to apply to every
  36     filename to solve symbolic link through {!FileUtil.readlink}.

In my situation a folder, which I'd like to traverse, is plenty of links. 
Maybe someone can clarify it for me?

03.07.2012, 10:19, "Stéphane Glondu" <steph@glondu.net>:
> Le 03/07/2012 05:30, Ivan a écrit :
>
>>  Are there any libraries that allow to walk through a filesystem (linux
>>  filesystem in particular). At best I would like to find some library
>>  that provides an ``iter'' function that will recursively iterate over
>>  files and subdir in the given folder. At the very best, a library
>>  resembling (by functionality) boost.filesystem will be very nice.
>
> http://forge.ocamlcore.org/projects/ocaml-fileutils/
>
> Cheers,
>
> --
> Stéphane
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] howto: recursively iterate over filesystem

[Gabriel Scherer]

Have you tried the dead simple solution first, with Sys.readdir and
Sys.is_directory from the standard library (
http://caml.inria.fr/pub/docs/manual-ocaml/libref/Sys.html ), to see
if it suits your need? You can always switch to richer but more
complex alternatives if you have more specific needs later.

On Tue, Jul 3, 2012 at 10:21 AM, Ivan <ivg@ieee.org> wrote:
> Merсi, it looks nice! But I didn't understand what's the issue with links. The FilePath.mli file contains a
> warning, that I've failed to understand:
>   32     This module manipulate abstract path that are not bound to a real filesystem.
>   33     In particular, it makes the assumption that there is no symbolic link that
>   34     should modify the meaning of a path. If you intend to use this module
>   35     against a real set of filename, the best solution is to apply to every
>   36     filename to solve symbolic link through {!FileUtil.readlink}.
>
> In my situation a folder, which I'd like to traverse, is plenty of links.
> Maybe someone can clarify it for me?
>
> 03.07.2012, 10:19, "Stéphane Glondu" <steph@glondu.net>:
>> Le 03/07/2012 05:30, Ivan a écrit :
>>
>>>  Are there any libraries that allow to walk through a filesystem (linux
>>>  filesystem in particular). At best I would like to find some library
>>>  that provides an ``iter'' function that will recursively iterate over
>>>  files and subdir in the given folder. At the very best, a library
>>>  resembling (by functionality) boost.filesystem will be very nice.
>>
>> http://forge.ocamlcore.org/projects/ocaml-fileutils/
>>
>> Cheers,
>>
>> --
>> Stéphane
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa-roc.inria.fr/wws/info/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] howto: recursively iterate over filesystem

[Ivan]

It seems that I would try it. I've just started to search for already available solutions. Just to exclude a chance to invent a bicycle =)

03.07.2012, 12:24, "Gabriel Scherer" <gabriel.scherer@gmail.com>:
> Have you tried the dead simple solution first, with Sys.readdir and
> Sys.is_directory from the standard library (
> http://caml.inria.fr/pub/docs/manual-ocaml/libref/Sys.html ), to see
> if it suits your need? You can always switch to richer but more
> complex alternatives if you have more specific needs later.
>
> On Tue, Jul 3, 2012 at 10:21 AM, Ivan <ivg@ieee.org> wrote:
>
>>  Merсi, it looks nice! But I didn't understand what's the issue with links. The FilePath.mli file contains a
>>  warning, that I've failed to understand:
>>    32     This module manipulate abstract path that are not bound to a real filesystem.
>>    33     In particular, it makes the assumption that there is no symbolic link that
>>    34     should modify the meaning of a path. If you intend to use this module
>>    35     against a real set of filename, the best solution is to apply to every
>>    36     filename to solve symbolic link through {!FileUtil.readlink}.
>>
>>  In my situation a folder, which I'd like to traverse, is plenty of links.
>>  Maybe someone can clarify it for me?
>>
>>  03.07.2012, 10:19, "Stéphane Glondu" <steph@glondu.net>:
>>>  Le 03/07/2012 05:30, Ivan a écrit :
>>>>   Are there any libraries that allow to walk through a filesystem (linux
>>>>   filesystem in particular). At best I would like to find some library
>>>>   that provides an ``iter'' function that will recursively iterate over
>>>>   files and subdir in the given folder. At the very best, a library
>>>>   resembling (by functionality) boost.filesystem will be very nice.
>>>  http://forge.ocamlcore.org/projects/ocaml-fileutils/
>>>
>>>  Cheers,
>>>
>>>  --
>>>  Stéphane
>>>
>>>  --
>>>  Caml-list mailing list.  Subscription management and archives:
>>>  https://sympa-roc.inria.fr/wws/info/caml-list
>>>  Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>  Bug reports: http://caml.inria.fr/bin/caml-bugs
>>  --
>>  Caml-list mailing list.  Subscription management and archives:
>>  https://sympa-roc.inria.fr/wws/info/caml-list
>>  Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>  Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] howto: recursively iterate over filesystem

[Adrien]

Hi,

Fileutils details what is going on a bit more than the stdlib, that's
probably why it can seem more "dangerous".
You shouldn't be less safe with it. IIRC, the type for a path in
fileutils is simply a string.
I manipulate windows paths on unix and vice-versa. Of course, trying
to access one from the other is going to cause issues. The way I
understand it, the issues are similar here.

-- 
Adrien Nader

Re: [Caml-list] howto: recursively iterate over filesystem

[Fabrice Le Fessant]

let rec iter_dir f dirname =
   let files = Sys.readdir dirname in
   Array.iter (fun file ->
      let file = Filename.concat dirname file in
      (f file : unit);
      if Sys.is_directory file then iter_dir f file
  ) files

?

On Tue, Jul 3, 2012 at 10:24 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> Have you tried the dead simple solution first, with Sys.readdir and
> Sys.is_directory from the standard library (
> http://caml.inria.fr/pub/docs/manual-ocaml/libref/Sys.html ), to see
> if it suits your need? You can always switch to richer but more
> complex alternatives if you have more specific needs later.
>
> On Tue, Jul 3, 2012 at 10:21 AM, Ivan <ivg@ieee.org> wrote:
>> Merсi, it looks nice! But I didn't understand what's the issue with links. The FilePath.mli file contains a
>> warning, that I've failed to understand:
>>   32     This module manipulate abstract path that are not bound to a real filesystem.
>>   33     In particular, it makes the assumption that there is no symbolic link that
>>   34     should modify the meaning of a path. If you intend to use this module
>>   35     against a real set of filename, the best solution is to apply to every
>>   36     filename to solve symbolic link through {!FileUtil.readlink}.
>>
>> In my situation a folder, which I'd like to traverse, is plenty of links.
>> Maybe someone can clarify it for me?
>>
>> 03.07.2012, 10:19, "Stéphane Glondu" <steph@glondu.net>:
>>> Le 03/07/2012 05:30, Ivan a écrit :
>>>
>>>>  Are there any libraries that allow to walk through a filesystem (linux
>>>>  filesystem in particular). At best I would like to find some library
>>>>  that provides an ``iter'' function that will recursively iterate over
>>>>  files and subdir in the given folder. At the very best, a library
>>>>  resembling (by functionality) boost.filesystem will be very nice.
>>>
>>> http://forge.ocamlcore.org/projects/ocaml-fileutils/
>>>
>>> Cheers,
>>>
>>> --
>>> Stéphane
>>>
>>> --
>>> Caml-list mailing list.  Subscription management and archives:
>>> https://sympa-roc.inria.fr/wws/info/caml-list
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa-roc.inria.fr/wws/info/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>



-- 
Fabrice LE FESSANT

Re: [Caml-list] howto: recursively iterate over filesystem

[Ivan]

03.07.2012, 12:46, "Fabrice Le Fessant" <fabrice@lefessant.net>:
> let rec iter_dir f dirname =
>    let files = Sys.readdir dirname in
>    Array.iter (fun file ->
>       let file = Filename.concat dirname file in
>       (f file : unit);
>       if Sys.is_directory file then iter_dir f file
>   ) files
>
> ?

Yes, this is exactly what I want. Thanks!

P.S. Didn't thought that it will be so simple.

Re: [Caml-list] howto: recursively iterate over filesystem

[Eric Cooper]

On Tue, Jul 03, 2012 at 01:25:34PM +0400, Ivan wrote:
> 
> 
> 03.07.2012, 12:46, "Fabrice Le Fessant" <fabrice@lefessant.net>:
> > let rec iter_dir f dirname =
> >    let files = Sys.readdir dirname in
> >    Array.iter (fun file ->
> >       let file = Filename.concat dirname file in
> >       (f file : unit);
> >       if Sys.is_directory file then iter_dir f file
> >   ) files
> >
> > ?
> 
> Yes, this is exactly what I want. Thanks!
> 
> P.S. Didn't thought that it will be so simple.

Note that this won't terminate when there are symbolic links that
create cycles.

-- 
Eric Cooper             e c c @ c m u . e d u

Re: [Caml-list] howto: recursively iterate over filesystem

[Daniel Bünzli]

Le mardi, 3 juillet 2012 à 14:04, Eric Cooper a écrit :

> Note that this won't terminate when there are symbolic links that
> create cycles.


At least on my system (osx) it does terminate by raising Sys_error with "Too many levels of symbolic links".

Best,

Daniel  

P.S. The code I posted handles and hides this exception.

[Caml-list] Re: howto: recursively iterate over filesystem

[Sylvain Le Gall]

On 03-07-2012, Ivan <ivg@ieee.org> wrote:
> Merсi, it looks nice! But I didn't understand what's the issue with links. The FilePath.mli file contains a 
> warning, that I've failed to understand:
>   32     This module manipulate abstract path that are not bound to a real filesystem.
>   33     In particular, it makes the assumption that there is no symbolic link that
>   34     should modify the meaning of a path. If you intend to use this module
>   35     against a real set of filename, the best solution is to apply to every
>   36     filename to solve symbolic link through {!FileUtil.readlink}.
>
> In my situation a folder, which I'd like to traverse, is plenty of links. 
> Maybe someone can clarify it for me?
>

Links modify the meaning of paths and it is not clear how to react to
this. 

E.g, 

/a/b/c is a link to /a/b/d/e. How do you interpet /a/b/c/.. ?

Will it reduce to /a/b or to /a/b/d ? 

The answer to this question is "open" to interpretation...

BTW, FileUtil.find is protected against recursive links and so on...

Cheers,
Sylvain Le Gall
-- 
Linkedin:   http://fr.linkedin.com/in/sylvainlegall
Start an OCaml project here: http://forge.ocamlcore.org
OCaml blogs:                 http://planet.ocamlcore.org

Re: [Caml-list] howto: recursively iterate over filesystem

[Daniel Bünzli]

Hello, 

The book on Unix system programming in OCaml shows how to write a `find` function. 

    http://ocamlunix.forge.ocamlcore.org/files.html#htoc11

But if you don't need tight control over the process and don't want to link against the Unix module, just roll your own with the functions in Sys. 

That's what I usually use (note that it does follow symlinks). 

    val fold_files_rec : string list -> ('a -> string -> 'a) -> 'a -> 'a
    (** [fold_files_rec dirs f acc] lists the files in [dirs], recursively
         in depth first order and folds the function [f] over the file names. *)

    let fold_files_rec dirs f acc = 
      let readdir d = try Array.to_list (Sys.readdir d) with Sys_error _ -> [] in
      let is_dir d = try Sys.is_directory d with Sys_error _ -> false in
      let rec loop f acc = function
      | (d :: ds) :: up -> 
         let files = List.rev (List.rev_map (Filename.concat d) (readdir d)) in
         let dirs, files = List.partition is_dir files in
         let acc = List.fold_left f acc files in 
         loop f acc (dirs :: ds :: up)
      | [] :: [] -> acc
      | [] :: up -> loop f acc up
      | _ -> assert false
      in
      loop f acc (dirs :: []) 

Best,

Daniel