Re: [9fans] data analysis on plan9

["J. R. Mauro" <jrm8005@gmail.com>]

On Jul 9, 2009, at 14:40, hugo rivera <uair00@gmail.com> wrote:

> Hi,
> since I discovered plan 9, about two years ago, I've been constantly
> amazed by its simple yet quite powerful design.
> From one year now, I am looking forward to move to plan 9 as my main
> OS, but I am not able to do so because it lacks the data analysis
> tools available in some other systems, like linux.
> Because my work involves dealing with data coming from experiments in
> astro-particle physics, I am more or less tied to data analysis
>
> software like the R programming language, Python's Numpy, Cern's ROOT

There a plan 9 R port, isn't there? If not, there might be an R done
in python or something.

>
> and even gnuplot. While using them, I realized that most of the time I
> deal with text files that go here and there as input or output of
> small specific programs that perform a given task (I don't know if
> this is the result of my Unix/Plan 9 background or just a
> coincidence). Say I have a command 'clean' that removes undesired
> points from a body of data, and another command 'four' that performs
> the FFT; so they are used together as
> clean data.txt | four > results.txt
> so it occurred to me that one can create single commands to interact
> among them to perform some analysis on data, just like in the original
> Unix style. Awk can be used as glue among them, with some other small
> glue utilities. Plotting data is another thing that I would like to
> integrate into this, since plots are quite frequent while analysing
> data, but I am not sure how.

Plan 9 has plot program that fit well in a pipeline. Even gnuplot can
go in a pipeline.

>
> Also, something similar to GSL (http://www.gnu.org/software/gsl/)
> would be invaluable or maybe even indispensable.
> Maybe some day I'll start to write some commands for plan 9 to begin
> working on it, but I want to convince myself that this is worth the
> time spent.

I think it's worth it. Parts of this idea are already there (sum,
sort, join, plot)

>
> What do you think of this? my main concern is that perhaps the "do one
> thing well" design falls short for data analysis. I've never seen
> people work like this on data analysis before (but I do not think I am
> the first to do it) because in general, they tend to use large data
> analysis frameworks. I'd really appreciate some feedback on this from
> people working on data analysis and also from the plan 9 community
> (otherwise I wouldn't be writing here :-)

I know someone who does astrophysics analysis and visualization
(including movies) on a special "OS" he wrote that works entirely like
Unix pipes and filters. I think developing an anlysis framework as a
pipes-and-filters toolbox is great.

>
> Saludos
>
> --
> Hugo
>