Re: [9fans] a question of file and the history of magic

Re: [9fans] a question of file and the history of magic

[erik quanstrom]

> In a sense, the question is more about the historical change and/or
> adoption of a new file command for Plan 9 that doesn't use a magic
> file for references.  Why opt out of a magic file other than the
> obvious performance hit of scanning it each run?  Is it worth
> repeating the old forms that used magic, or has anyone in the Plan 9
> community already improved upon the idea and introduced a new, more
> adaptable tool?

what is the upside to an external magic file?  as you've shown, you
can add a file type in 1 line of code.  while the external magic file
isn't c, i would argue that it's still code.

the disadvantage is that you need to write a parser for yet another
file format.  it turns out that linux file's maintainers felt that a text file
wasn't good enough so they implemented a magic compiler.  i really
don't understand the logic behind the compiler, since it would seem
to trade reduced cpu cycles for increased i/o.  that would seem to be
a terrible trade off these days.

; wc magic magic.mgc
  13469   69850  484372 magic
   1301   17997 1062400 magic.mgc		# compiled version

the source is pretty big, too:

; wc -l ffile-4.20/src/*.[ch]|grep total
  9273 total

according to wikipedia (http://en.wikipedia.org/wiki/File_(Unix)),
system v introduced the external magic file.  i don't think that system v
was in anyway an ancestor of plan 9.  but i don't know anything of
the history of plan 9 file.

- erik

Re: [9fans] a question of file and the history of magic

[Brantley Coile]

I remember the day I first saw a file magic file.  I welcomed it because
for the first time I didn't have access to the source code.  Those were
the days when you had to have $45k to get the source.  A hard thing to
ask for.  Today a separate magic file is just a leftover vestige of the
past.  There are a lot of things like that.  Do we still need to
compress man pages on 1TB disk driver? :)

erik quanstrom wrote:
>>In a sense, the question is more about the historical change and/or
>>adoption of a new file command for Plan 9 that doesn't use a magic
>>file for references.  Why opt out of a magic file other than the
>>obvious performance hit of scanning it each run?  Is it worth
>>repeating the old forms that used magic, or has anyone in the Plan 9
>>community already improved upon the idea and introduced a new, more
>>adaptable tool?
>
>
> what is the upside to an external magic file?  as you've shown, you
> can add a file type in 1 line of code.  while the external magic file
> isn't c, i would argue that it's still code.
>
> the disadvantage is that you need to write a parser for yet another
> file format.  it turns out that linux file's maintainers felt that a text file
> wasn't good enough so they implemented a magic compiler.  i really
> don't understand the logic behind the compiler, since it would seem
> to trade reduced cpu cycles for increased i/o.  that would seem to be
> a terrible trade off these days.
>
> ; wc magic magic.mgc
>   13469   69850  484372 magic
>    1301   17997 1062400 magic.mgc		# compiled version
>
> the source is pretty big, too:
>
> ; wc -l ffile-4.20/src/*.[ch]|grep total
>   9273 total
>
> according to wikipedia (http://en.wikipedia.org/wiki/File_(Unix)),
> system v introduced the external magic file.  i don't think that system v
> was in anyway an ancestor of plan 9.  but i don't know anything of
> the history of plan 9 file.
>
> - erik
>

Re: [9fans] a question of file and the history of magic^H^H^H^H^HUNIX

[Lyndon Nerenberg]

On 2008-Jul-6, at 14:59 , Brantley Coile wrote:

> I remember the day I first saw a file magic file.  I welcomed it
> because for the first time I didn't have access to the source code.
> Those were the days when you had to have $45k to get the source.

Closer to $100K for most people. I had great fun writing nroff (yup,
*n*roff) output device tables as binary blobs to interface with the
non-source UNIXen of the day. And the Convergent Technologies X.25
binary code was a wonder to configure/tune as an end user :-P

Remember kids: UNIX source code (BSD, really) wasn't free until 1994
(give or take a bit). You haven't lived until you've resolved device
driver configuration and ordering problems when trying to link a
binary version of SunOS 2 or 3 (no, not Solaris :-).  Or even better,
an NBI VME 68K box pretending it's a UNIBUS VAX.

Is alt.folklore.computers still alive? 99% of the list traffic
[cs]hould be redirected there.

-- (creaky/grumpy olde) lyndon

Re: [9fans] a question of file and the history of magic

[Bakul Shah]

On Sun, 06 Jul 2008 17:20:12 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> what is the upside to an external magic file?  as you've shown, you
> can add a file type in 1 line of code.  while the external magic file
> isn't c, i would argue that it's still code.

Yes it is code but the advantage is that the parser language
is factored out and anyone can add knowledge about new file
formats and it is easy to debug and experiment.

The main disadvantage of gnu file is performance.  As an
example, on about 6000 files totalling 200MB, gnu file takes
2s user, 1s system and 30s real time.  Compared to that p9p
file takes 1.65s user, 0.25s system and 1.9s real time.
Note: any cache effects have been accounted for by looking at
only the best 3 runs of of each test.  As per csh, there were
no page faults and no disk io.

The magic file should really be compiled and linked w/
file(1) -- if that was done right, the rest of file(1) code
would be pretty trivial.  On the other hand file is usually
not a performance bottleneck.  On the gripping hand there are
a lot of similarities between cracking file formats and
packet formats so may be there is value in factoring all that
out and sticking it in a library routine.

Re: [9fans] a question of file and the history of magic

[Charles Forsyth]

> The main disadvantage of gnu file is performance.

the magic file contains surprisingly many spells,
even excluding muttered incantations.