[9fans] 9bench: lmbench for Plan 9

[9fans] 9bench: lmbench for Plan 9

[john]

I've just finished porting what we thought were the 8 most useful
benchmarks from the lmbench v.2 suite to run natively in Plan 9.  The
benchmarks are:
bw_file_rd: Measure file read bandwidth
bw_mem: Measure memory bandwidth for a several different operations
bw_pipe: Measure pipe bandwidth
bw_tcp: Measure tcp bandwidth on the local machine
lat_ctx: Measure context switch latency
lat_syscall: Measure system call latency
lat_tcp: Measure tcp latency on local system
lat_udp: Measure udp latency on local system

As I note in the README, lmbench's license carries the additional
restriction that you cannot publish results from modified benchmarks,
so keep your results to yourself.

The source is in /n/sources/contrib/john/9bench.tgz


John

Re: [9fans] 9bench: lmbench for Plan 9

[erik quanstrom]

> As I note in the README, lmbench's license carries the additional
> restriction that you cannot publish results from modified benchmarks,
> so keep your results to yourself.

doesn't generating unpublishable results defeat the purpose of a benchmark?

- erik

Re: [9fans] 9bench: lmbench for Plan 9

[ron minnich]

On Fri, Mar 21, 2008 at 10:51 AM, erik quanstrom <quanstro@coraid.com> wrote:
> > As I note in the README, lmbench's license carries the additional
>  > restriction that you cannot publish results from modified benchmarks,
>  > so keep your results to yourself.
>
>  doesn't generating unpublishable results defeat the purpose of a benchmark?

what is the purpose?

ron

Re: [9fans] 9bench: lmbench for Plan 9

[erik quanstrom]

> On Fri, Mar 21, 2008 at 10:51 AM, erik quanstrom <quanstro@coraid.com> wrote:
> > > As I note in the README, lmbench's license carries the additional
> >  > restriction that you cannot publish results from modified benchmarks,
> >  > so keep your results to yourself.
> >
> >  doesn't generating unpublishable results defeat the purpose of a benchmark?
>
> what is the purpose?

i would assume that the main reason to go with an official benchmark is to
compare results.  if you don't need to compare results and you want to tell
if something you've changed has made things faster or slower, i would think
rolling one's own would be simple and effective.  "time dd ..." is often a great
relative measure.

- erik

Re: [9fans] 9bench: lmbench for Plan 9

[hiro]

Perhaps the copyright owner just wanted to make fun of intellectual property;)

Re: [9fans] 9bench: lmbench for Plan 9

[Francisco J Ballesteros]

Is a port of lmbench considered a modified benchmark?
I understand you cannot publish results saying it´s the output
of lmbench if you changed the bechmark, but only because you
used a ported one...?

Funny in any case.

On Sat, Mar 22, 2008 at 12:18 PM, hiro <23hiro@googlemail.com> wrote:
> Perhaps the copyright owner just wanted to make fun of intellectual property;)
>
>

Re: [9fans] 9bench: lmbench for Plan 9

[erik quanstrom]

> Is a port of lmbench considered a modified benchmark?
> I understand you cannot publish results saying it´s the output
> of lmbench if you changed the bechmark, but only because you
> used a ported one...?
>
> Funny in any case.

you're also not allowed to publish partial results.  they have
really overthought things.  they are worried that someone
will try to look better than they are by publishing partial results.

- erik

Re: [9fans] 9bench: lmbench for Plan 9

[john]

>> Is a port of lmbench considered a modified benchmark?
>> I understand you cannot publish results saying it´s the output
>> of lmbench if you changed the bechmark, but only because you
>> used a ported one...?
>>
>> Funny in any case.
>
> you're also not allowed to publish partial results.  they have
> really overthought things.  they are worried that someone
> will try to look better than they are by publishing partial results.
>
> - erik

Apparently he also didn't want companies (read: Sun) rewriting portions
of the benchmark to make their software/hardware look better.
Since Plan 9 doesn't have sockets or the bzero call or a number of other
things that you'll find on Unix, I had to do some rewriting. I made
sure to accomplish the same tasks as on Unix, but in some cases more
than others I had to do a lot of changing. The timing code should
still be pretty much the same, although I added a little to include
counting of clock cycles (approximate). The pipe bandwidth test, if
I remember correctly, took basically no changes to port (I think I
had to change sprintf to sprint or something like that), but others,
like the networking ones, required that I essentially cut out the
majority of the tests and replace them with a Plan 9 equivalent. If
anyone is running these and finds a test that has the wrong behavior,
please let me know so I can fix it.

As for the utility of the benchmarks, come on. I know there are people
on this list who write kernel code (hmm, I seem to recall somebody
on here who does a lot of work on networking drivers...) and they
may find it useful to get these kinds of measurements to see if
changes have improved performance. Besides, there's nothing to stop you
from running the benchmarks on Linux and Plan 9 and comparing (for
yourself) the difference in performance--you just can't try for
e-cred by posting your results on comp.os.linux (hint: the
comparisons don't come out very favorably for Plan 9 anyway).

Oh, and if anybody ever wants to see the Macros From Hell, just take
a look at bench.h for BENCH0, BENCH, BENCH1, and BENCH-INNER.
(disclaimer: I didn't write those)


John

Re: [9fans] 9bench: lmbench for Plan 9

[erik quanstrom]

> As for the utility of the benchmarks, come on. I know there are people
> on this list who write kernel code (hmm, I seem to recall somebody
> on here who does a lot of work on networking drivers...) and they
> may find it useful to get these kinds of measurements to see if
> changes have improved performance. Besides, there's nothing to stop you
> from running the benchmarks on Linux and Plan 9 and comparing (for
> yourself) the difference in performance--you just can't try for
> e-cred by posting your results on comp.os.linux (hint: the
> comparisons don't come out very favorably for Plan 9 anyway).

i tend to use very simple timing runs in my testing.  i use dd for AoE
and disk drivers and a small program named zerohose for (other)
network protcols.  time(1) seems good enough for timing given an
appropriate amount of i/o.

the reason for the simple timing tests is because good hardware is
hard to get wrong but marginal hardware (bad design, undersized
buffers, unsufficient pcie lanes) is typically very sensitve to cache
pecularities.  even with simple tests some hardware can have >20%
variance between runs.  it's very hard to reason about this sort of
thing.  i'd hate to have to also have to understand what the benchmark
is doing for^wto me, too.

oh, plan 9 can push packets faster on the nics i've tested than linux,
but for some reason, plan 9 tcp performance hasn't been very good,
especially on 10gbe hardware.  il and aoe fair much better.

- erik