[9fans] just in case anyone has written this

[9fans] just in case anyone has written this

[ron minnich]

before I write it.

I need a command that concentrates one socket to many (outbound) and
many to one (inbound)

But it needs to do a bit more.

On the inbound side, I need it to merge lines so that, e.g., a line from
11.1.1.1 and 11.1.1.2 if same, gets printed as
1-2: Mon may 8 2011

and if we have
11.1.1.1 and 11.1.1.2 and 11.1.1.4 and 11.1.1.5 you get
1-2,4-5: blah blah


I have a sort of command like this, which I can adapt to Plan 9. If,
however, someone already has this for Plan 9, it would save me time.
For now I only need 64 hosts but at some point need 1024 (which my
current command does)
and eventually 262144.

thanks

ron

Re: [9fans] just in case anyone has written this

[andrey mirtchovski]

the p9p version of xcpu used to do something similar, but the old svn
repository is gone now so i can't verify. i see an old version of rx.c
in 9grid:/usr/andrey/src/xcpu/rx.c which does something similar to
what you want with threads, channels and procs but doesn't combine the
input.

it doesn't look finished, so it may be useless. it creates a proc for
each remote host :(

it does something like this (from a perfunctory look, it's been a while):

---
# shared between all procs
create input chan
create output chan

for each node create a proc that:
    connects to node and set up a session
    creates proc for stdout
    creates proc for stderr
    waits on input chan for commands to send to the node

stdout and stderr procs do:
    read from remote host
    write to output chan
---

the aggregator part you want can be inserted between read the output
chan and whatever you need to output to, but that logic doesn't exist
in rx.c, i think :)

Re: [9fans] just in case anyone has written this

[roger peppe]

2009/4/28 ron minnich <rminnich@gmail.com>:
> On the inbound side, I need it to merge lines so that, e.g., a line from
> 11.1.1.1 and 11.1.1.2 if same, gets printed as
> 1-2: Mon may 8 2011

if you do this, then presumably you can't print a line
from any source until you've got a line from all of them.
is that what you want? or does the merging apply only
to lines received within some time interval of each other?

Re: [9fans] just in case anyone has written this

[ron minnich]

On Tue, Apr 28, 2009 at 2:05 AM, roger peppe <rogpeppe@gmail.com> wrote:
> 2009/4/28 ron minnich <rminnich@gmail.com>:
>> On the inbound side, I need it to merge lines so that, e.g., a line from
>> 11.1.1.1 and 11.1.1.2 if same, gets printed as
>> 1-2: Mon may 8 2011
>
> if you do this, then presumably you can't print a line
> from any source until you've got a line from all of them.
> is that what you want? or does the merging apply only
> to lines received within some time interval of each other?

you have to have a timeout (which I do) since nodes can always fail or
not talk to you

ron

Re: [9fans] just in case anyone has written this

[roger peppe]

2009/4/28 ron minnich <rminnich@gmail.com>:
> On Tue, Apr 28, 2009 at 2:05 AM, roger peppe <rogpeppe@gmail.com> wrote:
>> 2009/4/28 ron minnich <rminnich@gmail.com>:
>>> On the inbound side, I need it to merge lines so that, e.g., a line from
>>> 11.1.1.1 and 11.1.1.2 if same, gets printed as
>>> 1-2: Mon may 8 2011
>>
>> if you do this, then presumably you can't print a line
>> from any source until you've got a line from all of them.
>> is that what you want? or does the merging apply only
>> to lines received within some time interval of each other?
>
> you have to have a timeout (which I do) since nodes can always fail or
> not talk to you

if one node is just slow enough in responding that it
falls outside the timeout, you could get an annoying situation
where that node is out-of-step forever after. i guess it depends
how often incoming lines arrive. if it's an interactive
session, i guess it might not matter too much - i can see
how this might be useful controlling many shell
sessions at once. you could make things more reliable by
tagging requests and replies, but even then there's the
question of what to do when sessions diverge (assuming
that's what you're doing). interesting.

Re: [9fans] just in case anyone has written this

[andrey mirtchovski]

> if one node is just slow enough in responding that it
> falls outside the timeout, you could get an annoying situation
> where that node is out-of-step forever after.

worse yet, nodes may be sending more than one line at a time,
circumventing the aggregator. if they do it fast enough it becomes a
real mess and there's no amount of lookback one can do to ensure this
isn't happening :)

i'm routinely seeing syslog brought to its knees around here by a
particular cluster management software which decides to log two lines
instead of just one for a particular often-failing operation, so
instead of 'message repeated X times' (for some very large X) we get
'disk full'...

Re: [9fans] just in case anyone has written this

[erik quanstrom]

> > if one node is just slow enough in responding that it
> > falls outside the timeout, you could get an annoying situation
> > where that node is out-of-step forever after.
>

i fought some socket mgmt software for a few years that did
timeouts and rollup like this.  it seemed to me that between timeouts and
retransmission one could not dig oneself out of the hole without
a proper protocol.  and doing it on top of tcp was impossible.

> worse yet, nodes may be sending more than one line at a time,
> circumventing the aggregator. if they do it fast enough it becomes a
> real mess and there's no amount of lookback one can do to ensure this
> isn't happening :)

but even if it logs the message once, your disk full message will appear
in n*2 seconds.

do we need to take up a collection to get you some disks?

- erik

Re: [9fans] just in case anyone has written this

[andrey mirtchovski]

> do we need to take up a collection to get you some disks?

not yet, although the syslogs in question are on a somewhat
constrained budget since they're running inside a vm whose image is,
for various reasons, kept small...

Re: [9fans] just in case anyone has written this

[erik quanstrom]

> not yet, although the syslogs in question are on a somewhat
> constrained budget since they're running inside a vm whose image is,
> for various reasons, kept small...

sometimes i think us cs types operate with the following algorithm

	do{
		while(!resourceconstrainted())
			abstract();
	}while(budgetreqest() != -1);

- erik

Re: [9fans] just in case anyone has written this

[ron minnich]

On Tue, Apr 28, 2009 at 8:17 AM, roger peppe <rogpeppe@gmail.com> wrote:

> if one node is just slow enough in responding that it
> falls outside the timeout, you could get an annoying situation
> where that node is out-of-step forever after. i guess it depends
> how often incoming lines arrive.

Sure.

And things will always go wrong. Anything designed for things going
well will fail.

ron