State of the 9front mailing list archives

State of the 9front mailing list archives

[stanley lieber]

I skimmed IRC backlog this morning and noticed people complaining again about the cumbersome 9P interface and lack of web access to the mailing list archives.

The following are some notes about the future of the archive:

- A ctl file offering search functionality has been exposed through the 9P share (9fs 9front; ls /n/lists) for some time. Presently, the automatic indexing that powers the search feature is broken. Once that is fixed and tested a command for searching the archive will be published, most likely by being added to 9front.

- A werc app was written to present a web interface to the existing archive (sans search). It works pretty well but tanks performance of the system under normal (webcrawler) load. It is currently disabled pending review.

- There was some discussion on IRC about how the messages are stored, and someone offered to have the archive added to gmane. If would be very happy to facilitate this if the interested parties would send me an e-mail so we can begin the process. For future reference, messages are stored on disk in nupas' mdir format, one file per mesage. It is trivial to convert these files between mdir and mbox, or basically any other format.

Ironically, while I was not around to correct misconceptions during this latest iteration of the discussion on IRC, I'm pretty sure I've explained all of the above to these people at least once, on IRC. Obviously, some other venue for exchanging information and storing it in a persistent, searchable format is required.

Gmane seems like a great interim solution.

sl

Re: [9front] State of the 9front mailing list archives

[cinap_lenrek]

i'v rediscovered my email with the 9frontctl file ok.

solved my problem.

i was just wondering if there was a way to directly
link to an ealier email...

i just downloaded the message and put it on my server...

--
cinap

Re: [9front] State of the 9front mailing list archives

[stanley lieber]

In the case of the guy trying to update his auth server, the information was already in the fqa. Granted, the requirement for performing this procedure after the old protocol was disabled by default was never made clear to anyone. Shit just suddenly broke. If someone happened to be on IRC when people were discussing it they might have picked up a clue. I probably could have handled this better by at least putting a notice in a release announcement.

In my previous mail I was responding more to the same discussion that gets held over and over on IRC by the same people about how accessing the mailing list archive is awkward and slow. Everyone always concludes they haven't studied the problem closely and then the topic dies for a while until it's time to complain again. The solutions always involve handing off responsibility to some third party service (read: Linux).

In this case I think gmane is a great idea. Hopefully the person who suggested it will contact me and tell me what I need to do to make it happen.

sl

Re: [9front] State of the 9front mailing list archives

[cinap_lenrek]

it was indeed in the changelog of "the muscott icon of it! Why devil?" release:

"tcp567: run authserver with p9sk1 tickets disabled preventing offline password brute-force"

but i should have written a longger email announcing the change.

i know this is painfull. we'v almost spend a year on the whole topic,
implementing it so it could have a transition period.

regressions can happen at every release, no matter what. we do have
a procedure to update your stuff and be protected. we do not have
a procedure when your system got hacked.

for the lack of communicating this more loudly i'm sorry.

--
cinap

Re: [9front] State of the 9front mailing list archives

[BurnZeZ]

On Fri Jun  3 14:46:01 EDT 2016, sl@stanleylieber.com wrote:
> In my previous mail I was responding more to the same discussion that gets held over and over on IRC by the same people about how accessing the mailing list archive is awkward and slow.

That's expected with latency and any protocol that requires a lot of back and forth.


meiling% 9fs 9front >[2]/dev/null
meiling% time ls -l /n/9front/lists/9fans >/tmp/foo
0.90u 0.08s 120.38r 	 ls -l /n/9front/lists/9fans
meiling% wc -l /tmp/foo
  93845 /tmp/foo
meiling% echo 93845 / 120.38 | hoc
779.573018774 # files per second

meiling% ip/ping -n 5 9front.org
sending 5 64 byte messages 1000 ms apart to icmp!9front.org!1
0: rtt 49053 µs, avg rtt 49053 µs, ttl = 245
1: rtt 47224 µs, avg rtt 48138 µs, ttl = 245
2: rtt 47660 µs, avg rtt 47979 µs, ttl = 245
3: rtt 47340 µs, avg rtt 47819 µs, ttl = 245
4: rtt 47084 µs, avg rtt 47672 µs, ttl = 245
meiling% echo 120.38 / 0.04672 | hoc
2576.62671233 # number of possible round trips
meiling% echo 93845 / 2576 | hoc
36.4305124224 # estimated files per round trip

# /tmp/cache is an arbitrary directory I store some crap
meiling% time ls -l /tmp/cache >/tmp/bar
0.07u 0.01s 0.09r 	 ls -l /tmp/cache
meiling% wc -l /tmp/bar
   6161 /tmp/bar
meiling% echo 6161 / 0.09  | hoc
68455.5555556 # files per second, locally


This assumes we send and receive messages, one at a time.
It's better than I thought, but I suppose it doesn't scale well to
directories with hundreds of thousands of files, or a higher
latency link.

What about increasing the speed of light?

Re: [9front] State of the 9front mailing list archives

[Benjamin Purcell]

You would have to increase the speed of light only for 9p messages.

On Sat, Jun 4, 2016 at 1:31 AM,  <BurnZeZ@feline.systems> wrote:
>
> On Fri Jun  3 14:46:01 EDT 2016, sl@stanleylieber.com wrote:
>> In my previous mail I was responding more to the same discussion that gets held over and over on IRC by the same people about how accessing the mailing list archive is awkward and slow.
>
> That's expected with latency and any protocol that requires a lot of back and forth.
>
>
> meiling% 9fs 9front >[2]/dev/null
> meiling% time ls -l /n/9front/lists/9fans >/tmp/foo
> 0.90u 0.08s 120.38r      ls -l /n/9front/lists/9fans
> meiling% wc -l /tmp/foo
>   93845 /tmp/foo
> meiling% echo 93845 / 120.38 | hoc
> 779.573018774 # files per second
>
> meiling% ip/ping -n 5 9front.org
> sending 5 64 byte messages 1000 ms apart to icmp!9front.org!1
> 0: rtt 49053 µs, avg rtt 49053 µs, ttl = 245
> 1: rtt 47224 µs, avg rtt 48138 µs, ttl = 245
> 2: rtt 47660 µs, avg rtt 47979 µs, ttl = 245
> 3: rtt 47340 µs, avg rtt 47819 µs, ttl = 245
> 4: rtt 47084 µs, avg rtt 47672 µs, ttl = 245
> meiling% echo 120.38 / 0.04672 | hoc
> 2576.62671233 # number of possible round trips
> meiling% echo 93845 / 2576 | hoc
> 36.4305124224 # estimated files per round trip
>
> # /tmp/cache is an arbitrary directory I store some crap
> meiling% time ls -l /tmp/cache >/tmp/bar
> 0.07u 0.01s 0.09r        ls -l /tmp/cache
> meiling% wc -l /tmp/bar
>    6161 /tmp/bar
> meiling% echo 6161 / 0.09  | hoc
> 68455.5555556 # files per second, locally
>
>
> This assumes we send and receive messages, one at a time.
> It's better than I thought, but I suppose it doesn't scale well to
> directories with hundreds of thousands of files, or a higher
> latency link.
>
> What about increasing the speed of light?

Re: [9front] State of the 9front mailing list archives

[arisawa]

Hello,

this is my case
term% time ls -l /n/lists/9fans >/tmp/foo
0.35u 0.08s 405.16r 	 ls -l /n/lists/9fans
term% 

I have been considering this type of problems: large data in server.
grid computing might be the only solution.

Kenji Arisawa

> 2016/06/04 15:31、BurnZeZ@feline.systems のメール：
> 
> 
> On Fri Jun  3 14:46:01 EDT 2016, sl@stanleylieber.com wrote:
>> In my previous mail I was responding more to the same discussion that gets held over and over on IRC by the same people about how accessing the mailing list archive is awkward and slow.
> 
> That's expected with latency and any protocol that requires a lot of back and forth.
> 
> 
> meiling% 9fs 9front >[2]/dev/null
> meiling% time ls -l /n/9front/lists/9fans >/tmp/foo
> 0.90u 0.08s 120.38r 	 ls -l /n/9front/lists/9fans
> meiling% wc -l /tmp/foo
>  93845 /tmp/foo
> meiling% echo 93845 / 120.38 | hoc
> 779.573018774 # files per second
> 
> meiling% ip/ping -n 5 9front.org
> sending 5 64 byte messages 1000 ms apart to icmp!9front.org!1
> 0: rtt 49053 µs, avg rtt 49053 µs, ttl = 245
> 1: rtt 47224 µs, avg rtt 48138 µs, ttl = 245
> 2: rtt 47660 µs, avg rtt 47979 µs, ttl = 245
> 3: rtt 47340 µs, avg rtt 47819 µs, ttl = 245
> 4: rtt 47084 µs, avg rtt 47672 µs, ttl = 245
> meiling% echo 120.38 / 0.04672 | hoc
> 2576.62671233 # number of possible round trips
> meiling% echo 93845 / 2576 | hoc
> 36.4305124224 # estimated files per round trip
> 
> # /tmp/cache is an arbitrary directory I store some crap
> meiling% time ls -l /tmp/cache >/tmp/bar
> 0.07u 0.01s 0.09r 	 ls -l /tmp/cache
> meiling% wc -l /tmp/bar
>   6161 /tmp/bar
> meiling% echo 6161 / 0.09  | hoc
> 68455.5555556 # files per second, locally
> 
> 
> This assumes we send and receive messages, one at a time.
> It's better than I thought, but I suppose it doesn't scale well to
> directories with hundreds of thousands of files, or a higher
> latency link.
> 
> What about increasing the speed of light?

Re: [9front] State of the 9front mailing list archives

[Kurt H Maier]

On Fri, Jun 03, 2016 at 02:23:01PM -0400, stanley lieber wrote:
> - A werc app was written to present a web interface to the existing archive (sans search). It works pretty well but tanks performance of the system under normal (webcrawler) load. It is currently disabled pending review.

I wrote this app, and it works for medium- to low-quantity lists, but
absolutely wrecks the server for something like 9fans.  The app is
mostly awk and can be found at

http://code.9front.org/hg/mdir/file/7bc6acff8550/app.rc

I recommend that this app be converted to a script that can be run from
a cron job to generate static html which werc would easily be able to
serve.  Mailing list archives are theoretically append-only and I don't
think it would kill anyone if messages did not appear on the web
interface until some delay.

khm