notes on fossil, ANTS, and 9front/Bell labs controversies

notes on fossil, ANTS, and 9front/Bell labs controversies

[mycroftiv]

This is also posted at http://doc.9gridchan.org/blog/200110.fossil.9front

This post is a survey of technical and community issues connected to
Plan 9 root file servers.  The author is not a member of any Plan 9
tribe save Grid.  Opinions are entirely my own and a lot of unsourced
claims will be made based on my years of participation and observation
of the Plan 9 community.

ANTS installer: fossil root for 9front code

The ANTS iso installer scripts include fossil+venti root install
option.  This makes it easier to use the 9front codebase with
fossil+venti root.  The 9front developers have made a conscious and
deliberate choice to remove fossil support from their distribution, so
undoubtedly they don't see this as an improvement.  I have been using
a primarily 9front codebase on top of fossil+venti root fileservers
for my home grid of systems and for most of the 9gridchan.org servers
for the past several years with good results.  My datasets are small
and my results may not translate well to users with much larger
quantities of data.  The latest ANTS release contains support tooling
for venti replication as well as installer support.

- 64-bit 9ants5.64.iso.gz 
https://files.9gridchan.org/9ants5.64.iso.gz

This distribution of Plan 9 has admittedly flaky maintainership, but
competent users should be able to track 9front changes even if
mycroftiv has wandered off to some distant hypercube.

A highly subjective historical view

When 9front forked from the Bell Labs distribution of Plan 9, one of
the major changes was removing the option to use the fossil file
server as a root file system and making cwfs64x the primary fs in the
installer.  In the late 2000s era of Plan 9, many users reported
unreliable behavior of the fossil+venti combination.  The resources
being given to Plan 9 by the corporate heirs to Bell Labs were clearly
insufficient to properly develop and debug the operating system.  As I
understand it, there were two engineers working part-time on the
public Plan 9 infrastructure.  Without sufficient resource investment,
even the best efforts of skilled developers may not scale to the needs
of a full operating system project.

At the same time, the late and sorely missed Uriel was attempting to
grow an independent community of people dedicated to the software
principles espoused by the unix tradition and embodied in Plan 9.
Unfortunately, in my view, Uriel's health problems influenced his
communication, and his passionate desire to see Plan 9 succeed and
thrive led him to debates full of angry sarcasm and increasing
bitterness.  The way it all played out was to create divisions and bad
feelings between the corporate and university Plan 9 developers and
researchers and much of the grass-roots Plan 9 community.

Fossil-venti unreliability was a focal point of dissatisfaction.  When
9front forked, removing fossil support from the installer was part of
a clear message that the standard Bell Labs setup was not reliable.
Discussions of the motivations and necessity for the 9front fork have
often referenced poor user experiences with fossil+venti in the time
prior to the fork.  My own experience of trying to use fossil+venti in
the 2000s era supports this perspective.  In fact I found things so
unreliable that a major factor in the software that became ANTS was
"how to deal with the failure of the root filesystem" because that was
a regular and expected occurence in my use of Plan 9.

- The bug and fix

In 2012, after 9front had already forked, Richard Miller found and
fixed a bug in fossil that I believe was the primary source of many
fossil-venti reliability issues.  He wrote a very nice post describing
the bug and its solution.

https://marc.info/?l=9fans&m=133243392017939&w=2

I believe the precise nature of this bug explains a lot of things.  As
I understand it, the condition for triggering the bug is making
changes to filesystem data during the archival snapshot process.  For
experienced Plan 9 users with always-on fileservers set to do a daily
snapshot in the middle of the night, things were reliable.  For a new
user who installed Plan 9 and started immediately messing around and
installing software while the initial archival snapshot is happening,
there was a good chance that corruption would sneak into the fs from
the very beginning.  This matches my memory of frequent troubles with
installs that seemed to break right away.

In 2015, I migrated ANTS to 9front, and because I have always used and
valued the powerful operations that venti-backed fossil rootscores
support, I decided to figure out how to use 9front/ants with a fossil
root.  I was expecting trouble because of the prevailing sentiment in
the 9front community that was fossil was a no-good very-bad
filesystem, and my own previous struggles.  In practice however, what
I discovered was that fossil seemed to be working just fine.  I worked
out a system for getting 9front+ants+fossil onto vultr vps nodes and
started hosting 9gridchan.org websites on this platform as well as
using it for my home systems.

Between 2015 and now, I believe I have accumulated 25+ machine-years
of trouble free 9front/ants/fossil service on my home systems and
public servers.  Most of these systems only store a few gigabytes of
data total so the workload is light however.  Other users with large
data sets have expressed some disappointment with performance.

- The Tribes

Most people understand that the Plan 9 user community is both small,
and strongly fractured into tribes who seem to harbor personal grudges
toward each other.  It's easy to understand why the sense of humor and
strongly critical perspective of many "9front tribe" people rub some
the wrong way - especially the people who are the actual target of
satire.  People just aren't really into being criticized and satirized
as a general rule.

It's also easy to understand why what can only be referred to as the
"Old Boys Club" of Plan 9 has critics.  Plan 9 has always had
overtones of elitism - from my reading, Rob Pike's writings and
commentary set a lot of the tone which Uriel amplified.  The "haughty
style" of commentary on computing, in which the Sad and Fallen state
of computing into complexity and muddled design is Lamented by the
Sage goes back a long way - Dijkstra in particular perpetrated many
famous examples.  Humans love having social status, and nothing
signals high social status more than condescending to the
un-enlightened.

New users arriving in the Plan 9 community in the 2000s discovered a
mismatch between the rhetoric of how much superior everything was in
the Plan 9 world, and the reality of unreliable root fileservers,
challenging and complicated administration with documentation focused
on Murray Hill's system, and lots of rumors of intriguing code and
research projects that didn't seem to feed into the publicly available
Plan 9 experience.  The operating system seemed rather stagnant and
buggy, but the veterans of the Plan 9 community seemed to talk in a
parallel universe in which Plan 9 was perfectly meeting all of their
computing needs and any issues were the result of user error and
inexperience.

- The Present Day, SHA1, community

So, 9front tribe doesn't want to use fossil, and labs/legacy tribe
doesn't want to use 9front code.  I believe that both of these
perspectives at this point are based more on human social conflicts
than solid technical motivations.  I am trying to offer a solution to
users who are interested in exploring the most practical ways to use
Plan 9.  The major fly in the ointment is the outdated sha1 hash
function used by venti.  As a result of this yet-to-be-fixed
shortcoming, the world of venti-backed public plan 9 services is not
yet ready for large scale production use.  For personal and hobby use
howeveer, I believe the sha1 issues are manageable.

I hope that this post can serve a constructive purpose in building
mutual understanding between humans and increasing code-sharing and
collaboration.  Given the way things often go, I'm probably screwing
up and just pissing everyone on both sides off more by presenting my
own warped and subjective view of things.  I recall being told to
"stop alternating history" for sharing my perspective on the saga
of Plan 9 community and software - so please offer your own
corrections and perspective, especially if you think I have made any
factual mistakes or you have first hand knowledge and experience to
share.

I've always had pretty good interactions with other humans who use
Plan 9 and I'm really into the whole "tolerant and diverse community"
thing.  I'd like a lot more people from different places and with
different backgrounds and cultures and life experiences to get into
Plan 9 and find it to be a fun and welcoming experience.  I've been
trying since 2008 to help people learn to use Plan 9 to communicate
and connect and cooperate and have better and more direct control over
their software and system, outside of systems of hierarchical
authority and control and exploitation and coercion.

I've made a lot of mistakes in my own communication and community
building efforts.  2020 seems like a Year of Change and I'm hoping for
positive change in the Plan 9 community.  I'm particularly excited for
the reincarnation of a real-life Plan 9 conference/hackathon event.
Humans do better in real life social situations than internet
diatribes and perhaps the time is ripe for the ever-blooming Tree of
Files to send forth new fruits.

Peace, love, hypercubes, and ultrafilters to all

Mycroftiv

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[hiro]

> The major fly in the ointment is the outdated sha1 hash function used by venti

major? fly? there's no problem with how sha1 is used by venti right now.

a more substantial complaint would be that the venti/fossil system is
way too big.

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[Lucio De Re]

On 1/10/20, mycroftiv@sphericalharmony.com
<mycroftiv@sphericalharmony.com> wrote:
[ ... ]
>
> - The Present Day, SHA1, community
>
[ ... ]
> [ ...]  2020 seems like a Year of Change and I'm hoping for
> positive change in the Plan 9 community.  I'm particularly excited for
> the reincarnation of a real-life Plan 9 conference/hackathon event.
> Humans do better in real life social situations than internet
> diatribes and perhaps the time is ripe for the ever-blooming Tree of
> Files to send forth new fruits.
>
I'm totally with you, even if previous comments may have given a
different impression.

I'm probably the type of ally you would rather have as an enemy, but
if my behaviour is not found to be helpful, feel free to demand a
refund :-)

And I agree wholeheartedly: we can certainly make 2020 the year of
change. There may even be a wormhole between 2020 and 0x2020 (two
ASCII spaces) or 8224, by when surely Plan 9 will have conquered more
than just the Outer Space.

At this point I am starting to believe the superstition that there are
no coincidences. Your extremely sober comment is certainly one of many
such coincidences I have started to take note of. But then Isaac
Newton no doubt also believed in a hidden hand in his ageing years
:-).

Lucio.

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[Steve Simon]

fyi i have been running a fossil/venti pair since 2004 and it has been solid.

what is too big - honest question.

i thought the biggest issues are:

performance - the elegance and generality sacrifice some performance, though this is not a problem for me.

 lack of a tls fs interface - there is (imho) no excuse for a lack of tls these days.

no way for a user in (say) the ventidump group to request an immediate dump - handy for checkpointing a code release, though justing copying the release to a new dir is ok.

no way to push a vac score into the /archive hierarchy - nice for migrating in other backups.

all these are fixable and on my todo list but none have happened.

> On 10 Jan 2020, at 9:37 pm, hiro <23hiro@gmail.com> wrote:
> 
> 
>> 
>> The major fly in the ointment is the outdated sha1 hash function used by venti
> 
> major? fly? there's no problem with how sha1 is used by venti right now.
> 
> a more substantial complaint would be that the venti/fossil system is
> way too big.
> 
> ------------------------------------------
> 9fans: 9fans
> Permalink: https://9fans.topicbox.com/groups/9fans/Ta5071137bc09d294-Ma95d39bd7dce7deb0e654da7
> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[Rodrigo G. López]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

i'm sorry, what? lack of TLS?

go read 9front's tlssrv(8). it's been there for a while already.


-rodri

On Sat, Jan 11, 2020, 11:32 AM Steve Simon <steve@quintile.net> wrote:

>
> fyi i have been running a fossil/venti pair since 2004 and it has been
> solid.
>
> what is too big - honest question.
>
> i thought the biggest issues are:
>
> performance - the elegance and generality sacrifice some performance,
> though this is not a problem for me.
>
>  lack of a tls fs interface - there is (imho) no excuse for a lack of tls
> these days.
>
> no way for a user in (say) the ventidump group to request an immediate
> dump - handy for checkpointing a code release, though justing copying the
> release to a new dir is ok.
>
> no way to push a vac score into the /archive hierarchy - nice for
> migrating in other backups.
>
> all these are fixable and on my todo list but none have happened.
>
> > On 10 Jan 2020, at 9:37 pm, hiro <23hiro@gmail.com> wrote:
> >
> > 
> >>
> >> The major fly in the ointment is the outdated sha1 hash function used
> by venti
> >
> > major? fly? there's no problem with how sha1 is used by venti right now.
> >
> > a more substantial complaint would be that the venti/fossil system is
> > way too big.
>
> ------------------------------------------
> 9fans: 9fans
> Permalink:
> https://9fans.topicbox.com/groups/9fans/Ta5071137bc09d294-M7a56202f84a795cc3e7d9a1e
> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
>

[-- Attachment #2: Type: text/html, Size: 2327 bytes --]

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[Steve Simon]

[-- Attachment #1: Type: text/plain, Size: 2315 bytes --]


using tlssrv is an option, as is import, but what was saying it fossil’s native 9p server does not support tls, which it doesn't - i dont believe anyone has added it.

the issue is 9boot would need tls to be able download the kernel from a file server and it doesn’t have it. this is the labs boot procedure of course.

9front booting uses a full kernel to bootstrap (correct me if i am wrong) so tls is not a problem there.

anyway, fossil does not have tls and i would like to have. such a change would mean changes to the labs distro boot proceedure

-Steve


> On 11 Jan 2020, at 10:42 am, Rodrigo G. López <rodrigosloop@gmail.com> wrote:
> 
> 
> i'm sorry, what? lack of TLS?
> 
> go read 9front's tlssrv(8). it's been there for a while already.
> 
> 
> -rodri
> 
>> On Sat, Jan 11, 2020, 11:32 AM Steve Simon <steve@quintile.net> wrote:
>> 
>> fyi i have been running a fossil/venti pair since 2004 and it has been solid.
>> 
>> what is too big - honest question.
>> 
>> i thought the biggest issues are:
>> 
>> performance - the elegance and generality sacrifice some performance, though this is not a problem for me.
>> 
>>  lack of a tls fs interface - there is (imho) no excuse for a lack of tls these days.
>> 
>> no way for a user in (say) the ventidump group to request an immediate dump - handy for checkpointing a code release, though justing copying the release to a new dir is ok.
>> 
>> no way to push a vac score into the /archive hierarchy - nice for migrating in other backups.
>> 
>> all these are fixable and on my todo list but none have happened.
>> 
>> > On 10 Jan 2020, at 9:37 pm, hiro <23hiro@gmail.com> wrote:
>> > 
>> > 
>> >> 
>> >> The major fly in the ointment is the outdated sha1 hash function used by venti
>> > 
>> > major? fly? there's no problem with how sha1 is used by venti right now.
>> > 
>> > a more substantial complaint would be that the venti/fossil system is
>> > way too big.
>> 
>> ------------------------------------------
>> 9fans: 9fans
>> Permalink: https://9fans.topicbox.com/groups/9fans/Ta5071137bc09d294-M7a56202f84a795cc3e7d9a1e
>> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
> 
> 9fans / 9fans / see discussions + participants + delivery options Permalink

[-- Attachment #2: Type: text/html, Size: 4071 bytes --]

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[Rodrigo G. López]

[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]

yes that's right.

On Sat, Jan 11, 2020, 4:57 PM Steve Simon <steve@quintile.net> wrote:

>
> using tlssrv is an option, as is import, but what was saying it fossil’s
> native 9p server does not support tls, which it doesn't - i dont believe
> anyone has added it.
>
> the issue is 9boot would need tls to be able download the kernel from a
> file server and it doesn’t have it. this is the labs boot procedure of
> course.
>
> 9front booting uses a full kernel to bootstrap (correct me if i am wrong)
> so tls is not a problem there.
>
> anyway, fossil does not have tls and i would like to have. such a change
> would mean changes to the labs distro boot proceedure
>
> -Steve
>
>
> On 11 Jan 2020, at 10:42 am, Rodrigo G. López <rodrigosloop@gmail.com>
> wrote:
>
> 
> i'm sorry, what? lack of TLS?
>
> go read 9front's tlssrv(8). it's been there for a while already.
>
>
> -rodri
>
> On Sat, Jan 11, 2020, 11:32 AM Steve Simon <steve@quintile.net> wrote:
>
>>
>> fyi i have been running a fossil/venti pair since 2004 and it has been
>> solid.
>>
>> what is too big - honest question.
>>
>> i thought the biggest issues are:
>>
>> performance - the elegance and generality sacrifice some performance,
>> though this is not a problem for me.
>>
>>  lack of a tls fs interface - there is (imho) no excuse for a lack of tls
>> these days.
>>
>> no way for a user in (say) the ventidump group to request an immediate
>> dump - handy for checkpointing a code release, though justing copying the
>> release to a new dir is ok.
>>
>> no way to push a vac score into the /archive hierarchy - nice for
>> migrating in other backups.
>>
>> all these are fixable and on my todo list but none have happened.
>>
>> > On 10 Jan 2020, at 9:37 pm, hiro <23hiro@gmail.com> wrote:
>> >
>> > 
>> >>
>> >> The major fly in the ointment is the outdated sha1 hash function used
>> by venti
>> >
>> > major? fly? there's no problem with how sha1 is used by venti right now.
>> >
>> > a more substantial complaint would be that the venti/fossil system is
>> > way too big.
>>
>> ------------------------------------------
>> 9fans: 9fans
>> Permalink:
>> https://9fans.topicbox.com/groups/9fans/Ta5071137bc09d294-M7a56202f84a795cc3e7d9a1e
>> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
>>
> *9fans <https://9fans.topicbox.com/latest>* / 9fans / see discussions
> <https://9fans.topicbox.com/groups/9fans> + participants
> <https://9fans.topicbox.com/groups/9fans/members> + delivery options
> <https://9fans.topicbox.com/groups/9fans/subscription> Permalink
> <https://9fans.topicbox.com/groups/9fans/Ta5071137bc09d294-Mfc10314178ec66a8611fcd7b>
>

[-- Attachment #2: Type: text/html, Size: 4673 bytes --]

Re: [9fans] notes on fossil, ANTS, and 9front/Bell labs controversies

[Richard Miller]

steve@quintile.net:
> the issue is 9boot would need tls to be able download the kernel from a
> file server and it doesn’t have it. this is the labs boot procedure of
> course.
>
> 9front booting uses a full kernel to bootstrap (correct me if i am wrong)
> so tls is not a problem there.

There isn't a single "labs boot procedure".  Any 4th edition Plan 9 system
can use the kernel to load another kernel, with whatever embellishments you
care to dream up.  That's what /dev/reboot is for.

If you want to boot over the network mediated by tls, you need a bit of
local storage to hold a (stripped down if you like) first-stage kernel
and a few commands (which may be embedded in the in-kernel /boot filesystem).
That kernel can run a small rc script to do this:
  srv -e 'tlsclient tcp!$fs!777' reboot /n/reboot
  reboot /n/reboot/path/to/next/kernel
On the server you need a corresponding /bin/service/tcp777 with
  #!/bin/rc
  /bin/tlssrv -c /path/to/cert.pem -l listen /bin/aux/trampoline -9 tcp!127.1!564

No exotic extra facilities needed: tlssrv and tlsclient were introduced
with 4th edition Plan 9 in 2002.