The Unix Heritage Society mailing list
 help / color / mirror / Atom feed
From: Warner Losh <imp@bsdimp.com>
To: "Theodore Ts'o" <tytso@mit.edu>
Cc: tuhs@tuhs.org
Subject: [TUHS] Re: AIX moved into maintainance mode
Date: Thu, 19 Jan 2023 17:37:27 -0700	[thread overview]
Message-ID: <CANCZdfr6=EBvN=+EtL0_L5oC=sixQ9e4zsuhgtVUPbgPm2+cNA@mail.gmail.com> (raw)
In-Reply-To: <Y8nPPOhH1D4H0M0C@mit.edu>

[-- Attachment #1: Type: text/plain, Size: 8660 bytes --]

On Thu, Jan 19, 2023 at 4:16 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Thu, Jan 19, 2023 at 09:58:48AM -0700, Warner Losh wrote:
> > On Thu, Jan 19, 2023 at 9:41 AM Dan Cross <crossd@gmail.com> wrote:
> >
> > > But it's interesting the way the "Gods of BSD vs the rebel alliance"
> > > thing seems to have inverted itself. Getting stuff done in Linux these
> > > days is pretty hard; oh sure, I suppose if you whip off a patch fixing
> > > a typo in a comment or something, someone will just apply it. But if
> > > you want to do something substantial, you have to be willing to invest
> > > a lot of time and effort in shepherding it through the upstreaming
> > > process, which implies you've got to have resources backing that
> > > effort.....
> >
> > This matches my experience: Lots of gatekeepers, any one of which can
> take
> > a disliking to your submission for what, at times, seems like arbitrary
> and
> > capricious reasons. If you make it to the 'in crowd' it becomes more of a
> > rubber stamp at times...  I have recently been successful at submitting
> an
> > obvious fix to a tiny backwater area of the kernel without undue stress
> > though...  But I submitted it to the maintainer of the area, who then
> > submitted it to the 'greater maintainer', and then to the greater
> > maintainer and then to the tree. So my tiny fix has more Signed-off-by:
> > lines than lines of change and took about two weeks to make its way all
> the
> > way up into the repo...   For me, it took about 2x as long to prep and
> send
> > the change than it does for the direct commit access I have for FreeBSD,
> > but I've spent more than 10x on other submissions that ultimately didn't
> > make it in.
>
> I'll note that a lot of this is a matter of scale.  There are roughly
> 15,000 commits added to the Linux kernel per 9 week release cycle.
> That translates to roughly 10 commits per hour, 7 days a week, 24
> hours a day.
>

All of FreeBSD is right around 5-6 per hour: FreeBSD/ports is about 4 per
hour, FreeBSD/src is about 1 per hour FreeBSD's kernel is 1/2 per hour.
Docs is about 1/5/hr.

I'm surprised the commit rate to the Linux kernel is only 20x that of
FreeBSD's kernel and only about twice that of the whole FreeBSD project.


> From an upstream maintainer's perspective, what matters is whether a
> commit handled in time for it to make the next release train.  So
> whether it takes two days or two weeks to get a patch in the repo is
> not considered important.  What *is* important is that commits that
> are up for review before, say, at least two weeks before the opening
> of the merge window, are reviewed in time for them to be pulled into
> Linus's tree when the merge window next opens.  If it is an urgent bug
> fix, then it will be acted upon more expeditiously, but the reality is
> most users won't see it until the the fix is backported into the
> Long-Term Stable kernel that their distribution or product kernel is
> derived from.
>

I understand review. I have to do it on lots of other projects I contribute
to. However, there's a lot of inefficiency in how this task happens.


> For more substantial contributions, one of the reasons why there are
> so many gatekeepers is mainly due to the fact that we've had far too
> many cases of "drive-by contributions" where a company tries to dump a
> whole new file system, or massive changes to a large number of systems
> --- and then once the code lands in the tree, the developers vanish.
> And now we're left with the question of whether we just drop the
> subsystems, and screw-over the users who have started to depend on the
> new functionality.  This may be one of the places where the Linux
> culture of "thou shall never cause user-space visible regressions" has
> its downside --- it means that it's a lot harder to accept new
> functionality unless there is strong confidence that contributor is
> there for the long haul, and admittedly that's a lot easier if you are
> a member of the "in-crowd".
>

I understand why things were built, but that doesn't mean they are the
right solution to the problems the policies are trying to solve.


> > I looked into Linux's processes to improve FreeBSD's. And came to the
> > conclusion that in large part they succeed despite their processes, not
> > because of them. They have too much overhead, rely too much on magic bots
> > that are hard to replicate and I'm astonished that things work as well as
> > they do.
>
> I think that if you divide the total overhead by the number of commits
> that land in each release, the overhead isn't really that bad.  Sure,
> a tractor-trailer truck has a lot more overhead than say, a bicycle;
> but you can haul an awful lot more in a semi-trailer truck than you
> can in a bicycle rack.  Hence, the overhead of a semi is arguably much
> less, once you take into account how much you can fit in a
> tractor-trailer.
>

I feel like I've been hit by a tractor trailer when I try to contribute to
Linux sometimes... But the overhead is feels way higher than
other projects that are moving at the same order of magnitude as
the Linux kernel. I'd have expected the rate to be closer to 100/hr
or 1000/hr given the differential usage between it and FreeBSD...


> Also, it's actually not *that* expensive, even in absolute terms.  For
> example, I run about 26 hours worth of regression tests (using a dozen
> VM's, so the wall clock time is about 2 hours), using gce-xfstests[1],
> and the retail cost if I didn't have corporate sponsorship is less
> than $2 USD for a complete set of ext4 tests.  And I made a point of
> making well documented and easy for others to standup so they can test
> their own file systems if they want.  The reason why I did all of this
> packaging work was to try to get graduate students to understand how
> much work left to get a publishable file system, such as say
> BetrFS[3], into a production-ready state for real-world use.  :-)
>

Indeed. These sorts of tests are indeed a good thing, and are quite
a bit more extensive than similar projects I've contributed to.


> [1] https://thunk.org/gce-xfstests
> [2]
> https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md
> [3]
> https://www.usenix.org/conference/fast15/technical-sessions/presentation/jannen
>
>
> > It's a grown culture / process that relies on old tools mixed with
> > new in weird ways you'd never think of standing up today. Things can be
> > learned from it, but it seems to be a unique snowflake relative to all
> the
> > other projects I looked at...
>
> That's fair enough.  What is needed for a particular scale may be
> massive overkill for another.  All projects will probably be needing
> to adopt different processes or tools as they grow.
>
> Similar statements have made about whether startups or other small
> projects should use Kubernetes[4].  Kubernetes was designed by Google
> based on their learnings from their in-house cluster management
> systems, borg.  But if you aren't running at that scale, it may have
> more overhead and complexity than what really makes sense.
>
> [4] https://blog.porter.run/when-to-use-kubernetes-as-a-startup/
>
> I'm reminded of a talk that given by an engineer from Alibaba at
> LinuxCon China 2017 where he was bragging about how they had finally
> achieved the scaling necessasry so they could support over a thousand
> servers in a data center, and how this was an incredible achievement.
>
> That was especially funny to me, since right about that time, Google
> had just finished an engineering effort to scale our cluster
> management software *down* to a O(thousand) servers, in order to
> efficiently support "mini-clusters" that could be deployed in smaller
> data centers various countries in Europe, Asia, etc.  :-)
>

Indeed. Netflix has been operating a global video operation with
thousands of servers for many years now... Thought the problem
sets are different, and the non-video-serving folks use a lot more
in the cloud for all the data analysis...


> And later, even *more* engineering work was needed to efficiently
> support O(hundreds) servers for Stadia.
>
> What works well at one scale, may not work well at others, either when
> scaling up or scaling down.
>

Indeed. But one also should take a look at the overheads and inefficiencies,
or you run the risk that someone else will be able to operate at a similar
scale with less effort.
And it seems to me, as mostly an outsider, that the overheads might become
a bigger problem, especially if we have a prolonged downturn.  Just my
opinion,
though.

Warner


> Cheers,
>
>                                         - Ted
>

[-- Attachment #2: Type: text/html, Size: 11560 bytes --]

  reply	other threads:[~2023-01-20  0:38 UTC|newest]

Thread overview: 101+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-01-18  9:43 [TUHS] " arnold
2023-01-18 14:46 ` [TUHS] " Phil Budne
2023-01-18 14:55   ` Ralph Corderoy
2023-01-19 14:42     ` Liam Proven
2023-01-19 15:04       ` Warner Losh
2023-01-19 15:15         ` Liam Proven
2023-01-18 15:13 ` arnold
2023-01-18 15:14   ` Larry McVoy
2023-01-18 16:10     ` segaloco via TUHS
2023-01-18 16:19       ` Stuff Received
2023-01-18 16:19       ` Larry McVoy
2023-01-18 16:27         ` [TUHS] Maintenance mode on AIX Ron Natalie
2023-01-18 16:38           ` [TUHS] " Larry McVoy
2023-01-18 16:59             ` Clem Cole
2023-01-18 17:08               ` segaloco via TUHS
2023-01-18 17:21                 ` Will Senn
2023-01-18 19:50                   ` David Barto
2023-01-19 14:25                   ` Liam Proven
2023-01-18 20:34             ` Arno Griffioen via TUHS
2023-01-18 20:50               ` Brad Spencer
2023-01-18 16:36         ` [TUHS] Re: AIX moved into maintainance mode Will Senn
2023-01-18 16:42           ` Larry McVoy
2023-01-18 16:57             ` Will Senn
2023-01-18 17:16               ` Larry McVoy
2023-01-18 17:25                 ` Will Senn
2023-01-18 21:09                   ` segaloco via TUHS
2023-01-18 21:18                     ` Kevin Bowling
2023-01-19  1:13                     ` Joseph Holsten
2023-01-19 15:04                     ` Liam Proven
2023-01-18 19:25             ` Dave Horsfall
2023-01-19 15:02             ` Liam Proven
2023-01-19 15:12               ` arnold
2023-01-19 17:46                 ` Steffen Nurpmeso
2023-01-19 18:24               ` Doug McIntyre
2023-01-19 19:44                 ` Chet Ramey
2023-01-20 13:09                 ` Liam Proven
2023-01-20 14:37                   ` Harald Arnesen
2023-01-18 16:48         ` segaloco via TUHS
2023-01-19  0:54         ` Adam Thornton
2023-01-19  1:09           ` Larry McVoy
2023-01-20 18:38             ` Theodore Ts'o
2023-01-20 18:57               ` Dan Cross
2023-01-20 19:48                 ` John Cowan
2023-01-20 20:04                   ` Dan Cross
2023-01-20 19:08               ` Kevin Bowling
2023-01-19  1:17           ` Marc Donner
2023-01-19  1:26             ` Joseph Holsten
2023-01-20 15:53               ` Marc Donner
2023-01-19 14:45         ` Liam Proven
2023-01-19 15:05           ` Dan Cross
2023-01-19 16:59             ` Bakul Shah
2023-01-19 19:33               ` [TUHS] The death of general purpose computers, was - " Will Senn
2023-01-19 20:09                 ` [TUHS] " segaloco via TUHS
2023-01-19 20:59                   ` Rich Morin
2023-01-19 21:11                     ` segaloco via TUHS
2023-01-20 13:30                   ` Liam Proven
2023-01-20 15:51                     ` segaloco via TUHS
2023-01-20 15:56                       ` Rich Morin
2023-01-20 16:24                         ` segaloco via TUHS
2023-01-20 18:21                           ` G. Branden Robinson
2023-01-20 18:33                             ` segaloco via TUHS
2023-01-18 18:58       ` [TUHS] " Steve Nickolas
2023-01-19  8:02     ` arnold
2023-01-19 15:04       ` Larry McVoy
2023-01-19 15:20         ` Warner Losh
2023-01-19 15:23           ` Larry McVoy
2023-01-19 16:40           ` Dan Cross
2023-01-19 16:58             ` Warner Losh
2023-01-19 23:16               ` Theodore Ts'o
2023-01-20  0:37                 ` Warner Losh [this message]
2023-01-20  1:22                   ` Steve Nickolas
2023-01-19 17:02             ` Steve Nickolas
2023-01-19 17:19               ` Adam Thornton
2023-01-19 18:22                 ` segaloco via TUHS
2023-01-19 19:07                   ` Kevin Bowling
2023-01-19 21:08                     ` Joseph Holsten
2023-01-19 20:01                 ` [TUHS] The era of general purpose computing (Re: " Bakul Shah
2023-01-19 22:23                   ` [TUHS] " Luther Johnson
2023-01-20  1:10                     ` John Cowan
2023-01-20  1:15                       ` Luther Johnson
2023-01-21 18:12                         ` arnold
2023-01-21 18:43                           ` Luther Johnson
2023-01-19 22:29                   ` Rich Salz
2023-01-19 22:39                     ` Luther Johnson
2023-01-19 22:41                       ` Luther Johnson
2023-01-19 22:40                     ` Jon Steinhart
2023-01-19 23:24                     ` segaloco via TUHS
2023-01-19 23:44                       ` Rich Salz
2023-01-19 23:51                         ` segaloco via TUHS
2023-01-20  0:20                           ` [TUHS] owner maintenance (Re: " Charles H Sauer (he/him)
2023-01-20  0:36                             ` [TUHS] " Larry McVoy
2023-01-20  0:47                         ` [TUHS] " Yeechang Lee
2023-01-20  0:55                           ` George Michaelson
2023-01-20  1:05                             ` Rich Salz
2023-01-20  1:10                               ` George Michaelson
2023-01-20  2:27                     ` Dan Cross
2023-01-18 21:20 ` [TUHS] " Theodore Ts'o
2023-01-18 21:27   ` Kevin Bowling
2023-01-19  2:17   ` Jim Carpenter
2023-01-19 21:15 ` Will Senn
2023-01-19 21:34   ` Drew Diver

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CANCZdfr6=EBvN=+EtL0_L5oC=sixQ9e4zsuhgtVUPbgPm2+cNA@mail.gmail.com' \
    --to=imp@bsdimp.com \
    --cc=tuhs@tuhs.org \
    --cc=tytso@mit.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).