"And the key for all of this is motivation and incentives, as any good
historian will tell you.  This is true whether probing the start of
wars, or the decline of a technical community or tradition."

This.

I work for the Large Synoptic Survey Telescope.  I'm in the Data Management
group, and specifically in the Science Quality and Reliability Engineering
team.  The 50,000 foot view of what we do is try to bring software
engineering to astronomical software.

In general, the thing about scientific software is that, to put it crudely,
no one gets a Nobel Prize for software.  There's a very strong incentive to
write a thing that will solve whatever particular problem you need solved
for your paper, and no more.  There's also the (highly correlated) problem
that, to an established researcher, graduate student labor is free, and
your graduate students want to finish their thesis, not engineer quality
software.

Whereas what I'd like to do is factor the common infrastructure--of which
there is a lot--out of the various teetering stacks of special-purpose
software and create some sane and maintainable infrastructure that
individual researchers can easily and relatively gracefully extend to
answer their specific questions.

Adam

On Wed, Jun 26, 2019 at 8:12 AM Theodore Ts'o <tytso@mit.edu> wrote:

> On Tue, Jun 25, 2019 at 07:56:46PM -0700, Larry McVoy wrote:
> > > It might not be, but it is definitely relevant to Unix.  Arguably the
> > > drivers of Unix's development movement away from R&D-focused places and
> > > toward product-oriented entities had at least a little to do with
> > > Larry's topic of complaint.  Product managers gained the ammunition to
> > > demand sustainable development practices, while R&D got a little
> leaner,
> > > a little more focused on demonstrating the thesis, a little less
> focused
> > > on who might need to run this code five years on...
> >
> > In the good old days at Sun, we were very focussed on who would run
> > this code for decades to come.  I think the engineers at Sun were very
> > focussed on helping people, the reason we were there was because the
> > work we did helped people.  The leverage was how much work we could
> > do versus how much that helped people.  That is product oriented.
> >
> > I think the reason that any engineer works is because they feel like
> > their work helps someone.  As an engineer, I wanted to go to the place
> > and do the work that had the best chance of helping someone.  All of
> > Sun, when I was there, was like that.  We were there to help.  Yeah,
> > of course, we wanted to make money, but all of us wanted to help.
> > It's the dream, you do work, your work helps.
>
> Motivations and incentives are a very big and important aspect which
> is often overlooked in large scale projects.
>
> For example, one of the really big problems with device drivers in the
> embedded space is that the team that works on SOC version X gets
> disbanded, and immediately reassigned to SOC verison X+1, sometimes
> before product has even shipped.  Having one device driver that works
> for SOC versions N, N+1, N+2, ... N+5, is really important from a
> maintainability and being able to send out bug fixes for security
> flaws.  However, it means that whenever you make changes, you need to
> test on N different older versions.  And between the need to release
> product quickly, and the fact that engineers are !@#@! expensive, and
> the teams constantly getting formed and reformed, it's much easier to
> do code reuse by copying, and so you have N different versions of a
> device driver in a Board Support Package version of the Linux kernel
> shipping by a SOC vendor.
>
> Unfortunately, I have to disagree with Larry, there are many, many
> engineers who works because they get a paycheck, and so they go home
> at 5pm.  Some people might be free to improve their code on their own
> time, or late at night, but corporation also preach "work/life
> balance" --- and then don't fund time for making code long-term
> maintainable or reducing tech debt.
>
> Open source helps because embarassment can be a great motivator, but
> more important are the fact that there are people who are empowered to
> say "no" who don't work for the corporation who is trying to cut
> corners, and who have a higher allegiance to the codebase than their
> employer.
>
> There is a similar related issue around publishing papers to document
> great ideas.  This takes time away from product development, and it
> used to be that Sun was really prolific at documenting their technical
> innovations at conferences like Usenix.  Over time, the academic
> traditions started dying off, and managers who came from that
> tradition moved on, retired, or got promoted beyond the point where
> they could encourage engineers to do that work.  And it wasn't just at
> Sun; I was working at IBM when IBM decided to take away the (de
> minimus) bonus for publishing papers at conferences.  But at the
> Usenix board, I remember looking at a chart of the declining number of
> ATC papers coming from industry over time.   And it was very depressing...
>
> And the key for all of this is motivation and incentives, as any good
> historian will tell you.  This is true whether probing the start of
> wars, or the decline of a technical community or tradition.
>
>                                                  - Ted
>