[COFF] COBOL.

[clemc at ccc.com (Clem Cole)]

Dan - I think you hit the problem on the head.   Some of us broke into the
biz with large IBM Systems.   My own first system's programming job was for
TSS/360 (written in BAL of course).

The reason why this is acute is actually predicted by the late Prof Clay
Christensen in his book "*The Innovators **Delima*."     During the heady
days of IBM mainframes in the late 1960s, Wall street/business computing
systems were all custom built.   If you walked in the Morgan Stanley,
Chase, Citi, much less NYSE there were rooms and rooms of cobol programmers
writing custom code.  Just for that firm.  Nothing was shared, each used
his/her pool of programmers to survive and try to get an edge or the others.

More importantly, they were all using IBM ISAM databases behind all the
Cobol 'business logic.'  Hey, life was good.  The business boomed.  Guess
what, government programmers pretty much did the same thing, although they
were not competing, they were just trying to build systems to support the
different programs they had (collecting taxes, paying benefits, *etc*).
 What was common in all cases, was that the institution (commercial or
government) behind all that programming and system deployment had lots of
$s.   So IBM, and leaches like Ross Perot's firm (there were many like him
- his was just one of the biggest), performed what Christensen calls
'sustaining technology.'  They kept making the same things, faster big
better, *etc.*; because that's what their customers wanted and were asking
to buy.

But in the early 1970s four things happened:  1.) Edgar F. Codd, an
Oxford-educated mathematician working at the IBM San Jose Research Lab,
published a paper showing how information stored in large databases could
be accessed without knowing how the information was structured or where it
resided in the database - that is, he invented the Relational Model for
DBs,   2.) the Super-Mini's like the VAX or the like came on the scene, 3.)
Oracle would eventually clone Codd's work and created a cheap DB that ran
on #2, and 4.) companies like Oracle, BANN, SAP built >>re-configurable<<
applications that worked a lot like the custom ones that had been built by
the huge Cobol teams but were good enough for most firms. * i.e*. a general
ledger/accounting system, the small banking system. A car dealership or
super-market system.  In fact, what arose was a legion of folks like the
big-8 that would take those systems and set them up for smaller firms.
Customize some of the reports, but many customers used the same codes
(rinse and repeat...)

And something else happened...   Oracle did not try to attack IBM head-on.
Larry and team followed Christensen's idea of looking for *new customers/a
new market*, where what they sold might not be originally quite as sexy as
the custom things the big financial folks/government were building for
themselves, but they were a load cheaper and were 'good enough' for most
smaller firms.

The rest, as they say, is history.   Eventually, the systems got bigger and
bigger, and more capability and the Oracle, SAP, *et al* got better and
better and could do more.   Eventually, Wall Street started to switch to
the new less customized world.   And in many places, eventually, the
reconfigurable applications for accounting, payroll et al, slowly started
to get replaced >>on the commercial side<<.   The key point was that Oracle
and the commercial applications and database side of the world, disrupted
the original mainframe in the financial sector ---> except for one type of
customer: Government.

There were no new customers for these systems, so governments just kept
using the custom stuff they had.  They did start to put web style
front-ends one them.   But what elected official is going to try to get
money to replace the current system they have?  It works and
redo/rewrite/modernization is not cost-effective because there are no
economies to scale.   So unless the Fed's put together a program for all 50
states (which is what they did for the IRS and the new system is still
having issues).   Remember, redoing something like that is not going to get
you re-elected and it sure is going to take money away from a special
project that might.

That said, here is a question for you.   I suspect you have used something
sqlite(3), Berkeley DB or any simple relational DB for something at
some point.   But did you ever learn how to use a Codisyl/ISAM DB?   I
bet there
really was no reason for you to learn how and frankly it was not likely to
be on your system.  But take a look --> trying front-ending ISAM style
queries from JavaScript.   We have a whole set of routines and standards to
call SQL -- Cobol calling ISAM is much more ad hoc (IBM called these
'access methods').  The big UNIX idea of everything is file is the
complete antithesis of the codasyl.

Do you remember what happened during the Obama administration, when they
rolled out ACA with most states?   The web sites were crashing because the
databases couldn't hack it and keep up the queries.   The web-front ends
scaled, but those back-ends databases were never designed to be accessed in
that way. The problem is really not Cobol as much as the design of those
systems was all custom and assumes a very structured back-end with
structured data.

Those systems grew up over time with small incremental changes.  Most
changes were forced when new legislation came in and new features had to be
added.  But they stayed as ther were and were patched to keep going.   So
somewhere, somebody put a pretty front end with the web, but in back
office, it's still an old system.  Think about, you live in MA.   Have you
tried to get a new driver's license with the Real-ID from the DMV?  Same
issue -- great web front end.   But the DMV system behind is all is circa
late 1960 Cobol/ISAM -- never replaced.  It's a nightmare and a lot of it
manual.  Same with voter registration systems for the towns.   A couple of
years ago, my sister arranged to get a 9-track (EBCDIC) tape from some of
the towns with all voters on it for a story she was writing for the Boston
Globe and she needed to do some statistical analysis. Fortunately, I still
had a working 9-track and I can deal with almost anything.  It was clearly
an ISAM database dump.   Between dd(1) and a few sed/grep/awk scripts, she
was is business.  IIRC, I started to try to put it into Berkeley DB, but
mostly just converting to an ASCII form tabular form and being able to run
grep et al was good enough.   She shows her town clerk what she had, and
the clerk wanted it.  The clerk told my sister that she had been trying for
years to get the state to give her some of that data in the form we
wanted.   She had been told it could not be done etc...  The reason is they
no longer had anyone that knew anything about the DB.

So the problem is in middle and higher management.  What Christensen points
out, is when you can create a new market, you can change things quickly
(disrupt the old market in his terms).  But without those market forces,
the developers will continue to build you better and better systems that
are incremental over the old one.  If there is not a new market there to
disrupt the old one stays.


BTW:  In my world, HPC, this is why Fortran is still king for the same
reasons, and there it is worse because the math has not changed and we have
50-70 years of datasets that would have to be rechecked and revalidated if
the programs changed.  Check out:  https://www.archer.ac.uk/status/codes/





On Mon, Apr 13, 2020 at 2:00 PM Dan Cross <crossd at gmail.com> wrote:

> So I imagine that most readers of this list have heard that a number of US
> states are actively looking for COBOL programmers.
>
> If you have not, the background is that, in the US, a number of
> unemployment insurance systems have mainframe backends running applications
> mostly written in COBOL. Due to the economic downturn as a result of
> COVID-19, these systems are being overwhelmed with unprecedented numbers of
> newly-unemployed people filing claims. The situation is so dire that the
> Governor of New Jersey mentioned it during a press conference.
>
> This has led to a number of articles in the popular press about this
> situation, some rather sensational: "60 year old programming language
> prevents people filing for unemployment!" E.g.,
> https://www.cnn.com/2020/04/08/business/coronavirus-cobol-programmers-new-jersey-trnd/index.html
>
> On the other hand, some are rather more measured:
> https://spectrum.ieee.org/tech-talk/computing/software/cobol-programmers-answer-call-unemployment-benefits-systems
>
> I suspect the real problem is less "COBOL and mainframes" and more
> organizational along the lines of lack of investment in training,
> maintenance and modernization. I can't imagine that bureaucrats are
> particularly motivated to invest in technology that mostly "just works."
>
> But the news coverage has led to a predictable set of rebuttals from the
> mainframe faithful on places like Twitter; they point out that COBOL has
> been updated by recent standards in 2002 and 2014 and is being unfairly
> blamed for the present failures, which arguably have more to do with
> organizational issues than technology. However, the pendulum seems to have
> swung too far with their arguments in that they're now asserting that COBOL
> codebases are uniformly masterworks. I don't buy that.
>
> I find all of this interesting. I don't know COBOL, nor all that much
> about it, save for some generalities about its origin and Grace Hopper's
> involvement in its creation. However, in the last few days I've read up on
> it a bit and see very little to recommend it: the type and scoping rules
> are a mess, things like the 'ALTER' statement and the ability to cascade
> procedure invocations via the 'THRU' keyword seem like a recipe for
> spaghetti code, and while they added an object system in 2002, it doesn't
> seem to integrate with the rest of the language coherently and I don't see
> it doing anything that can't be done in any other OO language. And of
> course the syntax is abjectly horrible. All in all, it may not be the cause
> of the current problems, but I don't know why anyone would be much of a fan
> of it and unless you're already sitting on a mountain of COBOL code (which,
> in fairness, many organizations in government, insurance and finance
> are...) I wouldn't invest in it.
>
> I read an estimate somewhere that there are something like 380 billion
> lines of COBOL out there, and another 5 billion are written annually
> (mostly by body shops in the BRIC countries?). That's a lot of code; surely
> not all of it is good.
>
> So....What do folks think? Is COBOL being unfairly maligned simply due to
> age, or is it really a problem? How about continued reliance on IBM
> mainframes: strategic assets or mistakes?
>
>         - Dan C.
>
> (PS: I did look up the specs for the IBM z15. It's an impressive machine,
> but without an existing mainframe investment, I wouldn't get one.)
> _______________________________________________
> COFF mailing list
> COFF at minnie.tuhs.org
> https://minnie.tuhs.org/cgi-bin/mailman/listinfo/coff
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://minnie.tuhs.org/pipermail/coff/attachments/20200413/cc0c1d06/attachment-0001.html>