Re: [TUHS] PDP-11 legacy, C, and modern architectures

["Steve Johnson" <scj@yaccman.com>]

[-- Attachment #1: Type: text/plain, Size: 2880 bytes --]


I agree that C is a bad language for parallelism, and, like it or not,
that's what today's hardware is giving us -- not speed, but many
independent processors.  But I'd argue that its problem isn't that it
is not low-level, but that it is not high-level enough.  A language
like MATLAB, whose basic data object is an N-diemsional tensor, can
make impressive use of parallel hardware.

Consider matrix multiplication.   Multiplying two NxN arrays to get
another NxN array is a classic data-parallel problem -- each value in
the result matrix is completely independent of every other one -- in
theory, we could dedicate a processor to each output element, and
would not need any cache coherency or locking mechanism -- just let
them go at it -- the trickiest part is deciding you are finished.

The reason we know we are data parallel is not because of any feature
of the language -- it's because of the mathematical structure of the
problem.  While it's easy to write a matrix multiply function in C
(as it is in most languages), just the fact that the arguments are
pointers is enough to make data parallelism invisible from within the
function.  You can bolt on additional features that, in effect, tell
the compiler it should treat the inputs as independent and
non-overlapping, but this is just the tip of the iceberg -- real
parallel problems see this in spaces.  

The other hardware factor that comes into play is that hardware,
especially memories, have physical limits in what they can do.  So
the "ideal" matrix multiply with a processor for each output element
would suffer because many of the processors would be trying to read
the same memory at the same time.  Some would be bound to fail,
requiring the ability to stack requests and restart them, as well as
pause the processor until the data was available.   (note that, in
this and many other cases, we don't need cache coherency because the
input data is not changing while we are using it).  The obvious way
around this is to divide the memory in to many small memories that are
close to the processors, so memory access is not the bottleneck.

And this is where C (and Python) fall shortest.  The idea that there
is one memory space of semi-infinite size, and all pointers point into
it and all variables live in it almost forces attempts at parallelism
to be expensive and performance-killing.  And yet, because of C's
limited, "low-level" approach to data, we are stuck.  Being able to
declare that something is a tensor that will be unchanging when used,
can be distributed across many small memories to prevent data
bottlenecks when reading and writing, and changed only in limited and
controlled ways is the key to unlocking serious performance.

Steve

PS: for some further thoughts, see
https://wavecomp.ai/blog/auto-hardware-and-ai



[-- Attachment #2: Type: text/html, Size: 3018 bytes --]