Final (?) time64 proposal

Final (?) time64 proposal

[Rich Felker]

OK, doing the preliminaty work for time64 has helped to clarify a lot
of the undecideds about it. To summarize, so far I've:

- decoupled libc stat struct from kernel stat64 struct
- refactored all stat-family functions in terms of fstatat
- added support for statx syscall as a backend for fstatat
- refactored adjtime-family function in terms of clock_adjtime
- written all(?) compat shims for providing old ABI after switchover
  (out-of-tree, not committed)

My plan to go ahead looks like:

- time_t 64-bit
- suseconds_t 64-bit (to match kernel timeval with no conversion)
- timespec's long tv_nsec padded to 64-bit in endian-compatible manner
- struct stat layout kept, extended with 64-bit timespecs on the end
- struct utmpx layout kept, timeval moved to unused space at end
- struct rusage layout kept, timevals moved to unused space at end
- struct timex layout kept, timeval moved to unused space at end
- struct shmid_ds, msqid_ds, semid_ds layout kept, extended with
  64-bit time_t's on the ends
- new symbol names for all functions with time_t, timeval, timespec,
  itimerval, itimerspec, utimbuf, timeb, or stat in their signatures.
- keep existing symbol name for functions with utmpx or timex in their
  signatures, and fill in both old and new time fields.
- one of the above two (not sure which yet) for functions with rusage
  in their signatures. using the reserved space we have without a new
  symbol name is safe for musl ABI but not for glibc ABI-compat.
- compat shims (with the old symbol names) for all functions where a
  new symbol name is introduced.
- symbol redirection of the standard function names to the new symbols
  via the public headers.
- symbol redirection of the compat shim functions to the old symbol
  names via internal headers.
- new definitions of IPC_STAT, SHM_STAT, SHM_STAT_ANY, SEM_STAT,
  SEM_STAT_ANY, MSG_STAT, and MSG_STAT_ANY to expand the shmid_ds,
  semid_ds, and msqid_ds upper and lower time bits from their
  locations in the kernel struct to the new fields at the end.

Archs define via alltypes.h.in that they need symbol name redirection
and compat shims. this ensures that the knowledge is available to all
public headers but still arch-specific, since newly added 32-bit archs
(including a possible future ".2 ABI") will *not* have any redirection
or shims; they'll just be using 64-bit time_t from the start. This
includes even old archs we might add, like sparc.

One might wonder whether that affects glibc ABI-compat, e.g. for a
hypothetical sparc port. Yes. With the 64-bit time_t transition, and
the likelihood that glibc will do all this in a very different way
(e.g. different struct layouts, possibly even API-violating ones like
using long long for timex fields), I don't think it's going to be
practical to keep glibc ABI-compat for all the interfaces defined in
musl. Rather, I think it will end up making sense to preserve the
functionality via collaboration with the gcompat project, and probably
rewiring things so that gcompat or other foreign-ABI-compat libraries
get processed as intercepting symbol references from the app/lib
needing foreign-ABI-compat, rather than residing in the global symbol
namespace. This will also make it possible to fix the regexec/regoff_t
mismatch on 64-bit archs.

One possibility I'd like to consider is allowing a build with
something between the proposal and the ".2 ABI" - omitting the compat
shims from libc.a and/or libc.so. This is not the same thing as a ".2
ABI" because it wouldn't overhaul type definitions to be clean and
uniform and it wouldn't crete a *conflicting* ABI with the current .1
ABI. The new 64-bit time_t functions would use all their redirected
symbol names the same as described above, so binaries built with the
above scheme would work with or without this option. The only
difference would be that the compat shims would be missing, so that
old, 32-bit-time_t binaries and libraries would fail to load. This
would allow users concerned about the impact of having a mixed
ecosystem, or concerned about non-Y2038-compatible stuff creeping into
their systems, to ensure that everything is consistently using 64-bit
time_t.


With that said, the following are the functions I've identified as
still interfacing with the kernel in ways that involve time_t:

- clock_gettime **
- clock_settime **
- clock_adjtime **
- clock_nanosleep
- timer_gettime
- timer_settime
- timerfd_gettime
- timerfd_settime
- utimensat **
- mq_timedsend **
- mq_timedreceive **
- getitimer
- setitimer
- select
- pselect
- ppoll
- recvmmsg ?
- sigtimedwait
- semtimedop
- clock_getres
- sched_rr_get_interval
- __timedwait (backend for all timed futex waits)
- getrusage
- wait4
- wait3
- setsockopt
- getsockopt
- shmctl ***
- semctl ***
- msgctl ***
- ioctl
- recvmsg ?

The ones marked with ** absolutely need to use new kernel syscalls
when available, falling back to the old ones only if they're not,
because they need to accept or return absolute times that don't fit in
the legacy 32-bit interfaces. The ones marked with *** need to decode
new high bits provided through existing syscalls. The ones marked with
?, recvmsg and recvmmsg, potentially have to rewrite cmsgs if we want
SO_TIMESTAMP* to keep working on 32-bit archs without requiring a
kernel >=5.1. I'm not sure if this is a sufficiently widely-used
feature to warrant such a heavy hammer. We could just wait and see if
anything breaks, or ask distros to investigate, or do some code
search.

Aside from those special cases, the rest of the above only deal with
relative times, and can happily continue using the same syscalls
they're using; they just need to convert to/from userspace
timeval/timespec/etc. The special (**) ones above also need to convert
to/from userspace format in the fallback cases when new syscalls are
unavailable (ENOSYS).

Comments on any of the above are welcome. Especially please speak up
if you see any omissions/oversights.

Rich

Re: Final (?) time64 proposal

[Rich Felker]

On Wed, Jul 24, 2019 at 01:31:42AM -0400, Rich Felker wrote:
> With that said, the following are the functions I've identified as
> still interfacing with the kernel in ways that involve time_t:
> 
> - clock_gettime **
> - clock_settime **
> - clock_adjtime **
> - clock_nanosleep
> - timer_gettime
> - timer_settime
> - timerfd_gettime
> - timerfd_settime
> - utimensat **
> - mq_timedsend **
> - mq_timedreceive **
> - getitimer
> - setitimer
> - select
> - pselect
> - ppoll
> - recvmmsg ?
> - sigtimedwait
> - semtimedop
> - clock_getres
> - sched_rr_get_interval
> - __timedwait (backend for all timed futex waits)
> - getrusage
> - wait4
> - wait3
> - setsockopt
> - getsockopt
> - shmctl ***
> - semctl ***
> - msgctl ***
> - ioctl
> - recvmsg ?

I think a good next-step here would be modifying all of the above to
go through an intermediate kernel-type struct when using the existing
syscalls. This should not change behavior anywhere except for a slight
increase in size/time-spent, but will set things up so that, when the
time_t definition is switched over, everything should just start
working automatically with 64-bit time. It will also let us drop the
__fixup hacks in arch/x32/syscall_arch.h and
src/thread/x32/syscall_cp_fixup.c, analogous to how commit
01ae3fc6d48f4a45535189b7a6db286535af08ca let us drop the corresponding
mips hacks.

Rich

Re: Final (?) time64 proposal

[Rich Felker]

On Thu, Jul 25, 2019 at 01:18:32AM -0400, Rich Felker wrote:
> On Wed, Jul 24, 2019 at 01:31:42AM -0400, Rich Felker wrote:
> > With that said, the following are the functions I've identified as
> > still interfacing with the kernel in ways that involve time_t:
> > 
> > - clock_gettime **
> > [...]
> 
> I think a good next-step here would be modifying all of the above to
> go through an intermediate kernel-type struct when using the existing
> syscalls. This should not change behavior anywhere except for a slight
> increase in size/time-spent, but will set things up so that, when the
> time_t definition is switched over, everything should just start
> working automatically with 64-bit time. It will also let us drop the
> __fixup hacks in arch/x32/syscall_arch.h and
> src/thread/x32/syscall_cp_fixup.c, analogous to how commit
> 01ae3fc6d48f4a45535189b7a6db286535af08ca let us drop the corresponding
> mips hacks.

I'm not so sure about this approach now. Conversions will never be
needed for 64-bit archs, nor for new code paths using the time64
syscalls on 32-bit archs. Aside from x32 awfulness, the condition for
the new version of the code to be needed is SYS_*time64 being defined
and time_t being 64-bit. This suggests we might let x32/syscall_arch.h
define the SYS_*time64 macros, and undef the legacy names, so that it
gets treated like a 32-bit arch with no legacy time32 syscalls (i.e.
like riscv32 will be). The logic will roughly be:

#ifdef SYS_foo_time64
	if (sizeof(time_t)>4) {
		[if there's an input timespec, copy it and zero padding]
		SYS_foo_time64
		[if success, return]
	}
#ifdef SYS_foo [legacy time32]
	[if there's input, explicitly convert to legacy time32 form]
	SYS_foo
	[if success and there's output, convert back to libc time form]
#endif
#else
	SYS_foo [native 64-bit]
#endif

The "if (sizeof(time_t)>4)" condition will become always-false once
32-bit archs are converted over, and can be removed after that. But
for now, it would allow the code to be compile-tested on all archs and
to take effect on x32, with the above-described renaming of syscalls
made in x32/syscall_arch.h.

Granted this is somewhat ugly, but I don't see any other better way;
it seems minimal to meet the (hard) requirements of running on pre-5.1
kernels and using correct time representations.



Now, on to errors. Below is analysis of valid error handling for
conversions that can't be represented.


The easiest are timeouts. Whether the overflowing timeout is absolute
or relative, it can't happen before the non-Y2038-safe system dies in
2038, so the syscall can just be made with no timeout. That covers:

- clock_nanosleep
- mq_timedsend
- mq_timedreceive
- select
- pselect
- ppoll
- recvmmsg
- sigtimedwait
- semtimedop
- __timedwait

In the case of clock_nanosleep (and select, if we want to continue
providing the updated remaining time for select), we can't just use an
unlimited timeout, but instead need to do something like INT32_MAX so
that we can add back the remaining time if the syscall does return.



The following take as input time values that need conversion and that
may not be representable in the legacy type. Thus changing time to
64-bit introduces error cases for them when new syscalls are not
available.

In general, EOVERFLOW is probably not appropriate for any of these.
Its meaning is that the actual value of something is too great to be
represented in the C type used for conveying back the result, not that
the C type used for input can represent values outside the range of
what's supported by the implementation. ENOTSUP is probably the
closest to the latter - "Not supported. The implementation does not
support the requested feature or value."
                              ^^^^^^^^

- clock_settime - obviously this one can't work for times past Y2038.
  there's no standard error (note that EOVERFLOW is specified for
  clock_gettime and matches the above reasoning), so ENOTSUP.

- clock_adjtime - same.

- timer_settime - no related errors defined. setting max possible
  value is tempting but would read back inconsistent results via
  timer_gettime. probably should use ENOTSUP.

- timerfd_settime - not governed by standard but should be the same
  as timer_settime for consistency.

- utimensat - possibly EINVAL (defined by POSIX) or ENOTSUP. for this
  function, EINVAL is defined as:

  [EINVAL]
    A new file timestamp would be a value whose tv_sec component is
    not a value supported by the file system.

  arguably "the file system" includes the kernel's implementation of
  it via syscalls like SYS_utimensat.

- setitimer - no related errors defined. probably need ENOTSUP for
  same reasons as timer_settime.

- setsockopt (SO_RCVTIMEO and SO_SNDTIMEO, using "_NEW" values) - the
  timeouts set need to be readable back through getsockopt, so we need
  an error if falling back to the _OLD versions and the time
  overflows. ENOTSUP again.


The next group is easy. They only produce times as output, and any
legacy 32-bit output necessarily fits in the 64-bit type. Thus, these
have no error cases:

- clock_gettime
- timer_gettime
- timerfd_gettime
- getitimer
- clock_getres
- sched_rr_get_interval
- getrusage
- wait4
- wait3
- getsockopt (SO_RCVTIMEO and SO_SNDTIMEO, using "_NEW" values)
- shmctl
- semctl
- msgctl


There are a few ugly things I didn't cover yet:

setsockopt with SO_TIMESTAMP* either needs to hard-fail if the kernel
doesn't support the "_NEW" values, or some slightly-heavy emulation in
recvmsg is needed (translating cmsgs to 64-bit).

ioctl probably has a number of places where we need to do translations
for fallback ioctl numbers. Hopefully the kernel time64 commits show
us what these are.


That should cover all the interfaces I had on this list so far. I'll
probably try some conversions using the above #ifdef idiom and see how
it goes.

Rich

Re: Final (?) time64 proposal

[Rich Felker]

On Thu, Jul 25, 2019 at 04:01:07PM -0400, Rich Felker wrote:
> On Thu, Jul 25, 2019 at 01:18:32AM -0400, Rich Felker wrote:
> > On Wed, Jul 24, 2019 at 01:31:42AM -0400, Rich Felker wrote:
> > > With that said, the following are the functions I've identified as
> > > still interfacing with the kernel in ways that involve time_t:
> > > 
> > > - clock_gettime **
> > > [...]
> > 
> > I think a good next-step here would be modifying all of the above to
> > go through an intermediate kernel-type struct when using the existing
> > syscalls. This should not change behavior anywhere except for a slight
> > increase in size/time-spent, but will set things up so that, when the
> > time_t definition is switched over, everything should just start
> > working automatically with 64-bit time. It will also let us drop the
> > __fixup hacks in arch/x32/syscall_arch.h and
> > src/thread/x32/syscall_cp_fixup.c, analogous to how commit
> > 01ae3fc6d48f4a45535189b7a6db286535af08ca let us drop the corresponding
> > mips hacks.
> 
> I'm not so sure about this approach now. Conversions will never be
> needed for 64-bit archs, nor for new code paths using the time64
> syscalls on 32-bit archs. Aside from x32 awfulness, the condition for
> the new version of the code to be needed is SYS_*time64 being defined
> and time_t being 64-bit. This suggests we might let x32/syscall_arch.h
> define the SYS_*time64 macros, and undef the legacy names, so that it
> gets treated like a 32-bit arch with no legacy time32 syscalls (i.e.
> like riscv32 will be). The logic will roughly be:
> 
> #ifdef SYS_foo_time64
> 	if (sizeof(time_t)>4) {
> 		[if there's an input timespec, copy it and zero padding]
> 		SYS_foo_time64
> 		[if success, return]
> 	}
> #ifdef SYS_foo [legacy time32]
> 	[if there's input, explicitly convert to legacy time32 form]
> 	SYS_foo
> 	[if success and there's output, convert back to libc time form]
> #endif
> #else
> 	SYS_foo [native 64-bit]
> #endif

I'm working on this for the "timeout" style functions, and it's mostly
working out (in terms of the details making sense, not actually trying
to run code). I've added logic to only try the time64 syscall if a
timeout was provided (in practice often it's null), and then only if
tv_sec doesn't fit in 32 bits, so that performance won't suffer on
existing kernels without time64 syscalls.

> The "if (sizeof(time_t)>4)" condition will become always-false once
> 32-bit archs are converted over, and can be removed after that. But
> for now, it would allow the code to be compile-tested on all archs and
> to take effect on x32, with the above-described renaming of syscalls
> made in x32/syscall_arch.h.
> 
> Granted this is somewhat ugly, but I don't see any other better way;
> it seems minimal to meet the (hard) requirements of running on pre-5.1
> kernels and using correct time representations.

This seems to be working for x32, but one ugly detail is that
SYS_futex is used everywhere and isn't really a "time32" syscall
unless a WAIT/LOCK_PI operation is being used with a timeout. So,
rather than undefining the non-time64 syscall names after mapping
them, I'm just going to define both, and then instead of:

#ifdef SYS_foo_time64
...
#ifdef SYS_foo
...
#endif
...
#endif

we can do:

#ifdef SYS_foo_time64
...
#if defined(SYS_foo) && SYS_foo != SYS_foo_time64
...
#endif
...
#endif

> The easiest are timeouts. Whether the overflowing timeout is absolute
> or relative, it can't happen before the non-Y2038-safe system dies in
> 2038, so the syscall can just be made with no timeout. That covers:
> 
> - clock_nanosleep
> - mq_timedsend
> - mq_timedreceive
> - select
> - pselect
> - ppoll
> - recvmmsg
> - sigtimedwait
> - semtimedop
> - __timedwait

Also add pthread_mutex_timedlock. I thought __timedwait covered it,
but it performs FUTEX_LOCK_PI for PI mutexes, which also has a timeout
argument.

> In the case of clock_nanosleep (and select, if we want to continue
> providing the updated remaining time for select), we can't just use an
> unlimited timeout, but instead need to do something like INT32_MAX so
> that we can add back the remaining time if the syscall does return.

There doesn't seem to be any advantage to using an unlimited timeout
instead of just INT32_MAX timeout when the tv_sec doesn't fit in 32
bits. There's also (at least for absolute timeouts; it's not clear
what's supposed to happen for relative ones with negative tv_sec) the
possibility of times in the distant past, which should ETIMEDOUT
immediately rather than getting truncated or sleeping indefinitely. So
the approach I'm trying now is using a CLAMP() macro on both tv_sec
and tv_nsec when converting to 32-bit. This ensures that large
positive or negative tv_sec will do the right thing, and that large or
negative tv_nsec will map to values that error out rather than
possibly truncating to an in-range value.

Rich

Re: Final (?) time64 proposal

[Rich Felker]

On Wed, Jul 24, 2019 at 01:31:42AM -0400, Rich Felker wrote:
> My plan to go ahead looks like:
> 
> [...]
> - struct shmid_ds, msqid_ds, semid_ds layout kept, extended with
>   64-bit time_t's on the ends

This looks like the unnecessarily painful course of action. I worked
out a grid of all the existing archs' quirks, and except for mips and
mipsn32, all archs admit a solution that's just endian-swapping the
time_t members in-place.

The mips awfulness is isolated to shmid_ds and semid_ds (not affecting
msqid_ds). It's only shmid_ds where 64-bit time_t actually _doesn't
fit_ in the existing structure (the kernel limits high bits to 16-bit
so it can pack them in the limited available unused space). For
semid_ds, the space is there but it just requires moving one of the
two time_t's.

My leaning is to just do the endian-swap as needed (whether it's
needed varies by arch, so syscall_arch.h will need to define it) and
make some mechanism whereby mips can provide its own hideous thing to
do instead. For semid_ds, that's probably going to be:

	otime = sem_otime & 0xffffffff | (sem_ctime & 0xfffffff)<<32;
	ctime = sem_otime>>32 | sem_ctime & 0xffffffff00000000;
	sem_otime = otime;
	sem_ctime = ctime;

and for shmid_ds, it will probably just be adding 3 new time_t's to
the end.

Note that there's some tradeoff here between ugliness of the internal
code in libc to do the transformations, and ugliness of the public
interface (the structure definitions). I'd really like the public
interface to be what's clean and simple. If I do it this way, it can
be, everywhere but mips, and in fact almost all of the arch-specific
sem.h and shm.h variants for 32-bit archs collapse down to being
identical to the variant common to 64-bit archs.

> - new definitions of IPC_STAT, SHM_STAT, SHM_STAT_ANY, SEM_STAT,
>   SEM_STAT_ANY, MSG_STAT, and MSG_STAT_ANY to expand the shmid_ds,
>   semid_ds, and msqid_ds upper and lower time bits from their
>   locations in the kernel struct to the new fields at the end.

I'm still undecided on whether this is better than just wrapping them.
If I go with the above approach for struct layouts, then the generic
compat shims just have to endian-swap the hi/lo 32-bit words of the
time_t fields back -- and copy through a temp object, then memcpy,
since the caller's object may not be aligned suitably for accessing
64-bit types in-place. Then mips and mipsn32 would have to provide a
custom arch-specific version of the wrappers that, for semctl, invert
the above transformation, and for shmctl, just memcpy a truncated
structure (leaving the undersized fields from the kernel in-place).

If I go with the new definitions of IPC_STAT, etc., no compat shims or
symbols redirection are needed for ipc *ctl functions (but note:
semtimedop already has a redirection needed, so sys/sem.h will be
aware of them anyway), and it's easy to do the conversions for 64-bit
time_t only if the new cmd numbers are used; it doesn't even need to
be aware of specific commands, just a special bit for "do conversion".
But it does consume a bit of the command number.

Not sure which is more/less ugly...

Rich

Re: Final (?) time64 proposal

[Rich Felker]

On Mon, Jul 29, 2019 at 05:11:54PM -0400, Rich Felker wrote:
> On Wed, Jul 24, 2019 at 01:31:42AM -0400, Rich Felker wrote:
> > My plan to go ahead looks like:
> > 
> > [...]
> > - struct shmid_ds, msqid_ds, semid_ds layout kept, extended with
> >   64-bit time_t's on the ends
> 
> This looks like the unnecessarily painful course of action. I worked
> out a grid of all the existing archs' quirks, and except for mips and
> mipsn32, all archs admit a solution that's just endian-swapping the
> time_t members in-place.
> 
> The mips awfulness is isolated to shmid_ds and semid_ds (not affecting
> msqid_ds). It's only shmid_ds where 64-bit time_t actually _doesn't
> fit_ in the existing structure (the kernel limits high bits to 16-bit
> so it can pack them in the limited available unused space). For
> semid_ds, the space is there but it just requires moving one of the
> two time_t's.
> 
> My leaning is to just do the endian-swap as needed (whether it's
> needed varies by arch, so syscall_arch.h will need to define it) and
> make some mechanism whereby mips can provide its own hideous thing to
> do instead. For semid_ds, that's probably going to be:
> 
> 	otime = sem_otime & 0xffffffff | (sem_ctime & 0xfffffff)<<32;
> 	ctime = sem_otime>>32 | sem_ctime & 0xffffffff00000000;
> 	sem_otime = otime;
> 	sem_ctime = ctime;
> 
> and for shmid_ds, it will probably just be adding 3 new time_t's to
> the end.
> 
> Note that there's some tradeoff here between ugliness of the internal
> code in libc to do the transformations, and ugliness of the public
> interface (the structure definitions). I'd really like the public
> interface to be what's clean and simple. If I do it this way, it can
> be, everywhere but mips, and in fact almost all of the arch-specific
> sem.h and shm.h variants for 32-bit archs collapse down to being
> identical to the variant common to 64-bit archs.
> 
> > - new definitions of IPC_STAT, SHM_STAT, SHM_STAT_ANY, SEM_STAT,
> >   SEM_STAT_ANY, MSG_STAT, and MSG_STAT_ANY to expand the shmid_ds,
> >   semid_ds, and msqid_ds upper and lower time bits from their
> >   locations in the kernel struct to the new fields at the end.
> 
> I'm still undecided on whether this is better than just wrapping them.
> If I go with the above approach for struct layouts, then the generic
> compat shims just have to endian-swap the hi/lo 32-bit words of the
> time_t fields back -- and copy through a temp object, then memcpy,
> since the caller's object may not be aligned suitably for accessing
> 64-bit types in-place. Then mips and mipsn32 would have to provide a
> custom arch-specific version of the wrappers that, for semctl, invert
> the above transformation, and for shmctl, just memcpy a truncated
> structure (leaving the undersized fields from the kernel in-place).
> 
> If I go with the new definitions of IPC_STAT, etc., no compat shims or
> symbols redirection are needed for ipc *ctl functions (but note:
> semtimedop already has a redirection needed, so sys/sem.h will be
> aware of them anyway), and it's easy to do the conversions for 64-bit
> time_t only if the new cmd numbers are used; it doesn't even need to
> be aware of specific commands, just a special bit for "do conversion".
> But it does consume a bit of the command number.

Lucky find! There's already a bit we can use for this. On 32-bit
archs, there's an "IPC_64" bit that has to be set in cmd when making
the *ctl syscalls (without it you get some ancient broken form of the
result structure), but the bit is not set in the public definitions
for IPC_STAT, etc. Rather both musl and glibc or it into the cmd at
the time of the syscall.

So, we can just redefine IPC_STAT to 0x102, and likewise for the other
affected commands, on 32-bit archs using 64-bit time_t. Then the libc
code can perform the conversion after the syscall returns with success
if and only if the 0x100 bit is set in the caller-passed cmd (which
will be the case whenever caller is using 64-bit time_t headers).

This solution is really clean, and probably eliminates the need to
consider the other with redirects/compat-shims.

Rich

Re: Final (?) time64 proposal

[Rich Felker]

On Mon, Jul 29, 2019 at 05:11:54PM -0400, Rich Felker wrote:
> On Wed, Jul 24, 2019 at 01:31:42AM -0400, Rich Felker wrote:
> > My plan to go ahead looks like:
> > 
> > [...]
> > - struct shmid_ds, msqid_ds, semid_ds layout kept, extended with
> >   64-bit time_t's on the ends
> 
> This looks like the unnecessarily painful course of action. I worked
> out a grid of all the existing archs' quirks, and except for mips and
> mipsn32, all archs admit a solution that's just endian-swapping the
> time_t members in-place.

This turned out to be false, because the alignment is wrong. Almost
all the time_t's would have to start at offsets that are 4 mod 8. So
in-place use is out, and these structs will all get extended at the
end, as originally planned.

Still using new cmd numbers instead of redirecting symbols.

Rich