Portability of zwc files between zsh versions

Portability of zwc files between zsh versions

[Glenn Morris]

Hi,

If I use zcompile to create a digest with zsh 5.5, I find I cannot load
the resulting file with zsh 5.8 (or vice versa).

'zcompile -t' prints: "zcompile: zwc file has wrong version"

It seems from the source:
https://github.com/zsh-users/zsh/blob/master/Src/parse.c#L3250
that in order to use a zwc file, the version of zsh it was created with
must match exactly the version of zsh trying to load it?

I was expecting zwc files to be more portable than this, given the
comments in the manual:

     The created file always contains two versions of the compiled
     format, one for big-endian machines and one for small-endian
     machines. The upshot of this is that the compiled file is machine
     independent [...]

The comments in Src/parse.c make me think that the intent was that zwc
files should be loadable so long as FD_MAGIC|FD_OMAGIC haven't changed -
they last changed in 52b830353 (2001).
Ref also https://www.zsh.org/mla/workers/2000/msg01327.html

It seems that the ZSH_VERSION check was added back in d296535d3 (2007),
"23313: fix test for incorrect version in ZWC file".

I noticed this issue after zcompiling on a RHEL9 system (zsh 5.8) where
HOME is also accessible to RHEL8 (zsh 5.5) hosts. It's a bit of a
nuisance to have to maintain a separate zwc file for each zsh version.

Is it intentional that zwc files aren't portable between zsh versions?
If so, it would seem like the endian duplication wouldn't be that useful
in practice?

Thanks.

Re: Portability of zwc files between zsh versions

[Bart Schaefer]

On Sun, Aug 14, 2022 at 11:19 AM Glenn Morris <rgm@stanford.edu> wrote:
>
> The comments in Src/parse.c make me think that the intent was that zwc
> files should be loadable so long as FD_MAGIC|FD_OMAGIC haven't changed

In fact the files are not loadable in the event that any of the
wordcode definitions in parse.c have changed, such as the WC_FUNCDEF
edit in March 2020, among other possible conditions.

Possibly the right thing to do would be to remove the version check
and instead be more diligent about updating the *MAGIC values, I
suspect that the habit of doing the latter was forgotten when the
version check subsumed it.

> It's a bit of a
> nuisance to have to maintain a separate zwc file for each zsh version.

compinit already maintains separate compdumps of autoloads for each
version, so the expectation that the function bodies might need to be
re-zcompiled isn't especially outlandish?

> Is it intentional that zwc files aren't portable between zsh versions?
> If so, it would seem like the endian duplication wouldn't be that useful
> in practice?

The endian duplication is in case you have equivalent versions of zsh
on different machine architectures.  That might not be as common as it
once was.

There have also been some significant improvements in file parsing
speed, so zcompile might not be as advantageous as in the past.

Re: Portability of zwc files between zsh versions

[Peter Stephenson]

On Sun, 2022-08-14 at 12:38 -0700, Bart Schaefer wrote:
> Possibly the right thing to do would be to remove the version check
> and instead be more diligent about updating the *MAGIC values, I
> suspect that the habit of doing the latter was forgotten when the
> version check subsumed it.

Indeed, I think the shell version check was inspired by the knowledge
that we were already being less than diligent in updating the dumped
wordcode version number.

Maybe a bit of creative thinking about how we detect relevant changes is
needed.  The only real requirement is picking up incompatibilities when
we make a release.

pws

Re: Portability of zwc files between zsh versions

[Bart Schaefer]

On Sun, Aug 14, 2022 at 1:16 PM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> Maybe a bit of creative thinking about how we detect relevant changes is
> needed.  The only real requirement is picking up incompatibilities when
> we make a release.

We could drop a zcompile'd file in the Test directory and add a check
that it loads+runs correctly.  It might be a bit tricky to construct
an example that uses all possible syntax tokens, and there's a
question of how to rebuild it such that we know it wasn't just
forcibly recompiled using the current build without updating the magic
numbers.