reproducing release tarball for 5.0.7

reproducing release tarball for 5.0.7

[Phil Pennock]

Folks,

Given a clean repository checkout, what is needed to be able to create
the release tarballs for verification please?

Background:

Since there are no PGP signatures on releases, only an MD5SUM file which
is auto-updated, the process I documented at $workplace for verifying
the integrity of a zsh release tarball has us recreate the release
tarball and diff the results.  (We then generate a SHA256 of the release
tarball, which is propagated in our package system, so future builders
get the same download).  This at least lets us tie the tarball to what's
in git, so that we have a reliable history to back the tarball artifact.

    git checkout zsh-5.0.7
    git clean -fx .
    ./.preconfig
    ( export LC_ALL=C; ./configure && make targz-src )

At this point, I can't tell if this is a regression in zsh or just that
by adding the `git clean -fx` to remove ignored files, to try to
reproduce my first failure, I've exposed that the tarball creation
process has missing dependencies?

First failure:
  ./Util/mkdisttree.sh: can't find file ./Doc/Zsh/manmodmenu.yo

Switch to:

    ( export LC_ALL=C; ./configure && make && make targz-src )

Second failure:
  Processing directory ./Doc/help...
  mkdir /home/pdp/src/zsh/code/zsh-5.0.7/./Doc/help
  chmod 755 /home/pdp/src/zsh/code/zsh-5.0.7/./Doc/help
  ./Util/mkdisttree.sh: can't find file ./Doc/help/ChangeLog

Trying `make targz-doc` first is currently failing because I don't have
TeX on this MacOS box (yet) and ended up removing it a while back on my
FreeBSD box (because I run it nox11 and this just creates problems with
the only currently supported TeX install system).

Thus the question at the top.

Thanks,
-Phil

Re: reproducing release tarball for 5.0.7

[Peter Stephenson]

On Thu, 9 Oct 2014 20:16:29 +0000
Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> Given a clean repository checkout, what is needed to be able to create
> the release tarballs for verification please?

To get all the documentation files you'll need to have got as far as
"make" as well as "make everything" in the Doc directory for the
targz-doc bundle.

pws

Re: reproducing release tarball for 5.0.7

[Mikael Magnusson]

On 9 October 2014 22:52, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Thu, 9 Oct 2014 20:16:29 +0000
> Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
>> Given a clean repository checkout, what is needed to be able to create
>> the release tarballs for verification please?
>
> To get all the documentation files you'll need to have got as far as
> "make" as well as "make everything" in the Doc directory for the
> targz-doc bundle.
>
> pws

I had to copy texinfo.tex in from the automake-1.13 directory manually
for the make everything in Doc/ to complete. After that I ran into
this;

% make targz-doc
Processing directory ./Config...
Processing directory ./Etc...
mkdir -m 755 -p -- [...]/zsh-5.0.7/./Etc
./Util/mkdisttree.sh: can't find file ./Etc/FAQ*.html
make: *** [zsh-5.0.7-doc.tar.gz] Error 1
Makefile:401: recipe for target 'zsh-5.0.7-doc.tar.gz' failed

So I tried
% cd Etc
% make
cd . && yodl2txt FAQ.yo && mv -f FAQ.txt FAQ
FAQ.yo:1: Can't read `txt'
/usr/bin/yodl indicates failure!
make: *** [FAQ] Error 1
Makefile:191: recipe for target 'FAQ' failed

Which is a confusing error message.

-- 
Mikael Magnusson

Re: reproducing release tarball for 5.0.7

[Peter Stephenson]

On Thu, 09 Oct 2014 23:34:20 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> I had to copy texinfo.tex in from the automake-1.13 directory manually
> for the make everything in Doc/ to complete. After that I ran into
> this;
> 
> % make targz-doc
> Processing directory ./Config...
> Processing directory ./Etc...
> mkdir -m 755 -p -- [...]/zsh-5.0.7/./Etc
> ./Util/mkdisttree.sh: can't find file ./Etc/FAQ*.html
> make: *** [zsh-5.0.7-doc.tar.gz] Error 1
> Makefile:401: recipe for target 'zsh-5.0.7-doc.tar.gz' failed
> 
> So I tried
> % cd Etc
> % make
> cd . && yodl2txt FAQ.yo && mv -f FAQ.txt FAQ
> FAQ.yo:1: Can't read `txt'
> /usr/bin/yodl indicates failure!
> make: *** [FAQ] Error 1
> Makefile:191: recipe for target 'FAQ' failed
> 
> Which is a confusing error message.

You need a texinfo installation to make the docs, which should be
available as a package.

I've run across the missings dependency for making the FAQ before, but
if Yodl is set up actually making it should be unproblematic.  There may
be some version mixing or installation problem --- I can see the same
problem here, but it works at home.  We don't depend on the normal Yodl
converters in the main documentation which is why this is hard to
provoke.  It seems to be a path problem --- it's not looking in
/usr/share/yodl, but if I copy those files locally it works.

pws

-- 
Peter Stephenson <p.stephenson@samsung.com>  Principal Software Engineer
Tel: +44 (0)1223 434724                Samsung Cambridge Solution Centre
St John's House, St John's Innovation Park, Cowley Road,
Cambridge, CB4 0DS, UK

Re: reproducing release tarball for 5.0.7

[Peter Stephenson]

On Thu, 09 Oct 2014 23:34:20 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> So I tried
> % cd Etc
> % make
> cd . && yodl2txt FAQ.yo && mv -f FAQ.txt FAQ
> FAQ.yo:1: Can't read `txt'
> /usr/bin/yodl indicates failure!
> make: *** [FAQ] Error 1
> Makefile:191: recipe for target 'FAQ' failed
> 
> Which is a confusing error message.

This works around it if the Yodl installation is fairly standard, though
it's not clear to me it's our job to do so.

Did I miss some much simpler standard Bourne shell way of finding what
directory a command is in?

diff --git a/Etc/Makefile.in b/Etc/Makefile.in
index b6743a7..fed35c7 100644
--- a/Etc/Makefile.in
+++ b/Etc/Makefile.in
@@ -42,11 +42,42 @@ INSTALL         = @INSTALL@
 
 all: FAQ
 
+# Some Yodl installations don't have correct paths to include
+# the standard macros, so find them.
+
 FAQ: FAQ.yo
-	cd $(sdir) && $(YODL2TXT) FAQ.yo && mv -f FAQ.txt FAQ
+	pathsplit=`echo $$PATH | sed 's%:% %g'`; \
+	include=`for p in $$pathsplit; do \
+	  if [ -e $$p/$(YODL2TXT) ]; then \
+	    echo $$p | sed -e 's%/bin$$%%'; \
+	    break; \
+	  fi; \
+	done`; \
+	if [ x$$include != x ]; then \
+	  if [ -d $$include/share/yodl ]; then \
+	    include="-I$$include/share/yodl"; \
+	  else \
+	    include=''; \
+	  fi; \
+	fi; \
+	cd $(sdir) && $(YODL2TXT) $$include FAQ.yo && mv -f FAQ.txt FAQ
 
 FAQ.html: FAQ.yo
-	$(YODL2HTML) $(sdir)/FAQ.yo
+	pathsplit=`echo $$PATH | sed 's%:% %g'`; \
+	include=`for p in $$pathsplit; do \
+	  if [ -e $$p/$(YODL2HTML) ]; then \
+	    echo $$p | sed -e 's%/bin$$%%'; \
+	    break; \
+	  fi; \
+	done`; \
+	if [ x$$include != x ]; then \
+	  if [ -d $$include/share/yodl ]; then \
+	    include="-I$$include/share/yodl"; \
+	  else \
+	    include=''; \
+	  fi; \
+	fi; \
+	$(YODL2HTML) $$include $(sdir)/FAQ.yo
 
 # ========== DEPENDENCIES FOR CLEANUP ==========
 

pws

-- 
Peter Stephenson <p.stephenson@samsung.com>  Principal Software Engineer
Tel: +44 (0)1223 434724                Samsung Cambridge Solution Centre
St John's House, St John's Innovation Park, Cowley Road,
Cambridge, CB4 0DS, UK

Re: reproducing release tarball for 5.0.7

[Phil Pennock]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2014-10-09 at 21:52 +0100, Peter Stephenson wrote:
> On Thu, 9 Oct 2014 20:16:29 +0000
> Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> > Given a clean repository checkout, what is needed to be able to create
> > the release tarballs for verification please?
> 
> To get all the documentation files you'll need to have got as far as
> "make" as well as "make everything" in the Doc directory for the
> targz-doc bundle.

Forgive me my lack of clarity: I only care about `targz-src`.  When I
wrote "release tarballs" I didn't mean all of them, just the source
ones.  Sloppy of me.  The Doc attempt was just to see if this would
create Doc/help/ChangeLog and unwedge the src build.

I think that `Doc/help/ChangeLog` is appearing because the help dir is
empty and this ChangeLog bit is an artifact of mkdisttree.sh; the
directory is empty because of help.txt and zshbuiltins.yo being empty
and it looks like the build process is just not erroring out on a
missing yodl binary.  I missed yodl when installing my new work laptop.

Much head scratching caused by not being able to download tarballs from
sourceforge with a browser signed into my SF account; works fine in a
different profile which isn't.

icmake building requires GNU sed, the character classes aren't supported
by MacOS sed.  Attempts to install to a place other than directly under
/ reveal how buggy this install system is.  yodl also requires GNU sed,
plus hand-holding as it used a bad relative path to find xlatin1.tex and
then removing .dSYM/ dirs from bin because its build system really
doesn't handle MacOS well.


Having gone this far ...  for the documentation, roff2ps is not supplied
by Apple, so installing a full groff package is required; this then
fails on ps generation:
  set mode: no suitable display mode found under ps; at /usr/local/bin/groffer line 1230.
and unfortunately roff2ps swallows the exit status of groffer, so the
exit status of 2 does not make it back and our only indication of
failure is that the .ps files are of length 0.

After installing XQuartz, running it and running `export DISPLAY=:0`
the PS files build.  Using GNU sed is also required for the zsh docs
build; this gets me through to HTML files being missing; the only errors
left are TeX errors around hbox sizing.

- ----------------------------8< cut here >8------------------------------
Output written on zsh.pdf (430 pages, 1698895 bytes).
Transcript written on zsh.log.
roff2ps -ms -P-pa4 < ./intro.ms > intro.a4.ps
ps2pdf intro.a4.ps
roff2ps -ms -P-pletter < ./intro.ms > intro.us.ps
ps2pdf intro.us.ps
mkdir /Users/pdp/src/zsh/code/zsh-5.0.7
chmod 755 /Users/pdp/src/zsh/code/zsh-5.0.7
mkdir /Users/pdp/src/zsh/code/zsh-5.0.7/./Doc
chmod 755 /Users/pdp/src/zsh/code/zsh-5.0.7/./Doc
./Util/mkdisttree.sh: can't find file ./Doc/*.html
make: *** [zsh-5.0.7-doc.tar.gz] Error 1
- ----------------------------8< cut here >8------------------------------


Results of the build of the src tarball?

Yodl is version 3.03.0; autoconf is 2.69 (release used 2.68).

% diff -Bbwur zsh-5.0.7 pdp-zsh-507 |less

My zsh.texi gains a `@noindent` line; a couple of extra `.PP` lines in
man-pages; email addresses are being followed by a dup inside angle
brackets, which sometimes works okay (albeit redundant) and other times
demonstrates a bit of confusion somewhere in the macros around display
names?

- -Submitted-By: coordinator@zsh.org (Peter Stephenson)
+Submitted-By: coordinator@zsh.org (Peter Stephenson) <coordinator@zsh.org (Peter Stephenson)>

Numbering in the FAQ is borked:

- -  1) because of all the options it is not safe to assume a general
+  1. ) because of all the options it is not safe to assume a general

Some changes due to autoconf.  I have not thoroughly audited the shell
differences, but from a skim it looked safe.  Of course, this just means
that autoconf-generated files are ideal places to bury any malicious
changes; however, none of the differences looked like they'd change code
making it into a deployed zsh binary.

So the tarballs I retrieved look like they can be reproducibly tied to
the git tree and have not been tampered with.  Here are the checksums,
I'm PGP-signing this message, my key is in the strong set.

SHA256(zsh-5.0.7.tar.bz2)= 544e27de81740286b916d1d77c9f48ad7c26ad7943ed96d278abee67cf6704b3
SHA1(zsh-5.0.7.tar.bz2)= 1500191d16af8a71aec4f719a92775a074682096

- -Phil
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJUN3rQAAoJEE0ekA4UwcwEtG0QAJOfy7WeIm4Fp4NG420WSiqI
9XtDXf4/NBBvFgK5FBmcoq7BAnl7/NXaLojIQy2DrhCJYNWJ48z6/SK5aCyOQC3/
hmvFB4PwY6wDA9S9SDT/2sY90uhT0tLRdQhMn+uJXFOpA0U6QvT7iFp2OO31bXYy
5yjAS6aonq3FA6D99xrqhXUWtHH0jEQ6TrdV1Z4X2JPbPgVLaoJ/5ToIhV/Fl57z
IsqYg4IHEAsZlMNU4arzB96D1Wn71I2QJQ8fUsoAnGUnhWqwQ353mSpIv9+RAiWa
mxSaTZz0zuMxJNgeNXQOSmVcPw5jchIuh66J5wCmYCMiUCDbyMWlzZPNQ5Q091Qi
lhk64wHSTdVQ0aJ0+litIA2VYkYNqQ5wJTisXVLWlaVs/dZocM6L1X3J66a6z+hM
i4wtdBu6e2c2TZHDV5IOlkCSFRVvCzP6AUx5O+ciTUeMCL0n4fXgZdL5N8tigJ3C
PH8XmYW5iF+wcPv/R6+aRucPBL66FnIHVXrcgBgOvUOzQQROdJv0DzJPnwlKvEdr
p6vNn9vu2bXJFppY0FnazIaEfuzud70iJt631KaUZtcASnvsDxV1W+EbooKmynx2
QyzxLDNhnjmEWF8HcCi/dy/dPy30oEPXXKey66wlTVlMk3UyhFQk1G2DDYIsU6TM
BGNYZg0R3QDyWE51jTBE
=ckX+
-----END PGP SIGNATURE-----

Re: reproducing release tarball for 5.0.7

[Simon Ruderich]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

On Thu, Oct 09, 2014 at 08:16:29PM +0000, Phil Pennock wrote:
> Folks,
>
> Given a clean repository checkout, what is needed to be able to create
> the release tarballs for verification please?

Hello Peter,

How do you feel about providing GPG signatures for the tarballs
and the git tags? This would fix this issue and make it possible
for everybody to verify zsh's releases. For example Debian has
tools to automatically verify the upstream tarball after the
download if upstream provides signatures. This allows maintainers
to be sure they downloaded the correct tarball.

If you like I could prepare a patch for the Makefile to sign the
resulting tarballs, so a "make sign" is the only required action.
For Git it's even easier, instead of git tag $tag, you can just
use git tag -s -m 'optional message' $tag and it will be signed.
I'm already using signed tags for the website.

Regards
Simon
-- 
+ privacy is necessary
+ using gnupg http://gnupg.org
+ public key id: 0x92FEFDB7E44C32F9

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: reproducing release tarball for 5.0.7

[Peter Stephenson]

On Sat, 11 Oct 2014 02:19:08 +0200
Simon Ruderich <simon@ruderich.org> wrote:
> How do you feel about providing GPG signatures for the tarballs
> and the git tags? This would fix this issue and make it possible
> for everybody to verify zsh's releases. For example Debian has
> tools to automatically verify the upstream tarball after the
> download if upstream provides signatures. This allows maintainers
> to be sure they downloaded the correct tarball.
> 
> If you like I could prepare a patch for the Makefile to sign the
> resulting tarballs, so a "make sign" is the only required action.
> For Git it's even easier, instead of git tag $tag, you can just
> use git tag -s -m 'optional message' $tag and it will be signed.
> I'm already using signed tags for the website.

Could do, guess we need a new key for this.

pws

Re: reproducing release tarball for 5.0.7

[Phil Pennock]

On 2014-10-12 at 18:00 +0100, Peter Stephenson wrote:
[ PGP keys for zsh ]
>
> Could do, guess we need a new key for this.

Not really: role keys only make sense if there's a bunch of process and
control around their access and some people who can validate the key who
sign it to provide trust paths to the outside world.  For open source
projects, IMO it makes more sense to just have individual maintainers
use their own keys.

I wrote this, as part of Exim's release process documentation:
  https://github.com/Exim/exim/wiki/EximReleasePolicy#release-verification
and think it's a reasonable baseline for zsh too.  Probably drop the
$project.org UID bit, since @zsh.org email addresses aren't really used
by individuals.

-Phil