[RFC/PATCH] t0107: support older tar in zstd untar test

[RFC/PATCH] t0107: support older tar in zstd untar test

[Todd Zullinger]

The tar --zstd option was added in GNU tar-1.32 (2019-02-23)¹.  Avoid the
requirement on a (relatively) recent version of tar by piping the output
of `zstd -dcf` to tar.  As a real-world example, I ran into this test
failure while building for CentOS/RHEL 8, which ships with tar-1.30.

It may also be a slight benefit that this more closely matches what the
snapshot creation code does, so arguably we could adjust the other tests
to match.

Alternately, we could add PREREQ's for the version of tar which adds the
corresponding compression algorithm shortcut², but that seems like a bit
of work for very little gain.  And it differs (if only ever so slightly)
from the pipe method we use to create the tar archives being tested.

¹ Technically, it was first released in tar-1.31 (2019-01-02), but this
  release was very short-lived and is no longer listed on the GNU Tar
  release page.

² The compression shortcuts for the 3 algorithms we have behind PREREQ's
  are --xz, --lzip, and --zstd.  For reference, these were added to GNU
  Tar as follows:

  c10830a (Add xz support., 2009-03-04), tar-1.22 (2009-03-05)
  bffe107 (Add Lzip support, 2010-03-02), tar-1.23 (2010-03-10)
  3d45373 (Add support for zstd compression, 2018-03-18), tar-1.32 (2019-02-23)

Signed-off-by: Todd Zullinger <tmz@pobox.com>
---

Hi all,

As I mentioned in the commit message, I ran into this test failure when
I tried building for CentOS/RHEL 8.  While this solves that particular
issue, it did raise a few questions which I thought were worth answering
before applying this patch as-is.

Does cgit intend require GNU tar?  If not, perhaps we should instead
justify this primarily for compatibility with a wider range of tar
implementations in addition to supporting older versions of GNU tar?  (I
haven't looked at or tested on systems which use a non-GNU tar, so there
are posibly other issues with our tar invocations apart from these
compression shortcut options.  Finding and fixing that can be a
#lefoverbit for anyone more familiar with such a system(s).)

I'd be happy to make such an adjustment to the other tests and commit
message(s).  I think this would all fit nicely in a single commit, but I
can split it in one for each of the three current compression algorithms
which use the --$algo shortcuts if that's preferred.

If we wanted to convert all the tests to use a decompress pipe into tar,
I could do that instead.  The latter would likely be less to ensure
support for different tar implemtations and more for consistency in the
test suite.  Whether that's worth the churn is debatable. :)

Thanks,

 tests/t0107-snapshot.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/t0107-snapshot.sh b/tests/t0107-snapshot.sh
index c164d3e..5b5d727 100755
--- a/tests/t0107-snapshot.sh
+++ b/tests/t0107-snapshot.sh
@@ -151,7 +151,7 @@ test_expect_success ZSTD 'verify zstd format' '
 
 test_expect_success ZSTD 'untar' '
 	rm -rf master &&
-	tar --zstd -xf master.tar.zst
+	zstd -dcf master.tar.zst | tar -xf -
 '
 
 test_expect_success ZSTD 'count files' '

-- 
Todd

Re: [RFC/PATCH] t0107: support older tar in zstd untar test

[Todd Zullinger]

Yesterday, I wrote:
> Does cgit intend to require GNU tar?  If not, perhaps we should instead
> justify this primarily for compatibility with a wider range of tar
> implementations in addition to supporting older versions of GNU tar?

The more I thought about it, the more I liked that idea.  It
makes it simpler for new compression algorithms to re-use
the tests for the existing algorithms.  We could potentially
even create a helper function to handle each compression algo.

So here's that version in patch form.

-- 8< --
Subject: [PATCH] t0107: support older and/or non-GNU tar

The untar tests for various compression algorithms use shortcut options
from GNU tar to handle decompression.  These options may not be provided
by non-GNU tar nor even by slightly older GNU tar versions which ship on
many systems. 

An example of the latter case is the --zstd option.  This was added in
GNU tar-1.32 (2019-02-23)¹.  This version of tar is not provided by
CentOS/RHEL, in particular.  In Debian, --zstd has been backported to
the tar-1.30 release.

Avoid the requirement on any specific implementations or versions of tar
by piping decompressed output to tar.  This is compatible with older GNU
tar releases as well as tar implementations from other vendors.  (It may
also be a slight benefit that this more closely matches what the
snapshot creation code does.)

¹ Technically, the --zstd option was first released in tar-1.31
  (2019-01-02), but this release was very short-lived and is no longer
  listed on the GNU Tar release page.

Signed-off-by: Todd Zullinger <tmz@pobox.com>
---
 tests/t0107-snapshot.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/t0107-snapshot.sh b/tests/t0107-snapshot.sh
index c164d3e..8ea95ce 100755
--- a/tests/t0107-snapshot.sh
+++ b/tests/t0107-snapshot.sh
@@ -25,7 +25,7 @@ test_expect_success 'verify gzip format' '
 
 test_expect_success 'untar' '
 	rm -rf master &&
-	tar -xzf master.tar.gz
+	gzip -dc master.tar.gz | tar -xf -
 '
 
 test_expect_success 'count files' '
@@ -67,7 +67,7 @@ test_expect_success LZIP 'verify lzip format' '
 
 test_expect_success LZIP 'untar' '
 	rm -rf master &&
-	tar --lzip -xf master.tar.lz
+	lzip -dc master.tar.lz | tar -xf -
 '
 
 test_expect_success LZIP 'count files' '
@@ -109,7 +109,7 @@ test_expect_success XZ 'verify xz format' '
 
 test_expect_success XZ 'untar' '
 	rm -rf master &&
-	tar --xz -xf master.tar.xz
+	xz -dc master.tar.xz | tar -xf -
 '
 
 test_expect_success XZ 'count files' '
@@ -151,7 +151,7 @@ test_expect_success ZSTD 'verify zstd format' '
 
 test_expect_success ZSTD 'untar' '
 	rm -rf master &&
-	tar --zstd -xf master.tar.zst
+	zstd -dc master.tar.zst | tar -xf -
 '
 
 test_expect_success ZSTD 'count files' '

-- 
Todd

[PATCH] t0107: support older and/or non-GNU tar

[Todd Zullinger]

The untar tests for various compression algorithms use shortcut options
from GNU tar to handle decompression.  These options may not be provided
by non-GNU tar nor even by slightly older GNU tar versions which ship on
many systems.

An example of the latter case is the --zstd option.  This was added in
GNU tar-1.32 (2019-02-23)¹.  This version of tar is not provided by
CentOS/RHEL, in particular.  In Debian, --zstd has been backported to
the tar-1.30 release.

Avoid the requirement on any specific implementations or versions of tar
by piping decompressed output to tar.  This is compatible with older GNU
tar releases as well as tar implementations from other vendors.  (It may
also be a slight benefit that this more closely matches what the
snapshot creation code does.)

¹ Technically, the --zstd option was first released in tar-1.31
  (2019-01-02), but this release was very short-lived and is no longer
  listed on the GNU Tar release page.

Signed-off-by: Todd Zullinger <tmz@pobox.com>
---

Hi,

I'm sending again, in case the previous version which was in
reply to the RFC/PATCH was simply not seen.

It would be nice to get a fix to allow running the tests on
systems which don't have tar >= 1.32 (or Debian-based syttems
where the --zstd option was backported to 1.30).  I'd also
prefer to not carry it as a patch in the packages for Fedora's
EPEL branches if I can avoid it.

Thanks!

 tests/t0107-snapshot.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/t0107-snapshot.sh b/tests/t0107-snapshot.sh
index c164d3e..8ea95ce 100755
--- a/tests/t0107-snapshot.sh
+++ b/tests/t0107-snapshot.sh
@@ -25,7 +25,7 @@ test_expect_success 'verify gzip format' '
 
 test_expect_success 'untar' '
 	rm -rf master &&
-	tar -xzf master.tar.gz
+	gzip -dc master.tar.gz | tar -xf -
 '
 
 test_expect_success 'count files' '
@@ -67,7 +67,7 @@ test_expect_success LZIP 'verify lzip format' '
 
 test_expect_success LZIP 'untar' '
 	rm -rf master &&
-	tar --lzip -xf master.tar.lz
+	lzip -dc master.tar.lz | tar -xf -
 '
 
 test_expect_success LZIP 'count files' '
@@ -109,7 +109,7 @@ test_expect_success XZ 'verify xz format' '
 
 test_expect_success XZ 'untar' '
 	rm -rf master &&
-	tar --xz -xf master.tar.xz
+	xz -dc master.tar.xz | tar -xf -
 '
 
 test_expect_success XZ 'count files' '
@@ -151,7 +151,7 @@ test_expect_success ZSTD 'verify zstd format' '
 
 test_expect_success ZSTD 'untar' '
 	rm -rf master &&
-	tar --zstd -xf master.tar.zst
+	zstd -dc master.tar.zst | tar -xf -
 '
 
 test_expect_success ZSTD 'count files' '

Re: [PATCH] t0107: support older and/or non-GNU tar

[Jason A. Donenfeld]

Applied, thanks.