Cannot receive older ZFS dumps

Cannot receive older ZFS dumps

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 1652 bytes --]

Hi,

it seems that older zfs images (from zfs send) cannot be imported
into recent illumos versions. These are dumps from a "pre-features"
zpool. I get

ro suntc ~ # cat 32@20100825 | zfs receive -d rpool
cannot receive new filesystem stream: invalid backup stream

while I can still import those on an older illumos (in oi151.a8).

This is what zstreamdump sees:

# cat 32@20100825|zstreamdump
BEGIN record
         hdrtype = 1
         features = 0
         magic = 2f5bacbac
         creation_time = 4c751a60
         type = 2
         flags = 0x0
         toguid = d5549c3062863864
         fromguid = 0
         toname = Processor/Results/imk/32@20100825
END checksum = ac3419bb76a98f7/9a5ac336e407bdf3/c07a3f469bc7e40e/f03a369ef26634ee
SUMMARY:
         Total DRR_BEGIN records = 1
         Total DRR_END records = 1
         Total DRR_OBJECT records = 131771
         Total DRR_FREEOBJECTS records = 716
         Total DRR_WRITE records = 133062
         Total DRR_WRITE_BYREF records = 0
         Total DRR_WRITE_EMBEDDED records = 0
         Total DRR_FREE records = 300918
         Total DRR_SPILL records = 0
         Total records = 566469
         Total write size = 1690867200 (0x64c89600)
         Total stream length = 1902392632 (0x71643538)

Is this a bug or are "non-feature" dumps in general not importable
on "feature"-zpools ?
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Toomas Soome]

> On 1. Aug 2024, at 13:38, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
> 
> Hi,
> 
> it seems that older zfs images (from zfs send) cannot be imported
> into recent illumos versions. These are dumps from a "pre-features"
> zpool. I get
> 
> ro suntc ~ # cat 32@20100825 | zfs receive -d rpool
> cannot receive new filesystem stream: invalid backup stream


Since we virtually do not develop zfs, it is rather hard to say, but it does sound like bug. However, if you can receive it to pre-feature pool, I would upgrade pre-features pool and then check the send+receive again (from upgraded pool).

rgds,
toomas


> 
> while I can still import those on an older illumos (in oi151.a8).
> 
> This is what zstreamdump sees:
> 
> # cat 32@20100825|zstreamdump
> BEGIN record
>        hdrtype = 1
>        features = 0
>        magic = 2f5bacbac
>        creation_time = 4c751a60
>        type = 2
>        flags = 0x0
>        toguid = d5549c3062863864
>        fromguid = 0
>        toname = Processor/Results/imk/32@20100825
> END checksum = ac3419bb76a98f7/9a5ac336e407bdf3/c07a3f469bc7e40e/f03a369ef26634ee
> SUMMARY:
>        Total DRR_BEGIN records = 1
>        Total DRR_END records = 1
>        Total DRR_OBJECT records = 131771
>        Total DRR_FREEOBJECTS records = 716
>        Total DRR_WRITE records = 133062
>        Total DRR_WRITE_BYREF records = 0
>        Total DRR_WRITE_EMBEDDED records = 0
>        Total DRR_FREE records = 300918
>        Total DRR_SPILL records = 0
>        Total records = 566469
>        Total write size = 1690867200 (0x64c89600)
>        Total stream length = 1902392632 (0x71643538)
> 
> Is this a bug or are "non-feature" dumps in general not importable
> on "feature"-zpools ?
> -- 
> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
> https://www.imk-asf.kit.edu/english/sat.php
> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
> 
> 
> ------------------------------------------
> illumos: illumos-discuss
> Permalink: https://illumos.topicbox.com/groups/discuss/T575f82eecd117456-Mc6ac5b0fc6787f44ca997f22
> Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

Re: [discuss] Cannot receive older ZFS dumps

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 4280 bytes --]

On 01/08/2024 12:53, Toomas Soome via illumos-discuss wrote:
>
>
>> On 1. Aug 2024, at 13:38, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>>
>> Hi,
>>
>> it seems that older zfs images (from zfs send) cannot be imported
>> into recent illumos versions. These are dumps from a "pre-features"
>> zpool. I get
>>
>> ro suntc ~ # cat 32@20100825 | zfs receive -d rpool
>> cannot receive new filesystem stream: invalid backup stream
>
>
> Since we virtually do not develop zfs, it is rather hard to say, but it does sound like bug. However, if you can receive it to pre-feature pool, I would upgrade pre-features pool and then check the send+receive again (from upgraded pool).
>

Did some more testing, could also import it on an 8 years old pool
(illumos-a90d75b) with all the features (known at that time) active or enabled:

   async_destroy
   empty_bpobj
   lz4_compress
   multi_vdev_crash_dump
   spacemap_histogram
   enabled_txg
   hole_birth
   extensible_dataset
   embedded_data
   bookmarks
   filesystem_limits
   large_blocks
   sha512
   skein
   edonr

Unknown at that time were

   large_dnode
   device_removal
   obsolete_counts
   zpool_checkpoint
   spacemap_v2
   allocation_classes
   resilver_defer
   encryption
   bookmark_v2
   userobj_accounting
   project_quota
   log_spacemap

Maybe one of that is to blame for the incompatibility, or it's a completely
different bug. The error appears near the end of the zfs receive.

As I still have the original filesystems, and the old ones are not that large,
I will recreate the dumps.

But in principle, zfs should preserve the ability to import older versions as
much as possible, or at least have some up to date information what versions
are supported for a particular new version. And it should then fail with a
version mismatch (if it is such) as the error message.

By the way, for upgrading older OS versions I tested importing new zfs
dumps into an old zpool (without actually booting into the old OS, but for
booting into the new one and then upgrading the old zpool), but that doesn't
work either (probably expectable), the zfs receive hangs itself into an
uninterruptible state after a while. But I won't complain for that one, for
obvious reasons ...

>> while I can still import those on an older illumos (in oi151.a8).
>>
>> This is what zstreamdump sees:
>>
>> # cat 32@20100825|zstreamdump
>> BEGIN record
>>        hdrtype = 1
>>        features = 0
>>        magic = 2f5bacbac
>>        creation_time = 4c751a60
>>        type = 2
>>        flags = 0x0
>>        toguid = d5549c3062863864
>>        fromguid = 0
>>        toname = Processor/Results/imk/32@20100825
>> END checksum = ac3419bb76a98f7/9a5ac336e407bdf3/c07a3f469bc7e40e/f03a369ef26634ee
>> SUMMARY:
>>        Total DRR_BEGIN records = 1
>>        Total DRR_END records = 1
>>        Total DRR_OBJECT records = 131771
>>        Total DRR_FREEOBJECTS records = 716
>>        Total DRR_WRITE records = 133062
>>        Total DRR_WRITE_BYREF records = 0
>>        Total DRR_WRITE_EMBEDDED records = 0
>>        Total DRR_FREE records = 300918
>>        Total DRR_SPILL records = 0
>>        Total records = 566469
>>        Total write size = 1690867200 (0x64c89600)
>>        Total stream length = 1902392632 (0x71643538)
>>
>> Is this a bug or are "non-feature" dumps in general not importable
>> on "feature"-zpools ?
>> --
>> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
>> https://www.imk-asf.kit.edu/english/sat.php
>> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
>> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
>>
>
> ------------------------------------------
> illumos: illumos-discuss
> Permalink: https://illumos.topicbox.com/groups/discuss/T575f82eecd117456-M86e91c6b85ba15613f8f79c2
> Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
>


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Joshua M. Clulow]

On Thu, 1 Aug 2024 at 05:27, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
> On 01/08/2024 12:53, Toomas Soome via illumos-discuss wrote:
> > Since we virtually do not develop zfs, it is rather hard to say, but it does sound like bug.

I don't know what this means, but I would like to make it very clear:
we, the illumos project, maintain all of the code in the gate.  This
includes ZFS.  If you've found something that used to work and doesn't
work anymore, that's definitely a bug -- please report it, with as
much diagnostic and debugging information as you can gather.

> >> Is this a bug or are "non-feature" dumps in general not importable
> >> on "feature"-zpools ?

I don't think any regression like that is intentional.  If we make an
intentional decision like that, the tool should print a crisp error
that the operator can understand which describes the situation -- so
even if it isn't intended to work (definitely not clear that this is
the case), the absence of a crisp failure and a helpful error message
would be, on its own, a bug.

> Did some more testing, could also import it on an 8 years old pool
> (illumos-a90d75b) with all the features (known at that time) active or enabled:
> Maybe one of that is to blame for the incompatibility, or it's a completely
> different bug. The error appears near the end of the zfs receive.

I think you'll probably need to use DTrace to find out where the error
is originating.  Presumably something in the kernel is validating the
incoming stream records in a new way, which might well be too strict
or just wrong in some way.


Cheers.

-- 
Joshua M. Clulow
http://blog.sysmgr.org

Re: [discuss] Cannot receive older ZFS dumps

[John D Groenveld]

In message <1c2abfa3-df73-91cb-79da-c34eaeca3af7@kit.edu>, "Udo Grabowski (IMK)" writes:
>Did some more testing, could also import it on an 8 years old pool
>(illumos-a90d75b) with all the features (known at that time) active or enab=
>led:

I'm not able to reproduce.
Here's my test procedure:
Booted 151a8
Created two pools:
# mkfile 64M /root/features
# mkfile 64M /root/version28
# zpool create features /root/features
# zpool create -d -o version=28 version28 /root/version28
# zfs create features/test
# zfs create version28/test
# dd if=/dev/urandom of=/features/test/wee bs=1024k count=16
# md5sum /features/test/wee
# cp /features/test/wee /version28/test/
# md5sum /version28/test/wee
# zfs snapshot features/test@foo
# zfs snapshot version28/test@bar
# zfs send features/test@foo>features.snapshot
# zfs send version28/test@bar>version28.snapshot

copy the snapshots to today's OI, illumos-533d257436

# zfs receive -v rpool/features< features.snapshot
# zfs receive -v rpool/version28< version28.snapshot
# md5sum rpool/features/test/wee
# md5sum rpool/version28/test/wee

John
groenveld@acm.org

Re: [discuss] Cannot receive older ZFS dumps

[Toomas Soome]

> On 2. Aug 2024, at 20:09, John D Groenveld via illumos-discuss <discuss@lists.illumos.org> wrote:
> 
> In message <1c2abfa3-df73-91cb-79da-c34eaeca3af7@kit.edu>, "Udo Grabowski (IMK)" writes:
>> Did some more testing, could also import it on an 8 years old pool
>> (illumos-a90d75b) with all the features (known at that time) active or enab=
>> led:
> 
> I'm not able to reproduce.
> Here's my test procedure:
> Booted 151a8
> Created two pools:
> # mkfile 64M /root/features
> # mkfile 64M /root/version28
> # zpool create features /root/features
> # zpool create -d -o version=28 version28 /root/version28
> # zfs create features/test
> # zfs create version28/test
> # dd if=/dev/urandom of=/features/test/wee bs=1024k count=16
> # md5sum /features/test/wee
> # cp /features/test/wee /version28/test/
> # md5sum /version28/test/wee
> # zfs snapshot features/test@foo
> # zfs snapshot version28/test@bar
> # zfs send features/test@foo>features.snapshot
> # zfs send version28/test@bar>version28.snapshot
> 
> copy the snapshots to today's OI, illumos-533d257436
> 
> # zfs receive -v rpool/features< features.snapshot
> # zfs receive -v rpool/version28< version28.snapshot
> # md5sum rpool/features/test/wee
> # md5sum rpool/version28/test/wee
> 
> John
> groenveld@acm.org

I’m afraid it is related to specific feature; we would need to get list of enabled and active features from receiving side (as sending side does not have features).

And then there is a question what is specific about that file which is triggering the receive error…  is it possible to get hands on send stream maybe?

(I’m investigating one send+receive case which is triggering receiving pool to get to suspended state and it is quite a challenge).

rgds,
toomas

> 
> ------------------------------------------
> illumos: illumos-discuss
> Permalink: https://illumos.topicbox.com/groups/discuss/T575f82eecd117456-M395c4264e64bbddd3f0ef346
> Delivery options: https://illumos.topicbox.com/groups/discuss/subscription

Re: [discuss] Cannot receive older ZFS dumps

[Richard Lowe]

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

There's also the case where zfs recv can use large blockno's even when it
shouldn't, but that receives weirdly, not not at all

[-- Attachment #2: Type: text/html, Size: 162 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On 02/08/2024 19:09, John D Groenveld via illumos-discuss wrote:
> In message <1c2abfa3-df73-91cb-79da-c34eaeca3af7@kit.edu>, "Udo Grabowski (IMK)" writes:
>> Did some more testing, could also import it on an 8 years old pool
>> (illumos-a90d75b) with all the features (known at that time) active or enab=
>> led:
>
> I'm not able to reproduce.
> Here's my test procedure:
> Booted 151a8
> Created two pools:
> #...
> # zfs send version28/test@bar>version28.snapshot
>
> copy the snapshots to today's OI, illumos-533d257436
>

The snapshots are older than oi151a8, probably 151a2 or even from
pre-illumos Osol. On 151a8, I can import those, the youngest I could
test was Hipster 2018/04 where they still import. So the incompatibility
is rooted somewhere in commits of the last 6 years.
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 2500 bytes --]

On 02/08/2024 19:21, Toomas Soome via illumos-discuss wrote:
>
>
>> On 2. Aug 2024, at 20:09, John D Groenveld via illumos-discuss <discuss@lists.illumos.org> wrote:
>>
>> In message <1c2abfa3-df73-91cb-79da-c34eaeca3af7@kit.edu>, "Udo Grabowski (IMK)" writes:
>>> Did some more testing, could also import it on an 8 years old pool
>>> (illumos-a90d75b) with all the features (known at that time) active or enab=
>>> led:
>>
>> I'm not able to reproduce.
>> Here's my test procedure:
>> Booted 151a8
>> Created two pools:
>> # mkfile 64M /root/features
>> # mkfile 64M /root/version28
>> # zpool create features /root/features
>> # zpool create -d -o version=28 version28 /root/version28
>> # zfs create features/test
>> # zfs create version28/test
>> # dd if=/dev/urandom of=/features/test/wee bs=1024k count=16
>> # md5sum /features/test/wee
>> # cp /features/test/wee /version28/test/
>> # md5sum /version28/test/wee
>> # zfs snapshot features/test@foo
>> # zfs snapshot version28/test@bar
>> # zfs send features/test@foo>features.snapshot
>> # zfs send version28/test@bar>version28.snapshot
>>
>> copy the snapshots to today's OI, illumos-533d257436
>>
>> # zfs receive -v rpool/features< features.snapshot
>> # zfs receive -v rpool/version28< version28.snapshot
>> # md5sum rpool/features/test/wee
>> # md5sum rpool/version28/test/wee
>>
>> John
>> groenveld@acm.org
>
> I’m afraid it is related to specific feature; we would need to get list of enabled and active features from receiving side (as sending side does not have features).
>
> And then there is a question what is specific about that file which is triggering the receive error…  is it possible to get hands on send stream maybe?
>
> (I’m investigating one send+receive case which is triggering receiving pool to get to suspended state and it is quite a challenge).

Here is the ZFS dump (1.8GB) <https://imk-asf-mipas.imk.kit.edu/users/grabow/> :

32@20100825
32@20100825_xxhsum
features_importable
features_nonimportable

+ xxhsum checksum , + features active/enabled on 2018/4 pool where
it can be imported, and features on a very recent pool where it can't.
The zfs filesystem contained is Results/imk/32.
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 5630 bytes --]



> On 5. Aug 2024, at 11:03, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
> 
> On 02/08/2024 19:21, Toomas Soome via illumos-discuss wrote:
>> 
>> 
>>> On 2. Aug 2024, at 20:09, John D Groenveld via illumos-discuss <discuss@lists.illumos.org> wrote:
>>> 
>>> In message <1c2abfa3-df73-91cb-79da-c34eaeca3af7@kit.edu>, "Udo Grabowski (IMK)" writes:
>>>> Did some more testing, could also import it on an 8 years old pool
>>>> (illumos-a90d75b) with all the features (known at that time) active or enab=
>>>> led:
>>> 
>>> I'm not able to reproduce.
>>> Here's my test procedure:
>>> Booted 151a8
>>> Created two pools:
>>> # mkfile 64M /root/features
>>> # mkfile 64M /root/version28
>>> # zpool create features /root/features
>>> # zpool create -d -o version=28 version28 /root/version28
>>> # zfs create features/test
>>> # zfs create version28/test
>>> # dd if=/dev/urandom of=/features/test/wee bs=1024k count=16
>>> # md5sum /features/test/wee
>>> # cp /features/test/wee /version28/test/
>>> # md5sum /version28/test/wee
>>> # zfs snapshot features/test@foo
>>> # zfs snapshot version28/test@bar
>>> # zfs send features/test@foo>features.snapshot
>>> # zfs send version28/test@bar>version28.snapshot
>>> 
>>> copy the snapshots to today's OI, illumos-533d257436
>>> 
>>> # zfs receive -v rpool/features< features.snapshot
>>> # zfs receive -v rpool/version28< version28.snapshot
>>> # md5sum rpool/features/test/wee
>>> # md5sum rpool/version28/test/wee
>>> 
>>> John
>>> groenveld@acm.org
>> 
>> I’m afraid it is related to specific feature; we would need to get list of enabled and active features from receiving side (as sending side does not have features).
>> 
>> And then there is a question what is specific about that file which is triggering the receive error…  is it possible to get hands on send stream maybe?
>> 
>> (I’m investigating one send+receive case which is triggering receiving pool to get to suspended state and it is quite a challenge).
> 
> Here is the ZFS dump (1.8GB) <https://imk-asf-mipas.imk.kit.edu/users/grabow/> :
> 
> 32@20100825
> 32@20100825_xxhsum
> features_importable
> features_nonimportable
> 
> + xxhsum checksum , + features active/enabled on 2018/4 pool where
> it can be imported, and features on a very recent pool where it can't.
> The zfs filesystem contained is Results/imk/32.

ok, so the feature difference is:

tsoome@beastie:~/zfs-receive-problem$ diff -u features_*
--- features_importable E aug  5 11:30:53 2024
+++ features_nonimportable      E aug  5 11:32:31 2024
@@ -1,4 +1,4 @@
-OI Hipster 2018/4
+OI Hipster illumos-5a33fb2d62 (>2024/4)
 rpool  feature@async_destroy          enabled                        local
 rpool  feature@empty_bpobj            active                         local
 rpool  feature@lz4_compress           active                         local
@@ -6,11 +6,12 @@
 rpool  feature@spacemap_histogram     active                         local
 rpool  feature@enabled_txg            active                         local
 rpool  feature@hole_birth             active                         local
-rpool  feature@extensible_dataset     enabled                        local
+rpool  feature@extensible_dataset     active                         local
 rpool  feature@embedded_data          active                         local
 rpool  feature@bookmarks              enabled                        local
 rpool  feature@filesystem_limits      enabled                        local
 rpool  feature@large_blocks           enabled                        local
+rpool  feature@large_dnode            enabled                        local
 rpool  feature@sha512                 enabled                        local
 rpool  feature@skein                  enabled                        local
 rpool  feature@edonr                  enabled                        local
@@ -18,3 +19,10 @@
 rpool  feature@obsolete_counts        enabled                        local
 rpool  feature@zpool_checkpoint       enabled                        local
 rpool  feature@spacemap_v2            active                         local
+rpool  feature@allocation_classes     enabled                        local
+rpool  feature@resilver_defer         enabled                        local
+rpool  feature@encryption             enabled                        local
+rpool  feature@bookmark_v2            enabled                        local
+rpool  feature@userobj_accounting     active                         local
+rpool  feature@project_quota          active                         local
+rpool  feature@log_spacemap           active                         local
tsoome@beastie:~/zfs-receive-problem$

IMO the first suspects are added active features: extensible_dataset, userobj_accounting, project_quota, log_spacemap.

So one test scenario would be to test newer build, but with those features disabled, if receive is good, then enable one feature and re-test, till we have suspect:)

rgds,
toomas 

> -- 
> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
> https://www.imk-asf.kit.edu/english/sat.php
> KIT - Karlsruhe Institute of Technology          https://www.kit.edu <https://www.kit.edu/>
> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
> 
> 
> ------------------------------------------
> illumos: illumos-discuss
> Permalink: https://illumos.topicbox.com/groups/discuss/T575f82eecd117456-M9f0eb10cbc59ea9652b39fe0
> Delivery options: https://illumos.topicbox.com/groups/discuss/subscription


[-- Attachment #2: Type: text/html, Size: 45251 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 5080 bytes --]

On 05/08/2024 10:57, Toomas Soome via illumos-discuss wrote:
>
>
>> On 5. Aug 2024, at 11:03, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>>
>> On 02/08/2024 19:21, Toomas Soome via illumos-discuss wrote:
>>>
>>>> .....
>>>
>>> I’m afraid it is related to specific feature; we would need to get list of
>>> enabled and active features from receiving side (as sending side does not
>>> have features).
>>>
>>> And then there is a question what is specific about that file which is
>>> triggering the receive error…  is it possible to get hands on send stream maybe?
>>>
>>> (I’m investigating one send+receive case which is triggering receiving pool
>>> to get to suspended state and it is quite a challenge).
>>
>> Here is the ZFS dump (1.8GB) <https://imk-asf-mipas.imk.kit.edu/users/grabow/> :
>>
>> 32@20100825
>> 32@20100825_xxhsum
>> features_importable
>> features_nonimportable
>>
>> + xxhsum checksum , + features active/enabled on 2018/4 pool where
>> it can be imported, and features on a very recent pool where it can't.
>> The zfs filesystem contained is Results/imk/32.
>
> ok, so the feature difference is:
>
> tsoome@beastie:~/zfs-receive-problem$ diff -u features_*
>
> --- features_importable E aug  5 11:30:53 2024
>
> +++ features_nonimportable      E aug  5 11:32:31 2024
>
> @@ -1,4 +1,4 @@
>
> -OI Hipster 2018/4
>
> +OI Hipster illumos-5a33fb2d62 (>2024/4)
>
>  rpool  feature@async_destroy          enabled                        local
>
>  rpool  feature@empty_bpobj            active                         local
>
>  rpool  feature@lz4_compress           active                         local
>
> @@ -6,11 +6,12 @@
>
>  rpool  feature@spacemap_histogram     active                         local
>
>  rpool  feature@enabled_txg            active                         local
>
>  rpool  feature@hole_birth             active                         local
>
> -rpool  feature@extensible_dataset     enabled                        local
>
> +rpool  feature@extensible_dataset     active                         local
>
>  rpool  feature@embedded_data          active                         local
>
>  rpool  feature@bookmarks              enabled                        local
>
>  rpool  feature@filesystem_limits      enabled                        local
>
>  rpool  feature@large_blocks           enabled                        local
>
> +rpool  feature@large_dnode            enabled                        local
>
>  rpool  feature@sha512                 enabled                        local
>
>  rpool  feature@skein                  enabled                        local
>
>  rpool  feature@edonr                  enabled                        local
>
> @@ -18,3 +19,10 @@
>
>  rpool  feature@obsolete_counts        enabled                        local
>
>  rpool  feature@zpool_checkpoint       enabled                        local
>
>  rpool  feature@spacemap_v2            active                         local
>
> +rpool  feature@allocation_classes     enabled                        local
>
> +rpool  feature@resilver_defer         enabled                        local
>
> +rpool  feature@encryption             enabled                        local
>
> +rpool  feature@bookmark_v2            enabled                        local
>
> +rpool  feature@userobj_accounting     active                         local
>
> +rpool  feature@project_quota          active                         local
>
> +rpool  feature@log_spacemap           active                         local
>
> tsoome@beastie:~/zfs-receive-problem$
>
>
> IMO the first suspects are added active
> features: extensible_dataset, userobj_accounting, project_quota, log_spacemap.
>
> So one test scenario would be to test newer build, but with those features
> disabled, if receive is good, then enable one feature and re-test, till we have
> suspect:)
>tps://illumos.topicbox.com/groups/discuss/T575f82eecd117456-M7cca1547d1a99b935668d909>
>

Doesn't help at all, so that problem does not seem to be related to features:

# zpool create -o feature@extensible_dataset=disabled -o 
feature@userobj_accounting=disabled -o feature@project_quota=disabled -o 
feature@log_spacemap=disabled -o feature@allocation_classes=disabled -o 
feature@large_dnode=disabled -o feature@encryption=disabled -o 
feature@spacemap_v2=disabled -o feature@bookmark_v2=disabled -o 
feature@obsolete_counts=disabled -o feature@zpool_checkpoint=disabled -o 
feature@async_destroy=disabled -o feature@empty_bpobj=disabled -o 
feature@lz4_compress=disabled test /tmp/pool

# cat /home/lsdf/SAT/Processor/Results/imk/32@20100825 | zfs receive -d test
cannot receive new filesystem stream: invalid backup stream

#
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [discuss] Cannot receive older ZFS dumps

[Joshua M. Clulow]

On Mon, 5 Aug 2024 at 05:03, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
> On 05/08/2024 10:57, Toomas Soome via illumos-discuss wrote:
> > So one test scenario would be to test newer build, but with those features
> > disabled, if receive is good, then enable one feature and re-test, till we have
> > suspect:)
> Doesn't help at all, so that problem does not seem to be related to features:

Whether features are turned on or off, the _code_ has changed in the
meantime.  It seems like it would be possible on some level to bisect
back to previous package versions, but I expect it would be very
tedious, especially if they've been pruned from the repository in the
meantime, etc.

> # cat /home/lsdf/SAT/Processor/Results/imk/32@20100825 | zfs receive -d test
> cannot receive new filesystem stream: invalid backup stream

As I suggested earlier, I think it's best to start by looking at what
the current version of the software is actually doing and work
backwards to the problem; e.g.,

  * What code produces that error message?
    (presumably in zfs, the command, or libzfs)

  * What causes it to produce that error message?
    (presumably some errno returned from the kernel)

  * What causes the errno to be returned by the kernel?
    (presumably it's occuring in the attempt to process
     one of the stream records being fed in from usermode)

  * etc, until we find the reason it's confused

You should be able to catch the issue as it occurs with DTrace and a
little reading of the code, etc.  I would start by locating in the
source the specific error message you see, i.e., "invalid backup
stream", and see where it is printed, etc.


Cheers.

-- 
Joshua M. Clulow
http://blog.sysmgr.org