Re: [discuss] Illumos future and compatibility with Open-ZFS

[Toomas Soome <tsoome@me.com>]

[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]



> On 9. Sep 2024, at 13:41, Gea <gea@napp-it.org> wrote:
> 
> On 08.09.2024 11:55, d wrote:
>> This post enticed me to do a little research on zfs compression, which lead me to some interesting info an a question:
>> https://indico.fnal.gov/event/16264/contributions/36466/attachments/22610/28037/Zstd__LZ4.pdf
> 
> The simplified result seems lz4 is faster while zstd compress data more efficient.
> 
> When I look at my main pool, refcompressratio with lz4 is 1.04 for my main data filesystem with mixed data, some at 1.02 with mainly iso and media and up to 1.12 in rare cases. With expensive NVMe or hybrid pools with a special vdev, I would opt for zstd and better compressratio as performance is better than good enough. But I have this choice only with Open-ZFS and this for quite a long time similar to draid with many advantages on very large pools.

tsoome@beastie:~$ zfs get compression rpool/ROOT/openindiana-2024:08:23
NAME                               PROPERTY     VALUE           SOURCE
rpool/ROOT/openindiana-2024:08:23  compression  zstd            inherited from rpool
tsoome@beastie:~$ zfs get compressratio  rpool/ROOT/openindiana-2024:08:23
NAME                               PROPERTY       VALUE  SOURCE
rpool/ROOT/openindiana-2024:08:23  compressratio  2.75x  -

tsoome@beastie:~$ zfs get compression rpool/code/illumos-gate
NAME                     PROPERTY     VALUE           SOURCE
rpool/code/illumos-gate  compression  zstd            local
tsoome@beastie:~$ zfs get compressratio rpool/code/illumos-gate
NAME                     PROPERTY       VALUE  SOURCE
rpool/code/illumos-gate  compressratio  3.32x  -
tsoome@beastie:~$

(I probably still do have some lz4 blocks lurking around).

rgds,
toomas

> 
> The upcoming Fast Dedup in Open-ZFS where you can limit dedup table size with a quota,  place the dedup table not only on a dedicated dedup vdev but also a special vdev, can shrink dedup tables from single items or cache dedup in Arc seems a killer feature. Unlike current dedup, I would expect that you can enable Fast Dedup per default, either with a special vdev that also massively improve all small file actions and metadata access or with a quota like a few GB to limit RAM usage. Hybrid pools with a special vdev become more and more attractive as suggested default. Fast dedup is beta but near to be ready. I already can play with in in Open-ZFS on Windows rc. Will this appear in Illumos in the forseeable future. I doubt.
> 
> The stability aspect, ok. Software has bugs and you need to maintain does not matter if Open-ZFS or Illumos-ZFS. There have also been critical bugs in Illumos. When I remember correctly it was persistent L2Arc in Illumos where datalosses could have happened a few years ago (luckily I was not affected as silent errors could have happened).
> 
> Why do anyone expects that bugs in Illumos ZFS do not happen or fixed faster than bugs in Open-ZFS with 20x the user base and number of devs and many large commercial companies behind. Even the additional tests on taking over Open-ZFS code in more and more rare cases does not guarantee that there are no bugs remained (while helpful for sure and better as using Open-ZFS master or the stable branch directly)
> 



[-- Attachment #2: Type: text/html, Size: 15469 bytes --]