Status of NFSv4.1 support in Illumos?

Status of NFSv4.1 support in Illumos?

[Dan Shelton]

Hello!

Does Illumos support NFSv4.1 in the NFS server, and how complete is
this implementation?
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: [developer] Status of NFSv4.1 support in Illumos?

[Toomas Soome]

> On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> 
> Hello!
> 
> Does Illumos support NFSv4.1 in the NFS server, and how complete is
> this implementation?
> -- 
> Dan Shelton - Cluster Specialist Win/Lin/Bsd
> 

It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.

rgds,
toomas

Re: [developer] Status of NFSv4.1 support in Illumos?

[Cedric Blancher]

On Tue, 2 Jan 2024 at 09:22, Toomas Soome via illumos-developer
<developer@lists.illumos.org> wrote:
>
>
>
> > On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> >
> > Hello!
> >
> > Does Illumos support NFSv4.1 in the NFS server, and how complete is
> > this implementation?
> > --
> > Dan Shelton - Cluster Specialist Win/Lin/Bsd
> >
>
> It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.

Do you have an Openindiana ISO with those patches?

Ced

-- 
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur

Re: [developer] Status of NFSv4.1 support in Illumos?

[Dan Shelton]

On Tue, 2 Jan 2024 at 09:22, Toomas Soome via illumos-developer
<developer@lists.illumos.org> wrote:
>
>
>
> > On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> >
> > Hello!
> >
> > Does Illumos support NFSv4.1 in the NFS server, and how complete is
> > this implementation?
> > --
> > Dan Shelton - Cluster Specialist Win/Lin/Bsd
> >
>
> It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.

Why can the current work not be merged into the MASTER, off by
default, so people can test it?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: [developer] Status of NFSv4.1 support in Illumos?

[gusev.vitaliy]

> On 25 Jan 2024, at 06:23, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> 
> On Tue, 2 Jan 2024 at 09:22, Toomas Soome via illumos-developer
> <developer@lists.illumos.org> wrote:
>> 
>> 
>> 
>>> On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>> 
>>> Hello!
>>> 
>>> Does Illumos support NFSv4.1 in the NFS server, and how complete is
>>> this implementation?
>>> --
>>> Dan Shelton - Cluster Specialist Win/Lin/Bsd
>>> 
>> 
>> It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.
> 
> Why can the current work not be merged into the MASTER, off by
> default, so people can test it?
> 
> Dan
> -- 
> Dan Shelton - Cluster Specialist Win/Lin/Bsd
> 
> ------------------------------------------
> illumos: illumos-developer
> Permalink: https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M32b27a946cff13426d3024c4
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription

Re: [developer] Status of NFSv4.1 support in Illumos?

[gusev.vitaliy]

Hi,

> On 25 Jan 2024, at 06:23, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> 
> On Tue, 2 Jan 2024 at 09:22, Toomas Soome via illumos-developer
> <developer@lists.illumos.org> wrote:
>> 
>> 
>> 
>>> On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>> 
>>> Hello!
>>> 
>>> Does Illumos support NFSv4.1 in the NFS server, and how complete is
>>> this implementation?
>>> --
>>> Dan Shelton - Cluster Specialist Win/Lin/Bsd
>>> 
>> 
>> It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.
> 
> Why can the current work not be merged into the MASTER, off by
> default, so people can test it?

I believe it will be done soon. So please wait.

——
Vitaliy

Re: [developer] Status of NFSv4.1 support in Illumos?

[gusev.vitaliy]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

NFSv4.1+ change was integrated yesterday:

15405 Port NFSv41 base
https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
 
So please try it on and report results! 

Note, it doesn’t have backchannel yet.

—
Vitaliy Gusev

> On 25 Jan 2024, at 18:00, GUSEV.VITALIY@icloud.com <GUSEV.VITALIY@ICLOUD.COM> wrote:
> 
> 
> Hi,
> 
>> On 25 Jan 2024, at 06:23, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>> 
>> On Tue, 2 Jan 2024 at 09:22, Toomas Soome via illumos-developer
>> <developer@lists.illumos.org> wrote:
>>> 
>>> 
>>> 
>>>> On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>>> 
>>>> Hello!
>>>> 
>>>> Does Illumos support NFSv4.1 in the NFS server, and how complete is
>>>> this implementation?
>>>> --
>>>> Dan Shelton - Cluster Specialist Win/Lin/Bsd
>>>> 
>>> 
>>> It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.
>> 
>> Why can the current work not be merged into the MASTER, off by
>> default, so people can test it?
> 
> I believe it will be done soon. So please wait.
> 
> ——
> Vitaliy


[-- Attachment #2: Type: text/html, Size: 6046 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]



> On 8. Feb 2024, at 12:28, gusev.vitaliy via illumos-developer <developer@lists.illumos.org> wrote:
> 
> NFSv4.1+ change was integrated yesterday:
> 
> 15405 Port NFSv41 base
> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>  
> So please try it on and report results! 
> 
> Note, it doesn’t have backchannel yet.


Also please note, to try 4.1 (or 4.2), you need to: sharectl set -p server_versmax=4.1 nfs

Also, our client code is still 4.0.

rgds,
toomas


> 
> —
> Vitaliy Gusev
> 
>> On 25 Jan 2024, at 18:00, GUSEV.VITALIY@icloud.com <GUSEV.VITALIY@ICLOUD.COM> wrote:
>> 
>> 
>> Hi,
>> 
>>> On 25 Jan 2024, at 06:23, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>> 
>>> On Tue, 2 Jan 2024 at 09:22, Toomas Soome via illumos-developer
>>> <developer@lists.illumos.org> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On 2. Jan 2024, at 02:39, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>>>> 
>>>>> Hello!
>>>>> 
>>>>> Does Illumos support NFSv4.1 in the NFS server, and how complete is
>>>>> this implementation?
>>>>> --
>>>>> Dan Shelton - Cluster Specialist Win/Lin/Bsd
>>>>> 
>>>> 
>>>> It is currently not supported, there is ongoing work to get initial support, but this initial support is without backchannel implementation.
>>> 
>>> Why can the current work not be merged into the MASTER, off by
>>> default, so people can test it?
>> 
>> I believe it will be done soon. So please wait.
>> 
>> ——
>> Vitaliy
> 
> illumos <https://illumos.topicbox.com/latest> / illumos-developer / see discussions <https://illumos.topicbox.com/groups/developer> + participants <https://illumos.topicbox.com/groups/developer/members> + delivery options <https://illumos.topicbox.com/groups/developer/subscription>Permalink <https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-Md56f5e2c34213fa6bc34143c>

[-- Attachment #2: Type: text/html, Size: 10152 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
> NFSv4.1+ change was integrated yesterday:
>
>     15405 Port NFSv41 base
>     https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>
>
> So please try it on and report results!
>

With an old server, but a new client with that code, I get occasional compound
errors:

On client (illumos-8b0687e22a):

NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode 
arguments)

On server (oi_151a9):

[kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure 
RFS4_COMPOUND client

client mount parameters:

  -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4

   remote/read/write/setuid/devices/vers=4/xattr

-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Gordon Ross]

Does this happen with enough predictability that you can get a network capture?

On Fri, Mar 1, 2024 at 4:45 AM Udo Grabowski (IMK)
<udo.grabowski@kit.edu> wrote:
>
> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
> > NFSv4.1+ change was integrated yesterday:
> >
> >     15405 Port NFSv41 base
> >     https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
> >
> >
> > So please try it on and report results!
> >
>
> With an old server, but a new client with that code, I get occasional compound
> errors:
>
> On client (illumos-8b0687e22a):
>
> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
> arguments)
>
> On server (oi_151a9):
>
> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
> RFS4_COMPOUND client
>
> client mount parameters:
>
>   -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>
>    remote/read/write/setuid/devices/vers=4/xattr
>
> --
> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
> https://www.imk-asf.kit.edu/english/sat.php
> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
>
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink: https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-Mc0f86b18bb75cb530abebf3e
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 01/03/2024 16:04, Gordon Ross wrote:
> Does this happen with enough predictability that you can get a network capture?
>

This is typically happening at the closing stage of firefox, so I'll try to
grab it with wireshark.

> On Fri, Mar 1, 2024 at 4:45 AM Udo Grabowski (IMK)
> <udo.grabowski@kit.edu> wrote:
>>
>> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>>> NFSv4.1+ change was integrated yesterday:
>>>
>>>     15405 Port NFSv41 base
>>>     https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>
>>>
>>> So please try it on and report results!
>>>
>>
>> With an old server, but a new client with that code, I get occasional compound
>> errors:
>>
>> On client (illumos-8b0687e22a):
>>
>> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
>> arguments)
>>
>> On server (oi_151a9):
>>
>> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
>> RFS4_COMPOUND client
>>
>> client mount parameters:
>>
>> -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>>
>> remote/read/write/setuid/devices/vers=4/xattr
>>
>> --
>> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
>> https://www.imk-asf.kit.edu/english/sat.php
>> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
>> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
>>
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink: https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-Mefdae69b197d2feaca8302a7
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Dan Shelton]

On Fri, 1 Mar 2024 at 10:45, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>
> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
> > NFSv4.1+ change was integrated yesterday:
> >
> >     15405 Port NFSv41 base
> >     https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
> >
> >
> > So please try it on and report results!
> >
>
> With an old server, but a new client with that code, I get occasional compound
> errors:
>
> On client (illumos-8b0687e22a):
>
> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
> arguments)

Seriously, who wrote that error message? It should have more details,
which OP code, number of arguments?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

On 01/03/2024 10:44, Udo Grabowski (IMK) wrote:
> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>> NFSv4.1+ change was integrated yesterday:
>>
>>     15405 Port NFSv41 base
>>
>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>
>>
>>
>> So please try it on and report results!
>>
>
> With an old server, but a new client with that code, I get occasional compound
> errors:
>
> On client (illumos-8b0687e22a):
>
> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
> arguments)
>
> On server (oi_151a9):
>
> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
> RFS4_COMPOUND client
>
> client mount parameters:
>
>  -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>
>   remote/read/write/setuid/devices/vers=4/xattr
>

I've to correct myself - this happens also with new server and new client,
for instance, at closing firefox. I've not yet found the RPC packets
responsible in wireshark (there are so many). Firefox has
storage.nfs_filesystem set to true, so that's not the culprit. There are
a couple of bug reports, also at illumos from 7 years ago:
<https://www.illumos.org/issues/7434> ,
I see badcalls for NFS client rpc, but not the xdrcalls mentioned there.

Since this started just recently with the new code, some bug must
have been reintroduced with that. I've also experienced a lockup of
the NFS mount once with that error, unable to use it anymore, so
probably that's not something rooted in firefox itself.
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 6507 bytes --]

Got'em !

(mounted also with vers=4.0 instead of vers=4, but no change)

New server, new client (illumos-71b3c7bf69)

On Firefox close this happens two times (also got similar stuff two times
just with a simple cat, didn't catch it):

Client Call:
============
Frame 7270: 254 bytes on wire (2032 bits), 254 bytes captured (2032 bits)
Ethernet II, Src: 00:00:5a:71:78:7c (00:00:5a:71:78:7c), Dst: 00:1b:21:c0:07:e2 
(00:1b:21:c0:07:e2)
Internet Protocol Version 4, Src: 172.23.156.70, Dst: 172.23.156.24
Transmission Control Protocol, Src Port: 1005 (1005), Dst Port: 2049 (2049), 
Seq: 144845, Ack: 9997, Len: 188
Remote Procedure Call, Type:Call XID:0xda85d831
     Fragment header: Last fragment, 184 bytes
         1... .... .... .... .... .... .... .... = Last Fragment: Yes
         .000 0000 0000 0000 0000 0000 1011 1000 = Fragment Length: 184
     XID: 0xda85d831 (3666204721)
     Message Type: Call (0)
     RPC Version: 2
     Program: NFS (100003)
     Program Version: 4
     Procedure: COMPOUND (1)
     [The reply to this request is in frame 7271]
     Credentials
         Flavor: AUTH_UNIX (1)
         Length: 32
         Stamp: 0x660c13de
         Machine Name: imksunts
             length: 8
             contents: imksunts
         UID: 1001
         GID: 360
         Auxiliary GIDs (1) [360]
             GID: 360
     Verifier
         Flavor: AUTH_NULL (0)
         Length: 0
Network File System, Ops(3): PUTFH, GETATTR, OPEN_DOWNGRADE
     [Program Version: 4]
     [V4 Procedure: COMPOUND (1)]
     Tag: open dgrade
     minorversion: 0
     Operations (count: 3): PUTFH, GETATTR, OPEN_DOWNGRADE
[Malformed Packet: NFS]
     [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
         [Malformed Packet (Exception occurred)]
         [Severity level: Error]
         [Group: Malformed]

== Hexdump
0000   00 1b 21 c0 07 e2 00 00 5a 71 78 7c 08 00 45 00
0010   00 f0 78 b7 40 00 40 06 30 c3 ac 17 9c 46 ac 17
0020   9c 18 03 ed 08 01 fa 31 39 7d 50 81 51 16 80 18
0030   80 26 13 76 00 00 01 01 08 0a 00 03 49 f7 00 89
0040   12 87 80 00 00 b8 da 85 d8 31 00 00 00 00 00 00
0050   00 02 00 01 86 a3 00 00 00 04 00 00 00 01 00 00
0060   00 01 00 00 00 20 66 0c 13 de 00 00 00 08 69 6d
0070   6b 73 75 6e 74 73 00 00 03 e9 00 00 01 68 00 00
0080   00 01 00 00 01 68 00 00 00 00 00 00 00 00 00 00
0090   00 0c 6f 70 65 6e 20 64 67 72 61 64 65 20 00 00
00a0   00 00 00 00 00 03 00 00 00 16 00 00 00 24 f3 73
00b0   85 d4 08 ad 4a fb 0a 00 12 0e 00 00 00 00 b9 c0
00c0   41 00 0a 00 22 00 00 00 00 00 eb bf 41 00 00 00
00d0   00 00 00 00 00 09 00 00 00 02 00 10 01 1a 00 b0
00e0   a2 3a 00 00 00 15 00 00 00 01 b2 fa 0b 66 00 f0
00f0   54 00 00 00 00 00 00 00 02 5a 00 00 00 00


Server Answer:
==============
Frame 7271: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
Ethernet II, Src: 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2), Dst: 00:00:5a:71:78:7c 
(00:00:5a:71:78:7c)
Internet Protocol Version 4, Src: 172.23.156.24, Dst: 172.23.156.70
Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 1005 (1005), 
Seq: 9997, Ack: 145033, Len: 28
Remote Procedure Call, Type:Reply XID:0xda85d831
     Fragment header: Last fragment, 24 bytes
         1... .... .... .... .... .... .... .... = Last Fragment: Yes
         .000 0000 0000 0000 0000 0000 0001 1000 = Fragment Length: 24
     XID: 0xda85d831 (3666204721)
     Message Type: Reply (1)
     [Program: NFS (100003)]
     [Program Version: 4]
     [Procedure: COMPOUND (1)]
     Reply State: accepted (0)
     [This is a reply to a request in frame 7270]
     [Time from request: 0.000245000 seconds]
     Verifier
         Flavor: AUTH_NULL (0)
         Length: 0
     Accept State: procedure can't decode params (4)

== Hexdump
0000   00 00 5a 71 78 7c 00 1b 21 c0 07 e2 08 00 45 00
0010   00 50 a7 60 40 00 40 06 02 ba ac 17 9c 18 ac 17
0020   9c 46 08 01 03 ed 50 81 51 16 fa 31 3a 39 80 18
0030   80 26 f4 12 00 00 01 01 08 0a 00 89 12 89 00 03
0040   49 f7 80 00 00 18 da 85 d8 31 00 00 00 01 00 00
0050   00 00 00 00 00 00 00 00 00 00 00 00 00 04

NFS compound failed for server imksunth9: error 11 (RPC: Server can't decode 
arguments)


On 28/03/2024 17:48, Udo Grabowski (IMK) wrote:
> On 01/03/2024 10:44, Udo Grabowski (IMK) wrote:
>> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>>> NFSv4.1+ change was integrated yesterday:
>>>
>>>     15405 Port NFSv41 base
>>>
>>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>
>>>
>>>
>>>
>>> So please try it on and report results!
>>>
>>
>> With an old server, but a new client with that code, I get occasional compound
>> errors:
>>
>> On client (illumos-8b0687e22a):
>>
>> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
>> arguments)
>>
>> On server (oi_151a9):
>>
>> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
>> RFS4_COMPOUND client
>>
>> client mount parameters:
>>
>>  -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>>
>>   remote/read/write/setuid/devices/vers=4/xattr
>>
>
> I've to correct myself - this happens also with new server and new client,
> for instance, at closing firefox. I've not yet found the RPC packets
> responsible in wireshark (there are so many). Firefox has
> storage.nfs_filesystem set to true, so that's not the culprit. There are
> a couple of bug reports, also at illumos from 7 years ago:
> <https://www.illumos.org/issues/7434> ,
> I see badcalls for NFS client rpc, but not the xdrcalls mentioned there.
>
> Since this started just recently with the new code, some bug must
> have been reintroduced with that. I've also experienced a lockup of
> the NFS mount once with that error, unable to use it anymore, so
> probably that's not something rooted in firefox itself.
>
>
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink:
> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M4fed1844efa6ec0df19e0d10
>
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[gusev.vitaliy]

[-- Attachment #1: Type: text/plain, Size: 7225 bytes --]

Do I understand right  that didn’t have this issue with old server ? If no, could you check that?

Also can you send me pcap?

—
Vitaliy 

> On 2 Apr 2024, at 18:09, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
> 
> Got'em !
> 
> (mounted also with vers=4.0 instead of vers=4, but no change)
> 
> New server, new client (illumos-71b3c7bf69)
> 
> On Firefox close this happens two times (also got similar stuff two times
> just with a simple cat, didn't catch it):
> 
> Client Call:
> ============
> Frame 7270: 254 bytes on wire (2032 bits), 254 bytes captured (2032 bits)
> Ethernet II, Src: 00:00:5a:71:78:7c (00:00:5a:71:78:7c), Dst: 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2)
> Internet Protocol Version 4, Src: 172.23.156.70, Dst: 172.23.156.24
> Transmission Control Protocol, Src Port: 1005 (1005), Dst Port: 2049 (2049), Seq: 144845, Ack: 9997, Len: 188
> Remote Procedure Call, Type:Call XID:0xda85d831
>    Fragment header: Last fragment, 184 bytes
>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>        .000 0000 0000 0000 0000 0000 1011 1000 = Fragment Length: 184
>    XID: 0xda85d831 (3666204721)
>    Message Type: Call (0)
>    RPC Version: 2
>    Program: NFS (100003)
>    Program Version: 4
>    Procedure: COMPOUND (1)
>    [The reply to this request is in frame 7271]
>    Credentials
>        Flavor: AUTH_UNIX (1)
>        Length: 32
>        Stamp: 0x660c13de
>        Machine Name: imksunts
>            length: 8
>            contents: imksunts
>        UID: 1001
>        GID: 360
>        Auxiliary GIDs (1) [360]
>            GID: 360
>    Verifier
>        Flavor: AUTH_NULL (0)
>        Length: 0
> Network File System, Ops(3): PUTFH, GETATTR, OPEN_DOWNGRADE
>    [Program Version: 4]
>    [V4 Procedure: COMPOUND (1)]
>    Tag: open dgrade
>    minorversion: 0
>    Operations (count: 3): PUTFH, GETATTR, OPEN_DOWNGRADE
> [Malformed Packet: NFS]
>    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
>        [Malformed Packet (Exception occurred)]
>        [Severity level: Error]
>        [Group: Malformed]
> 
> == Hexdump
> 0000   00 1b 21 c0 07 e2 00 00 5a 71 78 7c 08 00 45 00
> 0010   00 f0 78 b7 40 00 40 06 30 c3 ac 17 9c 46 ac 17
> 0020   9c 18 03 ed 08 01 fa 31 39 7d 50 81 51 16 80 18
> 0030   80 26 13 76 00 00 01 01 08 0a 00 03 49 f7 00 89
> 0040   12 87 80 00 00 b8 da 85 d8 31 00 00 00 00 00 00
> 0050   00 02 00 01 86 a3 00 00 00 04 00 00 00 01 00 00
> 0060   00 01 00 00 00 20 66 0c 13 de 00 00 00 08 69 6d
> 0070   6b 73 75 6e 74 73 00 00 03 e9 00 00 01 68 00 00
> 0080   00 01 00 00 01 68 00 00 00 00 00 00 00 00 00 00
> 0090   00 0c 6f 70 65 6e 20 64 67 72 61 64 65 20 00 00
> 00a0   00 00 00 00 00 03 00 00 00 16 00 00 00 24 f3 73
> 00b0   85 d4 08 ad 4a fb 0a 00 12 0e 00 00 00 00 b9 c0
> 00c0   41 00 0a 00 22 00 00 00 00 00 eb bf 41 00 00 00
> 00d0   00 00 00 00 00 09 00 00 00 02 00 10 01 1a 00 b0
> 00e0   a2 3a 00 00 00 15 00 00 00 01 b2 fa 0b 66 00 f0
> 00f0   54 00 00 00 00 00 00 00 02 5a 00 00 00 00
> 
> 
> Server Answer:
> ==============
> Frame 7271: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
> Ethernet II, Src: 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2), Dst: 00:00:5a:71:78:7c (00:00:5a:71:78:7c)
> Internet Protocol Version 4, Src: 172.23.156.24, Dst: 172.23.156.70
> Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 1005 (1005), Seq: 9997, Ack: 145033, Len: 28
> Remote Procedure Call, Type:Reply XID:0xda85d831
>    Fragment header: Last fragment, 24 bytes
>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>        .000 0000 0000 0000 0000 0000 0001 1000 = Fragment Length: 24
>    XID: 0xda85d831 (3666204721)
>    Message Type: Reply (1)
>    [Program: NFS (100003)]
>    [Program Version: 4]
>    [Procedure: COMPOUND (1)]
>    Reply State: accepted (0)
>    [This is a reply to a request in frame 7270]
>    [Time from request: 0.000245000 seconds]
>    Verifier
>        Flavor: AUTH_NULL (0)
>        Length: 0
>    Accept State: procedure can't decode params (4)
> 
> == Hexdump
> 0000   00 00 5a 71 78 7c 00 1b 21 c0 07 e2 08 00 45 00
> 0010   00 50 a7 60 40 00 40 06 02 ba ac 17 9c 18 ac 17
> 0020   9c 46 08 01 03 ed 50 81 51 16 fa 31 3a 39 80 18
> 0030   80 26 f4 12 00 00 01 01 08 0a 00 89 12 89 00 03
> 0040   49 f7 80 00 00 18 da 85 d8 31 00 00 00 01 00 00
> 0050   00 00 00 00 00 00 00 00 00 00 00 00 00 04
> 
> NFS compound failed for server imksunth9: error 11 (RPC: Server can't decode arguments)
> 
> 
> On 28/03/2024 17:48, Udo Grabowski (IMK) wrote:
>> On 01/03/2024 10:44, Udo Grabowski (IMK) wrote:
>>> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>>>> NFSv4.1+ change was integrated yesterday:
>>>> 
>>>>    15405 Port NFSv41 base
>>>> 
>>>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>> 
>>>> 
>>>> 
>>>> 
>>>> So please try it on and report results!
>>>> 
>>> 
>>> With an old server, but a new client with that code, I get occasional compound
>>> errors:
>>> 
>>> On client (illumos-8b0687e22a):
>>> 
>>> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
>>> arguments)
>>> 
>>> On server (oi_151a9):
>>> 
>>> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
>>> RFS4_COMPOUND client
>>> 
>>> client mount parameters:
>>> 
>>> -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>>> 
>>>  remote/read/write/setuid/devices/vers=4/xattr
>>> 
>> 
>> I've to correct myself - this happens also with new server and new client,
>> for instance, at closing firefox. I've not yet found the RPC packets
>> responsible in wireshark (there are so many). Firefox has
>> storage.nfs_filesystem set to true, so that's not the culprit. There are
>> a couple of bug reports, also at illumos from 7 years ago:
>> <https://www.illumos.org/issues/7434> ,
>> I see badcalls for NFS client rpc, but not the xdrcalls mentioned there.
>> 
>> Since this started just recently with the new code, some bug must
>> have been reintroduced with that. I've also experienced a lockup of
>> the NFS mount once with that error, unable to use it anymore, so
>> probably that's not something rooted in firefox itself.
>> 
>> 
>> 
>> ------------------------------------------
>> illumos: illumos-developer
>> Permalink:
>> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M4fed1844efa6ec0df19e0d10
>> 
>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
> 
> 
> -- 
> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
> https://www.imk-asf.kit.edu/english/sat.php
> KIT - Karlsruhe Institute of Technology          https://www.kit.edu <https://www.kit.edu/>
> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
> 
> 
> ------------------------------------------
> illumos: illumos-developer
> Permalink: https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M8e2daabe8111730c5b8917c9
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription


[-- Attachment #2: Type: text/html, Size: 87315 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 8049 bytes --]

On 03/04/2024 00:34, gusev.vitaliy@icloud.com wrote:
> Do I understand right  that didn’t have this issue with old server ? If no,
> could you check that?
>

It never occurres between 151a9 server and client connection, but
also happens between illumos-.... (starting with the 4.1-2 feature)
clients and 151a9 servers. So it's certainly a client problem in the
new code, probably around that open_downgrade call.

> Also can you send me pcap?
>

I'll upload it tomorrow (about 10h from now) to somewhere.


> —
> Vitaliy
>
>> On 2 Apr 2024, at 18:09, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>>
>> Got'em !
>>
>> (mounted also with vers=4.0 instead of vers=4, but no change)
>>
>> New server, new client (illumos-71b3c7bf69)
>>
>> On Firefox close this happens two times (also got similar stuff two times
>> just with a simple cat, didn't catch it):
>>
>> Client Call:
>> ============
>> Frame 7270: 254 bytes on wire (2032 bits), 254 bytes captured (2032 bits)
>> Ethernet II, Src: 00:00:5a:71:78:7c (00:00:5a:71:78:7c), Dst:
>> 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2)
>> Internet Protocol Version 4, Src: 172.23.156.70, Dst: 172.23.156.24
>> Transmission Control Protocol, Src Port: 1005 (1005), Dst Port: 2049 (2049),
>> Seq: 144845, Ack: 9997, Len: 188
>> Remote Procedure Call, Type:Call XID:0xda85d831
>>    Fragment header: Last fragment, 184 bytes
>>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>>        .000 0000 0000 0000 0000 0000 1011 1000 = Fragment Length: 184
>>    XID: 0xda85d831 (3666204721)
>>    Message Type: Call (0)
>>    RPC Version: 2
>>    Program: NFS (100003)
>>    Program Version: 4
>>    Procedure: COMPOUND (1)
>>    [The reply to this request is in frame 7271]
>>    Credentials
>>        Flavor: AUTH_UNIX (1)
>>        Length: 32
>>        Stamp: 0x660c13de
>>        Machine Name: imksunts
>>            length: 8
>>            contents: imksunts
>>        UID: 1001
>>        GID: 360
>>        Auxiliary GIDs (1) [360]
>>            GID: 360
>>    Verifier
>>        Flavor: AUTH_NULL (0)
>>        Length: 0
>> Network File System, Ops(3): PUTFH, GETATTR, OPEN_DOWNGRADE
>>    [Program Version: 4]
>>    [V4 Procedure: COMPOUND (1)]
>>    Tag: open dgrade
>>    minorversion: 0
>>    Operations (count: 3): PUTFH, GETATTR, OPEN_DOWNGRADE
>> [Malformed Packet: NFS]
>>    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
>>        [Malformed Packet (Exception occurred)]
>>        [Severity level: Error]
>>        [Group: Malformed]
>>
>> == Hexdump
>> 0000   00 1b 21 c0 07 e2 00 00 5a 71 78 7c 08 00 45 00
>> 0010   00 f0 78 b7 40 00 40 06 30 c3 ac 17 9c 46 ac 17
>> 0020   9c 18 03 ed 08 01 fa 31 39 7d 50 81 51 16 80 18
>> 0030   80 26 13 76 00 00 01 01 08 0a 00 03 49 f7 00 89
>> 0040   12 87 80 00 00 b8 da 85 d8 31 00 00 00 00 00 00
>> 0050   00 02 00 01 86 a3 00 00 00 04 00 00 00 01 00 00
>> 0060   00 01 00 00 00 20 66 0c 13 de 00 00 00 08 69 6d
>> 0070   6b 73 75 6e 74 73 00 00 03 e9 00 00 01 68 00 00
>> 0080   00 01 00 00 01 68 00 00 00 00 00 00 00 00 00 00
>> 0090   00 0c 6f 70 65 6e 20 64 67 72 61 64 65 20 00 00
>> 00a0   00 00 00 00 00 03 00 00 00 16 00 00 00 24 f3 73
>> 00b0   85 d4 08 ad 4a fb 0a 00 12 0e 00 00 00 00 b9 c0
>> 00c0   41 00 0a 00 22 00 00 00 00 00 eb bf 41 00 00 00
>> 00d0   00 00 00 00 00 09 00 00 00 02 00 10 01 1a 00 b0
>> 00e0   a2 3a 00 00 00 15 00 00 00 01 b2 fa 0b 66 00 f0
>> 00f0   54 00 00 00 00 00 00 00 02 5a 00 00 00 00
>>
>>
>> Server Answer:
>> ==============
>> Frame 7271: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
>> Ethernet II, Src: 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2), Dst:
>> 00:00:5a:71:78:7c (00:00:5a:71:78:7c)
>> Internet Protocol Version 4, Src: 172.23.156.24, Dst: 172.23.156.70
>> Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 1005 (1005),
>> Seq: 9997, Ack: 145033, Len: 28
>> Remote Procedure Call, Type:Reply XID:0xda85d831
>>    Fragment header: Last fragment, 24 bytes
>>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>>        .000 0000 0000 0000 0000 0000 0001 1000 = Fragment Length: 24
>>    XID: 0xda85d831 (3666204721)
>>    Message Type: Reply (1)
>>    [Program: NFS (100003)]
>>    [Program Version: 4]
>>    [Procedure: COMPOUND (1)]
>>    Reply State: accepted (0)
>>    [This is a reply to a request in frame 7270]
>>    [Time from request: 0.000245000 seconds]
>>    Verifier
>>        Flavor: AUTH_NULL (0)
>>        Length: 0
>>    Accept State: procedure can't decode params (4)
>>
>> == Hexdump
>> 0000   00 00 5a 71 78 7c 00 1b 21 c0 07 e2 08 00 45 00
>> 0010   00 50 a7 60 40 00 40 06 02 ba ac 17 9c 18 ac 17
>> 0020   9c 46 08 01 03 ed 50 81 51 16 fa 31 3a 39 80 18
>> 0030   80 26 f4 12 00 00 01 01 08 0a 00 89 12 89 00 03
>> 0040   49 f7 80 00 00 18 da 85 d8 31 00 00 00 01 00 00
>> 0050   00 00 00 00 00 00 00 00 00 00 00 00 00 04
>>
>> NFS compound failed for server imksunth9: error 11 (RPC: Server can't decode
>> arguments)
>>
>>
>> On 28/03/2024 17:48, Udo Grabowski (IMK) wrote:
>>> On 01/03/2024 10:44, Udo Grabowski (IMK) wrote:
>>>> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>>>>> NFSv4.1+ change was integrated yesterday:
>>>>>
>>>>>    15405 Port NFSv41 base
>>>>>
>>>>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> So please try it on and report results!
>>>>>
>>>>
>>>> With an old server, but a new client with that code, I get occasional compound
>>>> errors:
>>>>
>>>> On client (illumos-8b0687e22a):
>>>>
>>>> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
>>>> arguments)
>>>>
>>>> On server (oi_151a9):
>>>>
>>>> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
>>>> RFS4_COMPOUND client
>>>>
>>>> client mount parameters:
>>>>
>>>> -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>>>>
>>>>  remote/read/write/setuid/devices/vers=4/xattr
>>>>
>>>
>>> I've to correct myself - this happens also with new server and new client,
>>> for instance, at closing firefox. I've not yet found the RPC packets
>>> responsible in wireshark (there are so many). Firefox has
>>> storage.nfs_filesystem set to true, so that's not the culprit. There are
>>> a couple of bug reports, also at illumos from 7 years ago:
>>> <https://www.illumos.org/issues/7434> ,
>>> I see badcalls for NFS client rpc, but not the xdrcalls mentioned there.
>>>
>>> Since this started just recently with the new code, some bug must
>>> have been reintroduced with that. I've also experienced a lockup of
>>> the NFS mount once with that error, unable to use it anymore, so
>>> probably that's not something rooted in firefox itself.
>>>
>>>
>>>
>>> ------------------------------------------
>>> illumos: illumos-developer
>>> Permalink:
>>> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M4fed1844efa6ec0df19e0d10
>>>
>>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>>
>>
>> --
>> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
>> https://www.imk-asf.kit.edu/english/sat.php
>> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
>> <https://www.kit.edu/>
>> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
>>
>>
>> ------------------------------------------
>> illumos: illumos-developer
>> Permalink: https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M8e2daabe8111730c5b8917c9
>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 8910 bytes --]

The capture file can be downloaded at:

<https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>

..156.70 is the client, ...156.24 the client. The errors are in
frames 7270/71 and 7346/47 .

This captures firefox in unwinding, just before shutdown. It probably hits the
open_downgrade path because of its several worker threads. Why that codepath
also had been triggered sometimes by a simple cat is a mystery to me, maybe the
pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
is the real culprit at all, of course).


On 03/04/2024 01:31, Udo Grabowski (IMK) wrote:
> On 03/04/2024 00:34, gusev.vitaliy@icloud.com wrote:
>> Do I understand right  that didn’t have this issue with old server ? If no,
>> could you check that?
>>
>
> It never occurres between 151a9 server and client connection, but
> also happens between illumos-.... (starting with the 4.1-2 feature)
> clients and 151a9 servers. So it's certainly a client problem in the
> new code, probably around that open_downgrade call.
>
>> Also can you send me pcap?
>>
>
> I'll upload it tomorrow (about 10h from now) to somewhere.
>
>
>> —
>> Vitaliy
>>
>>> On 2 Apr 2024, at 18:09, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>>>
>>> Got'em !
>>>
>>> (mounted also with vers=4.0 instead of vers=4, but no change)
>>>
>>> New server, new client (illumos-71b3c7bf69)
>>>
>>> On Firefox close this happens two times (also got similar stuff two times
>>> just with a simple cat, didn't catch it):
>>>
>>> Client Call:
>>> ============
>>> Frame 7270: 254 bytes on wire (2032 bits), 254 bytes captured (2032 bits)
>>> Ethernet II, Src: 00:00:5a:71:78:7c (00:00:5a:71:78:7c), Dst:
>>> 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2)
>>> Internet Protocol Version 4, Src: 172.23.156.70, Dst: 172.23.156.24
>>> Transmission Control Protocol, Src Port: 1005 (1005), Dst Port: 2049 (2049),
>>> Seq: 144845, Ack: 9997, Len: 188
>>> Remote Procedure Call, Type:Call XID:0xda85d831
>>>    Fragment header: Last fragment, 184 bytes
>>>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>>>        .000 0000 0000 0000 0000 0000 1011 1000 = Fragment Length: 184
>>>    XID: 0xda85d831 (3666204721)
>>>    Message Type: Call (0)
>>>    RPC Version: 2
>>>    Program: NFS (100003)
>>>    Program Version: 4
>>>    Procedure: COMPOUND (1)
>>>    [The reply to this request is in frame 7271]
>>>    Credentials
>>>        Flavor: AUTH_UNIX (1)
>>>        Length: 32
>>>        Stamp: 0x660c13de
>>>        Machine Name: imksunts
>>>            length: 8
>>>            contents: imksunts
>>>        UID: 1001
>>>        GID: 360
>>>        Auxiliary GIDs (1) [360]
>>>            GID: 360
>>>    Verifier
>>>        Flavor: AUTH_NULL (0)
>>>        Length: 0
>>> Network File System, Ops(3): PUTFH, GETATTR, OPEN_DOWNGRADE
>>>    [Program Version: 4]
>>>    [V4 Procedure: COMPOUND (1)]
>>>    Tag: open dgrade
>>>    minorversion: 0
>>>    Operations (count: 3): PUTFH, GETATTR, OPEN_DOWNGRADE
>>> [Malformed Packet: NFS]
>>>    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
>>>        [Malformed Packet (Exception occurred)]
>>>        [Severity level: Error]
>>>        [Group: Malformed]
>>>
>>> == Hexdump
>>> 0000   00 1b 21 c0 07 e2 00 00 5a 71 78 7c 08 00 45 00
>>> 0010   00 f0 78 b7 40 00 40 06 30 c3 ac 17 9c 46 ac 17
>>> 0020   9c 18 03 ed 08 01 fa 31 39 7d 50 81 51 16 80 18
>>> 0030   80 26 13 76 00 00 01 01 08 0a 00 03 49 f7 00 89
>>> 0040   12 87 80 00 00 b8 da 85 d8 31 00 00 00 00 00 00
>>> 0050   00 02 00 01 86 a3 00 00 00 04 00 00 00 01 00 00
>>> 0060   00 01 00 00 00 20 66 0c 13 de 00 00 00 08 69 6d
>>> 0070   6b 73 75 6e 74 73 00 00 03 e9 00 00 01 68 00 00
>>> 0080   00 01 00 00 01 68 00 00 00 00 00 00 00 00 00 00
>>> 0090   00 0c 6f 70 65 6e 20 64 67 72 61 64 65 20 00 00
>>> 00a0   00 00 00 00 00 03 00 00 00 16 00 00 00 24 f3 73
>>> 00b0   85 d4 08 ad 4a fb 0a 00 12 0e 00 00 00 00 b9 c0
>>> 00c0   41 00 0a 00 22 00 00 00 00 00 eb bf 41 00 00 00
>>> 00d0   00 00 00 00 00 09 00 00 00 02 00 10 01 1a 00 b0
>>> 00e0   a2 3a 00 00 00 15 00 00 00 01 b2 fa 0b 66 00 f0
>>> 00f0   54 00 00 00 00 00 00 00 02 5a 00 00 00 00
>>>
>>>
>>> Server Answer:
>>> ==============
>>> Frame 7271: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
>>> Ethernet II, Src: 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2), Dst:
>>> 00:00:5a:71:78:7c (00:00:5a:71:78:7c)
>>> Internet Protocol Version 4, Src: 172.23.156.24, Dst: 172.23.156.70
>>> Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 1005 (1005),
>>> Seq: 9997, Ack: 145033, Len: 28
>>> Remote Procedure Call, Type:Reply XID:0xda85d831
>>>    Fragment header: Last fragment, 24 bytes
>>>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>>>        .000 0000 0000 0000 0000 0000 0001 1000 = Fragment Length: 24
>>>    XID: 0xda85d831 (3666204721)
>>>    Message Type: Reply (1)
>>>    [Program: NFS (100003)]
>>>    [Program Version: 4]
>>>    [Procedure: COMPOUND (1)]
>>>    Reply State: accepted (0)
>>>    [This is a reply to a request in frame 7270]
>>>    [Time from request: 0.000245000 seconds]
>>>    Verifier
>>>        Flavor: AUTH_NULL (0)
>>>        Length: 0
>>>    Accept State: procedure can't decode params (4)
>>>
>>> == Hexdump
>>> 0000   00 00 5a 71 78 7c 00 1b 21 c0 07 e2 08 00 45 00
>>> 0010   00 50 a7 60 40 00 40 06 02 ba ac 17 9c 18 ac 17
>>> 0020   9c 46 08 01 03 ed 50 81 51 16 fa 31 3a 39 80 18
>>> 0030   80 26 f4 12 00 00 01 01 08 0a 00 89 12 89 00 03
>>> 0040   49 f7 80 00 00 18 da 85 d8 31 00 00 00 01 00 00
>>> 0050   00 00 00 00 00 00 00 00 00 00 00 00 00 04
>>>
>>> NFS compound failed for server imksunth9: error 11 (RPC: Server can't decode
>>> arguments)
>>>
>>>
>>> On 28/03/2024 17:48, Udo Grabowski (IMK) wrote:
>>>> On 01/03/2024 10:44, Udo Grabowski (IMK) wrote:
>>>>> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>>>>>> NFSv4.1+ change was integrated yesterday:
>>>>>>
>>>>>>    15405 Port NFSv41 base
>>>>>>
>>>>>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> So please try it on and report results!
>>>>>>
>>>>>
>>>>> With an old server, but a new client with that code, I get occasional compound
>>>>> errors:
>>>>>
>>>>> On client (illumos-8b0687e22a):
>>>>>
>>>>> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
>>>>> arguments)
>>>>>
>>>>> On server (oi_151a9):
>>>>>
>>>>> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
>>>>> RFS4_COMPOUND client
>>>>>
>>>>> client mount parameters:
>>>>>
>>>>> -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>>>>>
>>>>>  remote/read/write/setuid/devices/vers=4/xattr
>>>>>
>>>>
>>>> I've to correct myself - this happens also with new server and new client,
>>>> for instance, at closing firefox. I've not yet found the RPC packets
>>>> responsible in wireshark (there are so many). Firefox has
>>>> storage.nfs_filesystem set to true, so that's not the culprit. There are
>>>> a couple of bug reports, also at illumos from 7 years ago:
>>>> <https://www.illumos.org/issues/7434> ,
>>>> I see badcalls for NFS client rpc, but not the xdrcalls mentioned there.
>>>>
>>>> Since this started just recently with the new code, some bug must
>>>> have been reintroduced with that. I've also experienced a lockup of
>>>> the NFS mount once with that error, unable to use it anymore, so
>>>> probably that's not something rooted in firefox itself.
>>>>
>>>>
>>>>
>>>> ------------------------------------------
>>>> illumos: illumos-developer
>>>> Permalink:
>>>> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M4fed1844efa6ec0df19e0d10
>>>>
>>>>
>>>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>>>
>>>
>>> --
>>> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
>>> https://www.imk-asf.kit.edu/english/sat.php
>>> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
>>> <https://www.kit.edu/>
>>> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
>>>
>>>
>>> ------------------------------------------
>>> illumos: illumos-developer
>>> Permalink:
>>> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M8e2daabe8111730c5b8917c9
>>>
>>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>>
>
>


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
> The capture file can be downloaded at:
>
> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>
> ..156.70 is the client, ...156.24 the client.

..156.70 is the client, ...156.24 the server.
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 10026 bytes --]

On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
> The capture file can be downloaded at:
>
> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>
> ..156.70 is the client, ...156.24 the client. The errors are in
> frames 7270/71 and 7346/47 .
>
> This captures firefox in unwinding, just before shutdown. It probably hits the
> open_downgrade path because of its several worker threads. Why that codepath
> also had been triggered sometimes by a simple cat is a mystery to me, maybe the
> pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
> is the real culprit at all, of course).
>

Reading 18.18.3 of the NFS 4 RFC 5661 is says:


   "Valid values for the expression (share_access &
    ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ,
    OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH.  If the client
    specifies other values, the server MUST reply with NFS4ERR_INVAL."


But this open_downgrade request has:

    share_access: OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE (0)

Violation of the standard ?


Note also that the server has delegations switched off.

>
> On 03/04/2024 01:31, Udo Grabowski (IMK) wrote:
>> On 03/04/2024 00:34, gusev.vitaliy@icloud.com wrote:
>>> Do I understand right  that didn’t have this issue with old server ? If no,
>>> could you check that?
>>>
>>
>> It never occurres between 151a9 server and client connection, but
>> also happens between illumos-.... (starting with the 4.1-2 feature)
>> clients and 151a9 servers. So it's certainly a client problem in the
>> new code, probably around that open_downgrade call.
>>
>>> Also can you send me pcap?
>>>
>>
>> I'll upload it tomorrow (about 10h from now) to somewhere.
>>
>>
>>> —
>>> Vitaliy
>>>
>>>> On 2 Apr 2024, at 18:09, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>>>>
>>>> Got'em !
>>>>
>>>> (mounted also with vers=4.0 instead of vers=4, but no change)
>>>>
>>>> New server, new client (illumos-71b3c7bf69)
>>>>
>>>> On Firefox close this happens two times (also got similar stuff two times
>>>> just with a simple cat, didn't catch it):
>>>>
>>>> Client Call:
>>>> ============
>>>> Frame 7270: 254 bytes on wire (2032 bits), 254 bytes captured (2032 bits)
>>>> Ethernet II, Src: 00:00:5a:71:78:7c (00:00:5a:71:78:7c), Dst:
>>>> 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2)
>>>> Internet Protocol Version 4, Src: 172.23.156.70, Dst: 172.23.156.24
>>>> Transmission Control Protocol, Src Port: 1005 (1005), Dst Port: 2049 (2049),
>>>> Seq: 144845, Ack: 9997, Len: 188
>>>> Remote Procedure Call, Type:Call XID:0xda85d831
>>>>    Fragment header: Last fragment, 184 bytes
>>>>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>>>>        .000 0000 0000 0000 0000 0000 1011 1000 = Fragment Length: 184
>>>>    XID: 0xda85d831 (3666204721)
>>>>    Message Type: Call (0)
>>>>    RPC Version: 2
>>>>    Program: NFS (100003)
>>>>    Program Version: 4
>>>>    Procedure: COMPOUND (1)
>>>>    [The reply to this request is in frame 7271]
>>>>    Credentials
>>>>        Flavor: AUTH_UNIX (1)
>>>>        Length: 32
>>>>        Stamp: 0x660c13de
>>>>        Machine Name: imksunts
>>>>            length: 8
>>>>            contents: imksunts
>>>>        UID: 1001
>>>>        GID: 360
>>>>        Auxiliary GIDs (1) [360]
>>>>            GID: 360
>>>>    Verifier
>>>>        Flavor: AUTH_NULL (0)
>>>>        Length: 0
>>>> Network File System, Ops(3): PUTFH, GETATTR, OPEN_DOWNGRADE
>>>>    [Program Version: 4]
>>>>    [V4 Procedure: COMPOUND (1)]
>>>>    Tag: open dgrade
>>>>    minorversion: 0
>>>>    Operations (count: 3): PUTFH, GETATTR, OPEN_DOWNGRADE
>>>> [Malformed Packet: NFS]
>>>>    [Expert Info (Error/Malformed): Malformed Packet (Exception occurred)]
>>>>        [Malformed Packet (Exception occurred)]
>>>>        [Severity level: Error]
>>>>        [Group: Malformed]
>>>>
>>>> == Hexdump
>>>> 0000   00 1b 21 c0 07 e2 00 00 5a 71 78 7c 08 00 45 00
>>>> 0010   00 f0 78 b7 40 00 40 06 30 c3 ac 17 9c 46 ac 17
>>>> 0020   9c 18 03 ed 08 01 fa 31 39 7d 50 81 51 16 80 18
>>>> 0030   80 26 13 76 00 00 01 01 08 0a 00 03 49 f7 00 89
>>>> 0040   12 87 80 00 00 b8 da 85 d8 31 00 00 00 00 00 00
>>>> 0050   00 02 00 01 86 a3 00 00 00 04 00 00 00 01 00 00
>>>> 0060   00 01 00 00 00 20 66 0c 13 de 00 00 00 08 69 6d
>>>> 0070   6b 73 75 6e 74 73 00 00 03 e9 00 00 01 68 00 00
>>>> 0080   00 01 00 00 01 68 00 00 00 00 00 00 00 00 00 00
>>>> 0090   00 0c 6f 70 65 6e 20 64 67 72 61 64 65 20 00 00
>>>> 00a0   00 00 00 00 00 03 00 00 00 16 00 00 00 24 f3 73
>>>> 00b0   85 d4 08 ad 4a fb 0a 00 12 0e 00 00 00 00 b9 c0
>>>> 00c0   41 00 0a 00 22 00 00 00 00 00 eb bf 41 00 00 00
>>>> 00d0   00 00 00 00 00 09 00 00 00 02 00 10 01 1a 00 b0
>>>> 00e0   a2 3a 00 00 00 15 00 00 00 01 b2 fa 0b 66 00 f0
>>>> 00f0   54 00 00 00 00 00 00 00 02 5a 00 00 00 00
>>>>
>>>>
>>>> Server Answer:
>>>> ==============
>>>> Frame 7271: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
>>>> Ethernet II, Src: 00:1b:21:c0:07:e2 (00:1b:21:c0:07:e2), Dst:
>>>> 00:00:5a:71:78:7c (00:00:5a:71:78:7c)
>>>> Internet Protocol Version 4, Src: 172.23.156.24, Dst: 172.23.156.70
>>>> Transmission Control Protocol, Src Port: 2049 (2049), Dst Port: 1005 (1005),
>>>> Seq: 9997, Ack: 145033, Len: 28
>>>> Remote Procedure Call, Type:Reply XID:0xda85d831
>>>>    Fragment header: Last fragment, 24 bytes
>>>>        1... .... .... .... .... .... .... .... = Last Fragment: Yes
>>>>        .000 0000 0000 0000 0000 0000 0001 1000 = Fragment Length: 24
>>>>    XID: 0xda85d831 (3666204721)
>>>>    Message Type: Reply (1)
>>>>    [Program: NFS (100003)]
>>>>    [Program Version: 4]
>>>>    [Procedure: COMPOUND (1)]
>>>>    Reply State: accepted (0)
>>>>    [This is a reply to a request in frame 7270]
>>>>    [Time from request: 0.000245000 seconds]
>>>>    Verifier
>>>>        Flavor: AUTH_NULL (0)
>>>>        Length: 0
>>>>    Accept State: procedure can't decode params (4)
>>>>
>>>> == Hexdump
>>>> 0000   00 00 5a 71 78 7c 00 1b 21 c0 07 e2 08 00 45 00
>>>> 0010   00 50 a7 60 40 00 40 06 02 ba ac 17 9c 18 ac 17
>>>> 0020   9c 46 08 01 03 ed 50 81 51 16 fa 31 3a 39 80 18
>>>> 0030   80 26 f4 12 00 00 01 01 08 0a 00 89 12 89 00 03
>>>> 0040   49 f7 80 00 00 18 da 85 d8 31 00 00 00 01 00 00
>>>> 0050   00 00 00 00 00 00 00 00 00 00 00 00 00 04
>>>>
>>>> NFS compound failed for server imksunth9: error 11 (RPC: Server can't decode
>>>> arguments)
>>>>
>>>>
>>>> On 28/03/2024 17:48, Udo Grabowski (IMK) wrote:
>>>>> On 01/03/2024 10:44, Udo Grabowski (IMK) wrote:
>>>>>> On 08/02/2024 11:28, gusev.vitaliy via illumos-developer wrote:
>>>>>>> NFSv4.1+ change was integrated yesterday:
>>>>>>>
>>>>>>>    15405 Port NFSv41 base
>>>>>>>
>>>>>>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So please try it on and report results!
>>>>>>>
>>>>>>
>>>>>> With an old server, but a new client with that code, I get occasional
>>>>>> compound
>>>>>> errors:
>>>>>>
>>>>>> On client (illumos-8b0687e22a):
>>>>>>
>>>>>> NFS compound failed for server imksunth1: error 11 (RPC: Server can't decode
>>>>>> arguments)
>>>>>>
>>>>>> On server (oi_151a9):
>>>>>>
>>>>>> [kern.notice] NOTICE: Failed to decode arguments for NFS version 4 procedure
>>>>>> RFS4_COMPOUND client
>>>>>>
>>>>>> client mount parameters:
>>>>>>
>>>>>> -proto=tcp,rw,intr,noquota,actimeo=1,bg,retry=3,vers=4
>>>>>>
>>>>>>  remote/read/write/setuid/devices/vers=4/xattr
>>>>>>
>>>>>
>>>>> I've to correct myself - this happens also with new server and new client,
>>>>> for instance, at closing firefox. I've not yet found the RPC packets
>>>>> responsible in wireshark (there are so many). Firefox has
>>>>> storage.nfs_filesystem set to true, so that's not the culprit. There are
>>>>> a couple of bug reports, also at illumos from 7 years ago:
>>>>> <https://www.illumos.org/issues/7434> ,
>>>>> I see badcalls for NFS client rpc, but not the xdrcalls mentioned there.
>>>>>
>>>>> Since this started just recently with the new code, some bug must
>>>>> have been reintroduced with that. I've also experienced a lockup of
>>>>> the NFS mount once with that error, unable to use it anymore, so
>>>>> probably that's not something rooted in firefox itself.
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------
>>>>> illumos: illumos-developer
>>>>> Permalink:
>>>>> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M4fed1844efa6ec0df19e0d10
>>>>>
>>>>>
>>>>>
>>>>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>>>>
>>>>
>>>> --
>>>> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
>>>> https://www.imk-asf.kit.edu/english/sat.php
>>>> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
>>>> <https://www.kit.edu/>
>>>> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
>>>>
>>>>
>>>> ------------------------------------------
>>>> illumos: illumos-developer
>>>> Permalink:
>>>> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-M8e2daabe8111730c5b8917c9
>>>>
>>>>
>>>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>>>
>>
>>
>
>
>
>
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink:
> https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-Mb7f4f73cf4763d8a643d5c59
>
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]

On 03/04/2024 16:10, Udo Grabowski (IMK) wrote:
> On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
>> The capture file can be downloaded at:
>>
>> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>>
>> ..156.70 is the client, ...156.24 the client. The errors are in
>> frames 7270/71 and 7346/47 .
>>
>> This captures firefox in unwinding, just before shutdown. It probably hits the
>> open_downgrade path because of its several worker threads. Why that codepath
>> also had been triggered sometimes by a simple cat is a mystery to me, maybe the
>> pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
>> is the real culprit at all, of course).
>>
>
> Reading 18.18.3 of the NFS 4 RFC 5661 is says:
>
>
>   "Valid values for the expression (share_access &
>    ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ,
>    OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH.  If the client
>    specifies other values, the server MUST reply with NFS4ERR_INVAL."
>
>
> But this open_downgrade request has:
>
>    share_access: OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE (0)
>
> Violation of the standard ?
>

No, that NFS4.1 (and 8881 says the same), RFC 7530 for 4.0 only says:

    " The
    share_access and share_deny bits specified must be exactly equal to
    the union of the share_access and share_deny bits specified for some
    subset of the OPENs in effect for the current open-owner on the
    current file.  If that constraint is not respected, the error
    NFS4ERR_INVAL should be returned. "

But that does not mention "OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE" anywhere ?

But then the server should spit a NFS4ERR_INVAL ? And wireshark should
not see a malformed NFS packet.
-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Toomas Soome]

> On 3. Apr 2024, at 17:40, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
> 
> On 03/04/2024 16:10, Udo Grabowski (IMK) wrote:
>> On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
>>> The capture file can be downloaded at:
>>> 
>>> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>>> 
>>> ..156.70 is the client, ...156.24 the client. The errors are in
>>> frames 7270/71 and 7346/47 .
>>> 
>>> This captures firefox in unwinding, just before shutdown. It probably hits the
>>> open_downgrade path because of its several worker threads. Why that codepath
>>> also had been triggered sometimes by a simple cat is a mystery to me, maybe the
>>> pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
>>> is the real culprit at all, of course).
>>> 
>> 
>> Reading 18.18.3 of the NFS 4 RFC 5661 is says:
>> 
>> 
>>  "Valid values for the expression (share_access &
>>   ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ,
>>   OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH.  If the client
>>   specifies other values, the server MUST reply with NFS4ERR_INVAL."
>> 
>> 
>> But this open_downgrade request has:
>> 
>>   share_access: OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE (0)
>> 
>> Violation of the standard ?
>> 
> 
> No, that NFS4.1 (and 8881 says the same), RFC 7530 for 4.0 only says:
> 
>   " The
>   share_access and share_deny bits specified must be exactly equal to
>   the union of the share_access and share_deny bits specified for some
>   subset of the OPENs in effect for the current open-owner on the
>   current file.  If that constraint is not respected, the error
>   NFS4ERR_INVAL should be returned. "
> 
> But that does not mention "OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE" anywhere ?

because it was added to 4.1. 

> 
> But then the server should spit a NFS4ERR_INVAL ? And wireshark should
> not see a malformed NFS packet.

does it complain when mounted as 4.1? Of course, with 4.0 we should get NFS4ERR_INVAL, but since the  is not legal 4.0 flag, it is only natural that we do get complaint from wireshark.

rgds,
toomas

> -- 
> Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
> https://www.imk-asf.kit.edu/english/sat.php
> KIT - Karlsruhe Institute of Technology          https://www.kit.edu
> Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026
> 
> 
> ------------------------------------------
> illumos: illumos-developer
> Permalink: https://illumos.topicbox.com/groups/developer/T92046956b5fe6009-Mcae9ca445f3257422aaa1008
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 2627 bytes --]

On 03/04/2024 17:24, Toomas Soome via illumos-developer wrote:
>
>
>> On 3. Apr 2024, at 17:40, Udo Grabowski (IMK) <udo.grabowski@kit.edu> wrote:
>>
>> On 03/04/2024 16:10, Udo Grabowski (IMK) wrote:
>>> On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
>>>> The capture file can be downloaded at:
>>>>
>>>> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>>>>
>>>> ..156.70 is the client, ...156.24 the client. The errors are in
>>>> frames 7270/71 and 7346/47 .
>>>>
>>>> This captures firefox in unwinding, just before shutdown. It probably hits the
>>>> open_downgrade path because of its several worker threads. Why that codepath
>>>> also had been triggered sometimes by a simple cat is a mystery to me, maybe the
>>>> pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
>>>> is the real culprit at all, of course).
>>>>
>>>
>>> Reading 18.18.3 of the NFS 4 RFC 5661 is says:
>>>
>>>
>>>  "Valid values for the expression (share_access &
>>>   ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ,
>>>   OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH.  If the client
>>>   specifies other values, the server MUST reply with NFS4ERR_INVAL."
>>>
>>>
>>> But this open_downgrade request has:
>>>
>>>   share_access: OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE (0)
>>>
>>> Violation of the standard ?
>>>
>>
>> No, that NFS4.1 (and 8881 says the same), RFC 7530 for 4.0 only says:
>>
>>   " The
>>   share_access and share_deny bits specified must be exactly equal to
>>   the union of the share_access and share_deny bits specified for some
>>   subset of the OPENs in effect for the current open-owner on the
>>   current file.  If that constraint is not respected, the error
>>   NFS4ERR_INVAL should be returned. "
>>
>> But that does not mention "OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE" anywhere ?
>
> because it was added to 4.1.
>
>>
>> But then the server should spit a NFS4ERR_INVAL ? And wireshark should
>> not see a malformed NFS packet.
>
> does it complain when mounted as 4.1? Of course, with 4.0 we should get NFS4ERR_INVAL, but since the  is not legal 4.0 flag, it is only natural that we do get complaint from wireshark.
>

Exactly same behaviour with 4.1. The share_deny argument is missing, too,
and share_access is still 0.

-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

On 03/04/2024 16:40, Udo Grabowski (IMK) wrote:
> On 03/04/2024 16:10, Udo Grabowski (IMK) wrote:
>> On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
>>> The capture file can be downloaded at:
>>>
>>> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>>>
>>> ..156.70 is the client, ...156.24 the client. The errors are in
>>> frames 7270/71 and 7346/47 .
>>>
>>> This captures firefox in unwinding, just before shutdown. It probably hits the
>>> open_downgrade path because of its several worker threads. Why that codepath
>>> also had been triggered sometimes by a simple cat is a mystery to me, maybe the
>>> pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
>>> is the real culprit at all, of course).
>>>
>>
>> Reading 18.18.3 of the NFS 4 RFC 5661 is says:
>>
>>
>>   "Valid values for the expression (share_access &
>>    ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ,
>>    OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH.  If the client
>>    specifies other values, the server MUST reply with NFS4ERR_INVAL."
>>
>>
>> But this open_downgrade request has:
>>
>>    share_access: OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE (0)
>>
>> Violation of the standard ?
>>
>
> No, that NFS4.1 (and 8881 says the same), RFC 7530 for 4.0 only says:
>
>    " The
>    share_access and share_deny bits specified must be exactly equal to
>    the union of the share_access and share_deny bits specified for some
>    subset of the OPENs in effect for the current open-owner on the
>    current file.  If that constraint is not respected, the error
>    NFS4ERR_INVAL should be returned. "
>
> But that does not mention "OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE" anywhere ?
>
> But then the server should spit a NFS4ERR_INVAL ? And wireshark should
> not see a malformed NFS packet


I checked against an old 151a8 server client mount. There the open_downgrade
has a share_access AND share_deny ! Its missing in the new NFS code capture.

So a missing argument to the opcode ? Or missing due to missing byte swap ?

-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Udo Grabowski (IMK)]

[-- Attachment #1: Type: text/plain, Size: 2638 bytes --]

On 03/04/2024 17:35, Udo Grabowski (IMK) wrote:
> On 03/04/2024 16:40, Udo Grabowski (IMK) wrote:
>> On 03/04/2024 16:10, Udo Grabowski (IMK) wrote:
>>> On 03/04/2024 10:21, Udo Grabowski (IMK) wrote:
>>>> The capture file can be downloaded at:
>>>>
>>>> <https://imk-asf-mipas.imk.kit.edu/users/grabow/dump_nfs_error.pcap>
>>>>
>>>> ..156.70 is the client, ...156.24 the client. The errors are in
>>>> frames 7270/71 and 7346/47 .
>>>>
>>>> This captures firefox in unwinding, just before shutdown. It probably hits the
>>>> open_downgrade path because of its several worker threads. Why that codepath
>>>> also had been triggered sometimes by a simple cat is a mystery to me, maybe the
>>>> pathhandle was already corrupted by a firefox shutdown (if the open_downgrade
>>>> is the real culprit at all, of course).
>>>>
>>>
>>> Reading 18.18.3 of the NFS 4 RFC 5661 is says:
>>>
>>>
>>>   "Valid values for the expression (share_access &
>>>    ~OPEN4_SHARE_ACCESS_WANT_DELEG_MASK) are OPEN4_SHARE_ACCESS_READ,
>>>    OPEN4_SHARE_ACCESS_WRITE, or OPEN4_SHARE_ACCESS_BOTH.  If the client
>>>    specifies other values, the server MUST reply with NFS4ERR_INVAL."
>>>
>>>
>>> But this open_downgrade request has:
>>>
>>>    share_access: OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE (0)
>>>
>>> Violation of the standard ?
>>>
>>
>> No, that NFS4.1 (and 8881 says the same), RFC 7530 for 4.0 only says:
>>
>>    " The
>>    share_access and share_deny bits specified must be exactly equal to
>>    the union of the share_access and share_deny bits specified for some
>>    subset of the OPENs in effect for the current open-owner on the
>>    current file.  If that constraint is not respected, the error
>>    NFS4ERR_INVAL should be returned. "
>>
>> But that does not mention "OPEN4_SHARE_ACCESS_WANT_NO_PREFERENCE" anywhere ?
>>
>> But then the server should spit a NFS4ERR_INVAL ? And wireshark should
>> not see a malformed NFS packet
>
>
> I checked against an old 151a8 server client mount. There the open_downgrade
> has a share_access AND share_deny ! Its missing in the new NFS code capture.
>
> So a missing argument to the opcode ? Or missing due to missing byte swap ?
>

Which would nicely fit the RPC error message:

NFS compound failed for server imksunth9: error 11 (RPC: Server can't decode 
arguments)


-- 
Dr.Udo Grabowski  Inst.of Meteorology & Climate Research IMK-ASF-SAT
https://www.imk-asf.kit.edu/english/sat.php
KIT - Karlsruhe Institute of Technology          https://www.kit.edu
Postfach 3640,76021 Karlsruhe,Germany T:(+49)721 608-26026 F:-926026


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5804 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Bill Sommerfeld]

On 4/3/24 08:56, Udo Grabowski (IMK) wrote:
>> I checked against an old 151a8 server client mount. There the 
>> open_downgrade
>> has a share_access AND share_deny ! Its missing in the new NFS code 
>> capture.
>>
>> So a missing argument to the opcode ? Or missing due to missing byte 
>> swap ?
>>
> 
> Which would nicely fit the RPC error message:
> 
> NFS compound failed for server imksunth9: error 11 (RPC: Server can't 
> decode arguments)

I think I've found the bug.

usr/src/uts/common/fs/nfs/nfs4_xdr.c xdr_share_access() appears to only 
handle XDR_DECODE and is a no-op for encode.   On XDR_ENCODE, it should 
form the integer access mask and then do an xdr_u_int() with the value.

This function was introduced in 44e1126d9eae71c48c5d1de51e24750c6ec20a4 
("15405 Port NFSv41 base"); prior to that point, everything that passed 
a share_access mask would just call xdr_u_int()

					- Bill

Re: [developer] Status of NFSv4.1 support in Illumos?

[Bill Sommerfeld]

On 4/3/24 10:22, Bill Sommerfeld wrote:
> On 4/3/24 08:56, Udo Grabowski (IMK) wrote:
>>> I checked against an old 151a8 server client mount. There the 
>>> open_downgrade
>>> has a share_access AND share_deny ! Its missing in the new NFS code 
>>> capture.
>>>
>>> So a missing argument to the opcode ? Or missing due to missing byte 
>>> swap ?
>>>
>>
>> Which would nicely fit the RPC error message:
>>
>> NFS compound failed for server imksunth9: error 11 (RPC: Server can't 
>> decode arguments)
> 
> I think I've found the bug.
> 
> usr/src/uts/common/fs/nfs/nfs4_xdr.c xdr_share_access() appears to only 
> handle XDR_DECODE and is a no-op for encode.   On XDR_ENCODE, it should 
> form the integer access mask and then do an xdr_u_int() with the value.
> 
> This function was introduced in 44e1126d9eae71c48c5d1de51e24750c6ec20a4 
> ("15405 Port NFSv41 base"); prior to that point, everything that passed 
> a share_access mask would just call xdr_u_int()

I filed https://www.illumos.org/issues/16438

Re: [developer] Status of NFSv4.1 support in Illumos?

[Dan Shelton]

On Thu, 8 Feb 2024 at 11:29, <gusev.vitaliy@icloud.com> wrote:
>
> NFSv4.1+ change was integrated yesterday:
>
> 15405 Port NFSv41 base
> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>
>
> So please try it on and report results!

I need an ISO which contains the fix!

Also, can I use uname -a to identify whether the kernel has that fix?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: [developer] Status of NFSv4.1 support in Illumos?

[Dan McDonald]

On Mar 3, 2024, at 10:36 PM, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> 
> On Thu, 8 Feb 2024 at 11:29, <gusev.vitaliy@icloud.com> wrote:
>> 
>> NFSv4.1+ change was integrated yesterday:
>> 
>> 15405 Port NFSv41 base
>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>> 
>> 
>> So please try it on and report results!
> 
> I need an ISO which contains the fix!

Which distro?

> Also, can I use uname -a to identify whether the kernel has that fix?

Same question back:  Which distro?

Dan

Re: [developer] Status of NFSv4.1 support in Illumos?

[Dan Shelton]

On Mon, 4 Mar 2024 at 04:58, Dan McDonald <danmcd@mnx.io> wrote:
>
> On Mar 3, 2024, at 10:36 PM, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> >
> > On Thu, 8 Feb 2024 at 11:29, <gusev.vitaliy@icloud.com> wrote:
> >>
> >> NFSv4.1+ change was integrated yesterday:
> >>
> >> 15405 Port NFSv41 base
> >> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
> >>
> >>
> >> So please try it on and report results!
> >
> > I need an ISO which contains the fix!
>
> Which distro?
>
> > Also, can I use uname -a to identify whether the kernel has that fix?
>
> Same question back:  Which distro?

OpenIndiana, smartos?

Dan
-- 
Dan Shelton - Cluster Specialist Win/Lin/Bsd

Re: [developer] Status of NFSv4.1 support in Illumos?

[Dan McDonald]

On Mar 10, 2024, at 11:06 PM, Dan Shelton <dan.f.shelton@gmail.com> wrote:
> 
>> 
>> Same question back:  Which distro?
> 
> OpenIndiana, smartos?

Can't speak for OI, but for SmartOS, any release starting with 20240222 has the reactivated (but not on by default) NFSv4.1.

See here:  https://smartos.topicbox.com/groups/smartos-discuss/Te915a68fbd2ee2c5-Mbad0c6d00d467760e17e4ac6/smartos-release-20240222-loki

Dan

Re: [developer] Status of NFSv4.1 support in Illumos?

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]



> On 11. Mar 2024, at 06:00, Dan McDonald <danmcd@mnx.io> wrote:
> 
> On Mar 10, 2024, at 11:06 PM, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>> 
>>> Same question back:  Which distro?
>> 
>> OpenIndiana, smartos?
> 
> Can't speak for OI, but for SmartOS, any release starting with 20240222 has the reactivated (but not on by default) NFSv4.1.
> 
> See here:  https://smartos.topicbox.com/groups/smartos-discuss/Te915a68fbd2ee2c5-Mbad0c6d00d467760e17e4ac6/smartos-release-20240222-loki
> 
> Dan
> 

AFAIK, OI pkg repository updates should happen as the changes are popping up on illumos-gate, so the state should be the same - 4.1 base is there, but configuration defaults to 4.0. And this is because the initial code was missing backchannel support. But see also relation chain leading to https://code.illumos.org/c/illumos-gate/+/3366

rgds,
toomas

[-- Attachment #2: Type: text/html, Size: 1427 bytes --]

Re: [developer] Status of NFSv4.1 support in Illumos?

[Till Wegmueller]

On 11.03.2024 04:06, Dan Shelton wrote:
> On Mon, 4 Mar 2024 at 04:58, Dan McDonald <danmcd@mnx.io> wrote:
>>
>> On Mar 3, 2024, at 10:36 PM, Dan Shelton <dan.f.shelton@gmail.com> wrote:
>>>
>>> On Thu, 8 Feb 2024 at 11:29, <gusev.vitaliy@icloud.com> wrote:
>>>>
>>>> NFSv4.1+ change was integrated yesterday:
>>>>
>>>> 15405 Port NFSv41 base
>>>> https://github.com/illumos/illumos-gate/commit/f44e1126d9eae71c48c5d1de51e24750c6ec20a4
>>>>
>>>>
>>>> So please try it on and report results!
>>>
>>> I need an ISO which contains the fix!
>>
>> Which distro?
>>
>>> Also, can I use uname -a to identify whether the kernel has that fix?
>>
>> Same question back:  Which distro?
> 
> OpenIndiana, smartos?

uname -a reports the git commit of illumos-gate. You can check in git 
log if you are newer. illumos-gate is built and delivered every night.

-Till