Re: [discuss] shutdown hang with zpools on ISCSI devices

[Carsten Grzemba <grzemba@contac-dt.de>]

[-- Attachment #1: Type: text/plain, Size: 3581 bytes --]



Am 05.08.24 23:48 schrieb "Joshua M. Clulow via illumos-discuss"  <discuss@lists.illumos.org>: 
> 
> On Mon, 5 Aug 2024 at 13:50, Carsten Grzemba via illumos-discuss
> <discuss@lists.illumos.org> wrote:
> > I have a problem shutting down a server with Zpool on iscsi devices. The shutdown gets stuck. In the console log I see the message:
> > WARNING: Pool 'zones' has encountered an uncorrectable I/O failure and has been suspended; `zpool clear` will be required before the pool can be written to.
> > The message looks to me like the IP connection to the storage server was already terminated before the Zpool was unmounted.
> > I forced the shutdown with an NMI reset so that I have a crash dump. In the crash dump I see that a zone on the pool in question is still in shutting_down.
> > Is it possible to get information from crash dump like *svcs* of a running system?
> 
> Not really easily, no, but you can at least see the tree of processes
> that is running; e.g.,
> 
>  > ::ptree
>  fffffffffbc93760 sched
>  fffffeb1cedab008 zpool-extra
>  fffffeb1c8c2c000 fsflush
>  fffffeb1c8c2f020 pageout
>  fffffeb1c8c34018 init
>  fffffeb38ccb4020 screen
>  fffffeb5aaddd020 bash
>  fffffeb1f123e010 init
>  fffffeb3a3070000 image-builder
>  fffffeb263501020 pkg
>  fffffeb1cea71018 sac
>  fffffeb1ca8a7018 ttymon
>  ...
> 
> > How can I see in the crash dump whether the IP interfaces are still working? Normally the SMF
> > svc:/network/iscsi/initiator:default
> > should wait for the zones to shutdown and reach the state installed.
> 
> I'm not sure that it does do that. Looking at the manifest, I don't
> see any interaction with zones at all:
> 
>  https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/iscsid/iscsi-initiator.xml
> 
> There is some special handling in the shutdown code today for
> unmounting file systems that are listed in vfstab(5) as being mounted
> from iSCSI disks. That obviously doesn't help you with ZFS pools,
> though, and I expect the real bug here is that the "umount_iscsi()"
> function in that method script doesn't do anything to export ZFS pools
> that live on iSCSI disks:
> 
>  https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/iscsid/iscsi-initiator#L172-L199
> 

interesting, I didn't know that "mount at boot" could have values ​​other than "yes" or "no".   
> 
> (https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/iscsid/iscsi-initiator#L172-L199 )
> 
> How do you _import_ these pools today? It seems likely that pools
> that come from iSCSI devices should actually not appear in the global
> zpool cache file (i.e., "/etc/zfs/zpool.cache"), but rather should get
> imported transiently somehow by the iSCSI initiator service in
> "mount_iscsi()". Then they should get exported on the way to
> shutdown. In short: importing and exporting those pools should
> probably work like vfstab(5) entries with "iscsi" set in the automount
> field. Otherwise there does not appear to be any mechanism in place
> to handle the network going away as part of shutdown.
> 
> 
> Cheers.
> 
> -- 
> Joshua M. Clulow
> http://blog.sysmgr.org
> 
> ------------------------------------------
> illumos: illumos-discuss
> Permalink: https://illumos.topicbox.com/groups/discuss/T494e7618fdacd18b-Mffbe5a8bca2b03942d98d534
> Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
> 
-- 
Carsten Grzemba
Tel.: +49 3677 64740
Mobil: +49 171 9749479
Email: carsten.grzemba@contac-dt.de
contac Datentechnik GmbH

[-- Attachment #2: Type: text/html, Size: 5306 bytes --]