Re: [discuss] shutdown hang with zpools on ISCSI devices

["Joshua M. Clulow" <josh@sysmgr.org>]

On Mon, 5 Aug 2024 at 13:50, Carsten Grzemba via illumos-discuss
<discuss@lists.illumos.org> wrote:
> I have a problem shutting down a server with Zpool on iscsi devices. The shutdown gets stuck. In the console log I see the message:
> WARNING: Pool 'zones' has encountered an uncorrectable I/O failure and has been suspended; `zpool clear` will be required before the pool can be written to.
> The message looks to me like the IP connection to the storage server was already terminated before the Zpool was unmounted.
> I forced the shutdown with an NMI reset so that I have a crash dump. In the crash dump I see that a zone on the pool in question is still in shutting_down.
> Is it possible to get information from crash dump like *svcs* of a running system?

Not really easily, no, but you can at least see the tree of processes
that is running; e.g.,

    > ::ptree
    fffffffffbc93760  sched
         fffffeb1cedab008  zpool-extra
         fffffeb1c8c2c000  fsflush
         fffffeb1c8c2f020  pageout
         fffffeb1c8c34018  init
              fffffeb38ccb4020  screen
              fffffeb5aaddd020  bash
              fffffeb1f123e010  init
              fffffeb3a3070000  image-builder
              fffffeb263501020  pkg
              fffffeb1cea71018  sac
                   fffffeb1ca8a7018  ttymon
    ...

> How can I see in the crash dump whether the IP interfaces are still working? Normally the SMF
> svc:/network/iscsi/initiator:default
> should wait for the zones to shutdown and reach the state installed.

I'm not sure that it does do that.  Looking at the manifest, I don't
see any interaction with zones at all:

    https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/iscsid/iscsi-initiator.xml

There is some special handling in the shutdown code today for
unmounting file systems that are listed in vfstab(5) as being mounted
from iSCSI disks.  That obviously doesn't help you with ZFS pools,
though, and I expect the real bug here is that the "umount_iscsi()"
function in that method script doesn't do anything to export ZFS pools
that live on iSCSI disks:

    https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/iscsid/iscsi-initiator#L172-L199

How do you _import_ these pools today?  It seems likely that pools
that come from iSCSI devices should actually not appear in the global
zpool cache file (i.e., "/etc/zfs/zpool.cache"), but rather should get
imported transiently somehow by the iSCSI initiator service in
"mount_iscsi()".  Then they should get exported on the way to
shutdown.  In short: importing and exporting those pools should
probably work like vfstab(5) entries with "iscsi" set in the automount
field.  Otherwise there does not appear to be any mechanism in place
to handle the network going away as part of shutdown.


Cheers.

-- 
Joshua M. Clulow
http://blog.sysmgr.org