supervision - discussion about system services, daemon supervision, init, runlevel management, and tools such as s6 and runit
 help / color / mirror / Atom feed
From: "Laurent Bercot" <ska-supervision@skarnet.org>
To: "Joshua Ismael Haase Hernández" <hahj87@gmail.com>,
	supervision@list.skarnet.org
Subject: Re: How to recover from s6-rc broken pipe?
Date: Wed, 16 Dec 2020 20:11:12 +0000	[thread overview]
Message-ID: <em732a4d4a-739d-4752-8a7e-f08d0894b2bc@elzian> (raw)
In-Reply-To: <CA+ypG2a_qGJtHhcS78TxQENpyPBQRhpfSDGMZETK0uqPYi7r-Q@mail.gmail.com>


>I'm using s6-rc to manage services and have been changing databases.
>
>For some unknown reason sometimes the update fails with the error:
>
>s6-rc-update: fatal: unable to read /run/s6-rc/state: Broken pipe

  That should definitely not happen.
  Have your databases been built with the same version of s6-rc as
the one you're using? (Normally they're compatible, but there has
been an incompatible change between 0.3 and 0.4)


>When that happens, I cannot use s6-rc anymore:
>
>/run/s6-rc does not exists, but s6-rc declares it as if it does:

  s6-rc-update removes the /run/s6-rc symlinks when it fails? If it
does, it's a bug that I will fix for the next release.


>s6-rc-init: fatal: unable to supervise service directories in
>/run/s6-rc/servicedirs: File exists
>
>Creating the s6-rc symlink does not improve the situation.
>
>How should I recover from this error?

  It is possible that you have a bunch of dangling symlinks in
/run/service, that were pointing to your old live directory, are not
valid anymore, but are still preventing s6-rc from doing its job.

  Generally speaking, s6-rc-update failing is bad news, because it is
difficult to do the proper cleanups (either automatically when
s6-rc-update fails, because some operations cannot be rolled back, or
manually afterwards), so yeah, your scandir may be in an ugly state
and you may need to remove all the symlinks there, delete all the
/run/s6-rc* directories and/or symlinks, and restart from scratch.
Depending on the changes your oneshots made on your system, you may
get error when running them again, too, so in the worst case, the
only good option might be to reboot. Sorry.

  But really, the original cause of the problem should not happen, and
s6-rc-update should not be failing like this. If it does not happen
all the time, is something overwriting your state file? or removing
the /run/s6-rc symlink? If your databases are compatible, then there
is definitely some external interference here.

--
  Laurent


  reply	other threads:[~2020-12-16 20:11 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-16 17:56 Joshua Ismael Haase Hernández
2020-12-16 20:11 ` Laurent Bercot [this message]
     [not found]   ` <CA+ypG2Y2ogrRxESR+r-_RkN9gUovRbk_vhbXr9DWQEd3kW0ZzQ@mail.gmail.com>
2020-12-16 21:34     ` Laurent Bercot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=em732a4d4a-739d-4752-8a7e-f08d0894b2bc@elzian \
    --to=ska-supervision@skarnet.org \
    --cc=hahj87@gmail.com \
    --cc=supervision@list.skarnet.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).