supervision - discussion about system services, daemon supervision, init, runlevel management, and tools such as s6 and runit
 help / color / mirror / Atom feed
* How to recover from s6-rc broken pipe?
@ 2020-12-16 17:56 Joshua Ismael Haase Hernández
  2020-12-16 20:11 ` Laurent Bercot
  0 siblings, 1 reply; 3+ messages in thread
From: Joshua Ismael Haase Hernández @ 2020-12-16 17:56 UTC (permalink / raw)
  To: supervision

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

I'm using s6-rc to manage services and have been changing databases.

For some unknown reason sometimes the update fails with the error:

s6-rc-update: fatal: unable to read /run/s6-rc/state: Broken pipe

When that happens, I cannot use s6-rc anymore:

/run/s6-rc does not exists, but s6-rc declares it as if it does:

sudo s6-rc-init /run/s6
s6-rc-init: fatal: unable to supervise service directories in
/run/s6-rc/servicedirs: File exists

Creating the s6-rc symlink does not improve the situation.

How should I recover from this error?

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: How to recover from s6-rc broken pipe?
  2020-12-16 17:56 How to recover from s6-rc broken pipe? Joshua Ismael Haase Hernández
@ 2020-12-16 20:11 ` Laurent Bercot
       [not found]   ` <CA+ypG2Y2ogrRxESR+r-_RkN9gUovRbk_vhbXr9DWQEd3kW0ZzQ@mail.gmail.com>
  0 siblings, 1 reply; 3+ messages in thread
From: Laurent Bercot @ 2020-12-16 20:11 UTC (permalink / raw)
  To: Joshua Ismael Haase Hernández, supervision


>I'm using s6-rc to manage services and have been changing databases.
>
>For some unknown reason sometimes the update fails with the error:
>
>s6-rc-update: fatal: unable to read /run/s6-rc/state: Broken pipe

  That should definitely not happen.
  Have your databases been built with the same version of s6-rc as
the one you're using? (Normally they're compatible, but there has
been an incompatible change between 0.3 and 0.4)


>When that happens, I cannot use s6-rc anymore:
>
>/run/s6-rc does not exists, but s6-rc declares it as if it does:

  s6-rc-update removes the /run/s6-rc symlinks when it fails? If it
does, it's a bug that I will fix for the next release.


>s6-rc-init: fatal: unable to supervise service directories in
>/run/s6-rc/servicedirs: File exists
>
>Creating the s6-rc symlink does not improve the situation.
>
>How should I recover from this error?

  It is possible that you have a bunch of dangling symlinks in
/run/service, that were pointing to your old live directory, are not
valid anymore, but are still preventing s6-rc from doing its job.

  Generally speaking, s6-rc-update failing is bad news, because it is
difficult to do the proper cleanups (either automatically when
s6-rc-update fails, because some operations cannot be rolled back, or
manually afterwards), so yeah, your scandir may be in an ugly state
and you may need to remove all the symlinks there, delete all the
/run/s6-rc* directories and/or symlinks, and restart from scratch.
Depending on the changes your oneshots made on your system, you may
get error when running them again, too, so in the worst case, the
only good option might be to reboot. Sorry.

  But really, the original cause of the problem should not happen, and
s6-rc-update should not be failing like this. If it does not happen
all the time, is something overwriting your state file? or removing
the /run/s6-rc symlink? If your databases are compatible, then there
is definitely some external interference here.

--
  Laurent


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: How to recover from s6-rc broken pipe?
       [not found]   ` <CA+ypG2Y2ogrRxESR+r-_RkN9gUovRbk_vhbXr9DWQEd3kW0ZzQ@mail.gmail.com>
@ 2020-12-16 21:34     ` Laurent Bercot
  0 siblings, 0 replies; 3+ messages in thread
From: Laurent Bercot @ 2020-12-16 21:34 UTC (permalink / raw)
  To: Joshua Ismael Haase Hernández, supervision

>     sudo rsync -avr --delete compiled/ /etc/s6-rc/compiled/
>     sudo s6-rc-update /etc/s6-rc/compiled/

  There's your problem. You're overwriting the live database; that will
throw a wrench into everything.
  s6-rc-update needs to run while the live database is still the old
one. Only after s6-rc-update has completed can you delete the old
database.

  Do something like this instead:

stamp=`echo | s6-tai64n`
s6-rc-compile /etc/s6-rc/compiled-$stamp sources...
s6-rc-update /etc/s6-rc/compiled-$stamp
old=`readlink /etc/s6-rc/compiled`
ln -sf compiled-$stamp /etc/s6-rc/compiled.new
rename /etc/s6-rc/compiled.new /etc/s6-rc/compiled
rm -rf /etc/s6-rc/$old

--
  Laurent


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-12-16 21:34 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-16 17:56 How to recover from s6-rc broken pipe? Joshua Ismael Haase Hernández
2020-12-16 20:11 ` Laurent Bercot
     [not found]   ` <CA+ypG2Y2ogrRxESR+r-_RkN9gUovRbk_vhbXr9DWQEd3kW0ZzQ@mail.gmail.com>
2020-12-16 21:34     ` Laurent Bercot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).