Re: graceful restart under runit

[prj@po.cwru.edu (Paul Jarc)]

Alex Efros <powerman@powerman.asdfGroup.com> wrote:
> Yep, looks like you right. Looks like I've confused this case and case
> when process with listening socket doing fork and so result in two
> listening sockets for same ip/port in two different processes.

Now that you mention this, I think there is a way to hand off to a new
server process with no unavailability at all.  But if you can tolerate
a small outage during the switchover, you may be better off with the
simpler method of sending a signal to make the old process close its
listening socket, then forgetting the old process using Gerrit's patch
and starting a new one, which will just open a listening socket as
usual.

So, for zero-unavailability: there is a metaserver which listens on a
filesystem socket and takes care of opening other listening sockets
for other servers.  Instead of opening a listening socket directly,
another server would connect to this filesystem socket and ask the
metaserver to open it.  If there is not yet any socket open for the
requested address, the metaserver opens one, and passes the descriptor
to the requestor over the filesystem socket connection with sendmsg().
(The metaserver also keeps the listening socket open for itself, but
never calls accept() on it.)  Both sides keep the filesystem socket
connection open for as long as the requestor is accepting connections.
If the metaserver receives a request for a socket that is already
open, it notifies the previous requestor, which still holds the
listening socket, over the filesystem connection.  The old process can
then close its listening socket (new connections will only be delayed
at this point, not refused, since the metaserver still has the
listening socket open as well), close the filesystem connection,
finish servicing its current connections, and exit.  Once the
metaserver sees that the old requestor's filesystem connection has
been closed, it sends the listening socket to the new requestor.

While a server is handling connections, it would have to use
select()/poll() to notice activity on either the listening socket or
the filesystem connection; it couldn't just block on accept() in a
loop.  (Well, it could, but that would mean that when a switchover
starts, it wouldn't be completed until the next client connected.)

If the requestor exits, and no other requestors are around to pass the
listening socket to, the metaserver could close it immediately, or
could keep it open for a few seconds to see if a new requestor show
up.  So quick, non-overlapping restarts would be transparent to the
end clients.

To trigger the switchover, you wouldn't need any signals - just make
runsv forget about the old process using Gerrit's patch.  When the new
process starts up and connects to the filesystem socket, that will
trigger everything else.


paul