Hello *, After running an outdated version of sqmail for a few years, I decided to move inbox.vuxu.org to Postfix 3.8 which is better maintained, has more eyes on the code, and a good track record as well. Migration was almost trivial, the only interesting thing is that a .qmail file to forward e.g. to public-inbox-mda needs an explicit env call: # cat ~public-inbox/.qmail-default |HOME=/var/lib/public-inbox ORIGINAL_RECIPIENT="$EXT@inbox.vuxu.org" /usr/bin/public-inbox-mda # cat ~public-inbox/.forward "|env HOME=/var/lib/public-inbox /usr/bin/public-inbox-mda" Postfix sets ORIGINAL_RECIPIENT out of the box. (We use explicit HOME for reasons I don't want to go into here.) cu, -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
Leah Neukirchen <leah@vuxu.org> writes: > On Friday 2022-07-15 around 12 CEST the host that runs inbox.vuxu.org > and git.vuxu.org will be migrated, which will incur a short downtime. > > This will also result in new IP addresses: > > 168.119.90.161 inbox.vuxu.org > 2a01:4f8:251:5391:5054:ff:fecd:d25c inbox.vuxu.org > > Another announce will be sent after migration. The move was successful. It has also been noticed that the TUHS and COFF archives were outdated as a List-Id configuration needed to be adapted. This has been fixed now. -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
On Friday 2022-07-15 around 12 CEST the host that runs inbox.vuxu.org and git.vuxu.org will be migrated, which will incur a short downtime. This will also result in new IP addresses: 168.119.90.161 inbox.vuxu.org 2a01:4f8:251:5391:5054:ff:fecd:d25c inbox.vuxu.org Another announce will be sent after migration. -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
Hi, during the public-inbox update to 1.8.0 I noticed that inbox.vuxu.org did not show new mails since 2022-04-14. The root cause of this was that the tmpfs at /tmp was full due to runaway log output from a script I used for debugging. This stopped spamc from working, and blocked mail indexing in public-inbox-watch. Mail reception was not affected. To the best of my knowledge, no mails were lost. Luckily, public-inbox is very robust, and restarting public-inbox-watch after cleaning up /tmp started backfilling the indices again. I have set up some basic disk usage monitoring to avoid this problem in the future. The public-inbox 1.8.0 update itself was unproblematic. Cheers, -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
I was getting a lot of errors that looked like extrace: process vanished before we found its parent: pid 1: redirfd Which are easily avoidable. --- extrace.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/extrace.c b/extrace.c index e3a441a..4848a6f 100644 --- a/extrace.c +++ b/extrace.c @@ -140,6 +140,9 @@ pid_depth(pid_t pid) char *s; int fd, d, i; + if (pid == 1) + return 0; + snprintf(name, sizeof name, "/proc/%d/stat", pid); if ((fd = open(name, O_RDONLY)) < 0) -- 2.31.1
Hello, On 2020-11-26, I added a mailing list to inbox.vuxu.org and incorrectly modified the ~ml/.mailfilter, resulting in a broken local MDA setup. This was only discovered and immediately fixed today, 2020-12-06. Unfortunately, qmail bounces undeliverable mail after 7 days, so mail that has been sent between 2020-11-26 and 2020-11-29 is likely lost. I have backfilled the zsh-workers list from their official archives. If you notice anything missing, do not hesitate to contact me and forward when possible. Sorry, -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
[-- Attachment #1: Type: text/plain, Size: 1907 bytes --] Hello, As of 2020-01-01 I'm using a new GPG key: pub ed25519 2020-01-01 [C] [expires: 2021-12-31] E6F6848A1B95EE313CF5B7EE95FF633C90A8F025 uid [ultimate] Leah Neukirchen <leah@vuxu.org> sub ed25519 2020-01-01 [S] [expires: 2021-12-31] sub ed25519 2020-01-01 [A] [expires: 2021-12-31] sub cv25519 2020-01-01 [E] [expires: 2021-12-31] -----BEGIN PGP PUBLIC KEY BLOCK----- mDMEXgyTLhYJKwYBBAHaRw8BAQdAFHreNZ5ceEr5D//IDgKrn/c1/GRbkWEvja+B 3AQFYde0H0xlYWggTmV1a2lyY2hlbiA8bGVhaEB2dXh1Lm9yZz6IlgQTFggAPhYh BOb2hIoble4xPPW37pX/YzyQqPAlBQJeDJMuAhsBBQkDwmcABQsJCAcCBhUKCQgL AgQWAgMBAh4BAheAAAoJEJX/YzyQqPAltQEA/3SBMWcwo8FNdFwUFIg87BQEJE/a 0QYHJ2n/VqBwlH+/AP9HxUuATSW2ks8Zvpr67taDYIlqR1p9JLs6o8rwYMnLDIhd BBARAgAdFiEEI2yO8QRn0lr8O3e2ZU3w2bwbBMgFAl4MpGwACgkQZU3w2bwbBMit mACeP4PkuVF+DJiaCMAdQFm6ITXlgOIAn24nnxdb+E9DFk6Soc/B9IwB/U0NuDME XgyTjBYJKwYBBAHaRw8BAQdAQdsZA/cEcqI0FZv3owpmbXWz42HIbm6pw0VRXFbd MLaI9QQYFggAJhYhBOb2hIoble4xPPW37pX/YzyQqPAlBQJeDJOMAhsCBQkDwmcA AIEJEJX/YzyQqPAldiAEGRYIAB0WIQTTAByP94+fgpN7n6np3nB0LP5b+QUCXgyT jAAKCRDp3nB0LP5b+eZeAQCKAdVAeFHOZahplNRX6gEiNs7Bm3JOMOQV3WG4OzcZ pQD/RMuUdSZC5MmNmuwMDZfFGTAiAGnbwLQPq9Hip5j1tg6s/gD/e+r/4CQpTd6L noZ5JuKndyJxtkvK/skOouc/o0WxYAkBALzMTAmUptBcItpI7EKOlOjmVZX50mkG MRXpljMrVpIMuDMEXgyTkhYJKwYBBAHaRw8BAQdAAij+ifYX6h1NpDOQ2VbYF9+z yr/A/4s1h8q9Bxe6KMmIfgQYFggAJhYhBOb2hIoble4xPPW37pX/YzyQqPAlBQJe DJOSAhsgBQkDwmcAAAoJEJX/YzyQqPAloQIA/j5XRwmsATXhx6KdI6+Qu/foMyVS Fr9avwwmJXhVqz4DAP93J3KDmqZ+iNp5bzFCZzhKzsSxtTHIHp8y4NBRjgFYC7g4 BF4Mk5cSCisGAQQBl1UBBQEBB0AyRbpcbWqBN7Zb7EMRXNw/Q9baMjUDGcdChSzu AVY+SQMBCAeIfgQYFggAJhYhBOb2hIoble4xPPW37pX/YzyQqPAlBQJeDJOXAhsM BQkDwmcAAAoJEJX/YzyQqPAlLpEA/13lLoDoIZb9da/mSyaEIFIiJ4zvtBNLdRfm OEfVZsA4AQCt2uo8WZxoDfXCqJ+745shiTFw5A0bSmIWKYzuAxyxDA== =Qd3v -----END PGP PUBLIC KEY BLOCK----- cu, -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/ [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 227 bytes --]
Hi *, inbox.vuxu.org has been updated to public-inbox 1.2.0. Do not hesitate to contact me if you notice any failures. Best, -- Leah Neukirchen <leah@vuxu.org> https://leahneukirchen.org/
Diego Augusto Molina <diegoaugustomolina@gmail.com> writes: > Here's my suggestion: > > # ss -nlpt | grep 3722 > > That should include your offending instance of socat listening on TCP > 3722, stating the PID that has the resource (a.k.a., the socat process > that opened the port). Killing that PID blindly might not always do > the trick (e.g. "while true; do socat ...; sleep 1; done") so you may > want to kill parents/children too. With that PID in mind use "ps faux" > to navigate through the process tree. My way is: Yes, this didn't help here as the socket was in TIME-WAIT. ss -napt however works. -- Leah Neukirchen <leah@vuxu.org> http://leah.zone
On 2/14/19, Leah Neukirchen <leah@vuxu.org> wrote: > > At 00:20 CET tonight (2019-02-14) epona.vuxu.org, a virtual machine > host that runs, among others, git.vuxu.org/inbox.vuxu.org and > hestia.vuxu.org, the aarch64 builder for Void Linux, went down and > only came up around 11 CET again. > > This was entirely my fault, but how it happened is interesting: > > I was informed a user-space port forwarding was not working. It was > realized using socat, supervised by runit (the init system of Void > Linux): > > socat TCP4-LISTEN:3722,fork,su=nobody TCP6:hestia.vuxu.org:22 > > However, starting this showed the address was already in use: > > 2019/02/14 00:20:44 socat[5049] E bind(5, {AF=2 0.0.0.0:3722}, 16): Address > already in use > > My assumption was there was a runaway instance of socat running (for > unknown reasons), and I decided to kill all socat instances. My usual > tool of choice would have been `killall socat`, but as there were other > socat instances running on the machine, I only wanted to kill the port > 3722 ones. > > A quick test with `pgrep` showed a plausible list of PIDs, so I ran > > kill $(pgrep -f socat.*3722) > > which seemed to work fine at first. > > Several seconds later I was greeted with this message: > > Connection to epona.vuxu.org closed by remote host. > Connection to epona.vuxu.org closed. > > And the box didn't ping anymore... > > As experienced SSH user, this indicated that the host shut down in > some controlled way, else I would have gotten a `broken pipe` message. > > But how could the `pkill` shut down the machine? I could not come up > with any plausible theory, but it was already late and there was alcohol > involved as well. So I decided to leave it at rest until the morning. "Oh, no! How could someone possibly drink and sysadmin!" Said no other sysadmin ever. > > In the morning, the box still was not up and there was no evidence of > a network issue or anything. I decided to enter the Hetzner Control > Panel and trigger an "automated reset". Nothing changed, the box > still didn't ping. I tried to activate the "vKVM rescue system", to > no avail. > > At this point I actually assumed some hardware issue, and I called for > a "manual reset", which means someone has to get up, walk to the > machine, restart it, and watch a bit whether it seems to boot > properly. > > Of course, the true reason was much simpler: the box was powered off. > > Unfortuately, nothing about the Hetzner Control Panel shows you this > simple fact, so I guess I'm not the only one to send poor support > folks to go boot other people's machines. > > The box booted fine and all services were restored within minutes. > > The remaining question is how it's possible that the command shut down > the machine, and it's easy to answer too: > `runsvdir`, the main runit process that controls "stage 2", i.e. > while the system is up, displays error messages of all direct > child processes in it's `argv[0]`, so you can check for unlogged > messages with `ps`: > > runsvdir -P /run/runit/runsvdir/current log: ....logs here.... > > Unfortunately, in above sitation this resulted in both "socat" and > "3722" to appear in the error messages, and thus the process title, > which made `pkill -f` match it and, as commanded, kill `runsvdir`, > which results in exiting stage 2 and runit performing an orderly > shutdown of the system. Duh. > > Lessons learned: > - The first intuition is often right, even if it's not plausible at first. > - Don't use `pkill -f` as root, at least not without careful checking > and regexp anchoring. > - If a box doesn't react to reset requests, try sending wake-on-lan to > turn it on. > - runit should reboot by default, not shutdown! > > -- > Leah Neukirchen <leah@vuxu.org> http://leah.zone > Here's my suggestion: # ss -nlpt | grep 3722 That should include your offending instance of socat listening on TCP 3722, stating the PID that has the resource (a.k.a., the socat process that opened the port). Killing that PID blindly might not always do the trick (e.g. "while true; do socat ...; sleep 1; done") so you may want to kill parents/children too. With that PID in mind use "ps faux" to navigate through the process tree. My way is: # ps faux | grep -vF \[ | less -SRI The grep is to remove kernel processes which drown the output. Bye.
next time consider to begin with grepping the netstat output ;)
At 00:20 CET tonight (2019-02-14) epona.vuxu.org, a virtual machine host that runs, among others, git.vuxu.org/inbox.vuxu.org and hestia.vuxu.org, the aarch64 builder for Void Linux, went down and only came up around 11 CET again. This was entirely my fault, but how it happened is interesting: I was informed a user-space port forwarding was not working. It was realized using socat, supervised by runit (the init system of Void Linux): socat TCP4-LISTEN:3722,fork,su=nobody TCP6:hestia.vuxu.org:22 However, starting this showed the address was already in use: 2019/02/14 00:20:44 socat[5049] E bind(5, {AF=2 0.0.0.0:3722}, 16): Address already in use My assumption was there was a runaway instance of socat running (for unknown reasons), and I decided to kill all socat instances. My usual tool of choice would have been `killall socat`, but as there were other socat instances running on the machine, I only wanted to kill the port 3722 ones. A quick test with `pgrep` showed a plausible list of PIDs, so I ran kill $(pgrep -f socat.*3722) which seemed to work fine at first. Several seconds later I was greeted with this message: Connection to epona.vuxu.org closed by remote host. Connection to epona.vuxu.org closed. And the box didn't ping anymore... As experienced SSH user, this indicated that the host shut down in some controlled way, else I would have gotten a `broken pipe` message. But how could the `pkill` shut down the machine? I could not come up with any plausible theory, but it was already late and there was alcohol involved as well. So I decided to leave it at rest until the morning. In the morning, the box still was not up and there was no evidence of a network issue or anything. I decided to enter the Hetzner Control Panel and trigger an "automated reset". Nothing changed, the box still didn't ping. I tried to activate the "vKVM rescue system", to no avail. At this point I actually assumed some hardware issue, and I called for a "manual reset", which means someone has to get up, walk to the machine, restart it, and watch a bit whether it seems to boot properly. Of course, the true reason was much simpler: the box was powered off. Unfortuately, nothing about the Hetzner Control Panel shows you this simple fact, so I guess I'm not the only one to send poor support folks to go boot other people's machines. The box booted fine and all services were restored within minutes. The remaining question is how it's possible that the command shut down the machine, and it's easy to answer too: `runsvdir`, the main runit process that controls "stage 2", i.e. while the system is up, displays error messages of all direct child processes in it's `argv[0]`, so you can check for unlogged messages with `ps`: runsvdir -P /run/runit/runsvdir/current log: ....logs here.... Unfortunately, in above sitation this resulted in both "socat" and "3722" to appear in the error messages, and thus the process title, which made `pkill -f` match it and, as commanded, kill `runsvdir`, which results in exiting stage 2 and runit performing an orderly shutdown of the system. Duh. Lessons learned: - The first intuition is often right, even if it's not plausible at first. - Don't use `pkill -f` as root, at least not without careful checking and regexp anchoring. - If a box doesn't react to reset requests, try sending wake-on-lan to turn it on. - runit should reboot by default, not shutdown! -- Leah Neukirchen <leah@vuxu.org> http://leah.zone
Hi, this is a new public-inbox for development of leahutils. Please send bugs, patches and pull requests to leahutils@inbox.vuxu.org Archives are at http://inbox.vuxu.org/leahutils/ cu, -- Leah Neukirchen <leah@vuxu.org> http://leah.zone