[9fans] strange things in /sys/log/auth

[9fans] strange things in /sys/log/auth

[andrey mirtchovski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 585 bytes --]

i have the following appearing in my logs several times a day:


[ see the attached file, i snipped this to ensure that my mailer doesn't
mangle it ]

the number of entries match the number of non-default users that I have
added. for each one of them I've chosen "never" as the 'expire' option. the
users were added using the fossil console, and then their options were set
using auth/changeuser.

is this a bug? should I change the way I'm adding users?

there's nothing else wrong with authentication -- it works fine for all
users, as far as I can see...

andrey


[-- Attachment #2: Type: TEXT/plain, Size: 725 bytes --]

plan9 Jun 30 18:54:18 keyfs starting warnings: 3f00dbba 3eff8a2d
warning: can't open bootes/expire:r
warning: can't open rminnich/expire:r
warning: can't open €\€€€)€͝€m\x05X€h%€K€>€9F€X€€€€Ԟ€$€€€߶€€€v?\x1f€€'€€\x06,?D%+€€€5€\x7fx€€yyk5%2iR/expire:r
warning: can't open €\€€€)€͝€m\x05X€h%€K€>€9F€X€€€€Ԟ€$€€€߶€€€v?\x1f€€'€€\x06,?D%+€€€€€€)>€"€\ra€€\x17>/expire:r
warning: can't open €v4/expire:r
warning: can't open €\€€€)€͝€m\x05X€h%€K€>€9F€X€€€€Ԟ€$€€€߶€€€v?\x1f€€'€€\x06,?D%+€€€€\x1a€€Z€€ڋ\aP€3/expire:r
warning: can't open €\€€€)€͝€m\x05X€h%€K€>€9F€X€€€€Ԟ€$€€€߶€€€v?\x1f€€'€€\x06,?D%+€€€€\x1a^[v€f€€€~€€/expire:r

Re: [9fans] strange things in /sys/log/auth

[Dan Cross]

> there's nothing else wrong with authentication -- it works fine for all
> users, as far as I can see...

I've seen this before.  It looks like the key that keyfs is using
(presumably taken from nvram) doesn't match the key that was used when
initially creating /adm/keys.  Odds are good you added all your users
after the initial corruption due to the key mismatch, though, which is
why authentication still works correctly.

If you have a small number of users, I'd suggest blowing away the
current keys file and recreating it using the password that's in nvram,
and then reassigning passwords.  If that's too much of a hassle, then
you could cd into /mnt/keys and just remove all the corrupt entries,
but I'm not sure if you wouldn't run into other problems later on down
the road.

Or, I could be completely wrong.  However, the last time I saw behavior
like what you're describing, it was due to a key mismatch.  Luckily,
since I had just set up the auth server and there was no one in it, it
was easy for me to just recreate it.

	- Dan C.

Re: [9fans] strange things in /sys/log/auth

[Geoff Collyer, geoff]

What Dan said.

It looks like what I saw when I gave my cpu server (and thus its
keyfs) the wrong key.  What does "ls -l /mnt/keys" on your cpu's
console print?  It should be something like this:

cpu% ls /mnt/keys
/mnt/keys/bootes
/mnt/keys/claudia
/mnt/keys/cross
/mnt/keys/dhog
/mnt/keys/geoff
/mnt/keys/martha
/mnt/keys/plus
/mnt/keys/upas

If instead it looks like:

/mnt/keys/€9F€X€€€€Ԟ€$€€
/mnt/keys/+€€€€\x1a^[v€fÂ

Then keyfs has the wrong key in nvram.

Re: [9fans] strange things in /sys/log/auth

[andrey mirtchovski]

well, it only has two users -- bootes and me, nobody else that I've added
appears there...

it must be said that the events i reported do not appear in the logs from
the very beginning. first it was a complaint about bootes' expire, then
another user appeared there, and then i got the corrupted strings...

the fact that the keys were just moved from a kfs server with all the other
data from the disk probably wouldn't have helped.

i'll follow dan's suggestion and reset everything once i can sit behind the
console...

andrey

ps: /mnt/keys of a cpu server:

kn9% ls -l /mnt/keys
d-r-xr-xr-x M 117 bootes sys 0 Jun  3 12:38 /mnt/keys/bootes
d-r-xr-xr-x M 117 bootes sys 0 Jun 24 09:13 /mnt/keys/andrey
plan9%

On Mon, 30 Jun 2003, Geoff Collyer wrote:

> What Dan said.
>
> It looks like what I saw when I gave my cpu server (and thus its
> keyfs) the wrong key.  What does "ls -l /mnt/keys" on your cpu's
> console print?  It should be something like this:
>
> cpu% ls /mnt/keys
> /mnt/keys/bootes
> /mnt/keys/claudia
> /mnt/keys/cross
> /mnt/keys/dhog
> /mnt/keys/geoff
> /mnt/keys/martha
> /mnt/keys/plus
> /mnt/keys/upas
>
> If instead it looks like:
>
> /mnt/keys/€9F€X€€€€Ԟ€$€€
> /mnt/keys/+€€€€\x1a^[v€fÂ
>
> Then keyfs has the wrong key in nvram.
>

Re: [9fans] strange things in /sys/log/auth

[andrey mirtchovski]

changed bootes' password using 'passwd' and didn't reboot, now nobody
can login, including bootes him(her?)self :) this must mean something...

andrey

Re: [9fans] strange things in /sys/log/auth

[Geoff Collyer, geoff]

Curious, my files in /mnt/keys are owned by auth and auth, not bootes
and sys.  Your permissions are also stricter than mine.  Are you
running the latest keyfs?

Re: [9fans] strange things in /sys/log/auth

[andrey mirtchovski]

diff (cmp) doesn't give any difference between keyfs on my fossil system and
sources (i do updates daily, sometimes more so :)..

neither the source nor the binary differ...

the machine is in a private network, so I could explain you my entire
auth configuration. what exactly would be most helpful?

andrey

ps: i just observed that /mnt/keys is an exact duplicate of the one that is
on the kfs server (i just did mkfs < /n/kfs | mkext > /n/fossil). the logs
on the kfs server show complaints about bootes' 'expire' but none of the
garbage that i posted couple of messages ago...

pps: if you don't see anything obvious i suggest we just wait until i get a
chance to reset the auth info on wednesday (tomorrow is canada day, so we
drink beer... i think... we always drink beer around here :)

there is a big chance i screwed something up while converting the kfs into a
fossil, i was just hoping that it'll all work, and so far it has, apart from
these silly log messages :)

On Mon, 30 Jun 2003, Geoff Collyer wrote:

> Curious, my files in /mnt/keys are owned by auth and auth, not bootes
> and sys.  Your permissions are also stricter than mine.  Are you
> running the latest keyfs?
>

Re: [9fans] strange things in /sys/log/auth

[okamoto]

> Curious, my files in /mnt/keys are owned by auth and auth, not bootes
> and sys.

By the way, I'm wordering where my 'auth' user and group names come from.
I have no auth entry in our /adm/users files.

Kenji

Re: [9fans] strange things in /sys/log/auth

[Geoff Collyer, geoff]

[-- Attachment #1: Type: text/plain, Size: 133 bytes --]

Each file server has its own set of users.  Unfortunately, most of
them don't read and don't publish a corresponding "users" file.

[-- Attachment #2: Type: message/rfc822, Size: 1549 bytes --]

From: okamoto@granite.cias.osakafu-u.ac.jp
To: 9fans@cse.psu.edu
Subject: Re: [9fans] strange things in /sys/log/auth
Date: Tue, 1 Jul 2003 15:37:22 +0900
Message-ID: <2379be97243831ab61196ac2b0fa95e6@granite.cias.osakafu-u.ac.jp>

> Curious, my files in /mnt/keys are owned by auth and auth, not bootes
> and sys.

By the way, I'm wordering where my 'auth' user and group names come from.
I have no auth entry in our /adm/users files.

Kenji

Re: [9fans] strange things in /sys/log/auth

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

It means you have to:

1) kill keyfs
2) run auth/convkeys on the keyfile
3) reboot the auth server (or just give the new key to factotum on the auth server)
4) if you use an nvram on the auth server to store the key, use auth/wrkey
  to rewrite it

[-- Attachment #2: Type: message/rfc822, Size: 1928 bytes --]

From: andrey mirtchovski <mirtchov@cpsc.ucalgary.ca>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] strange things in /sys/log/auth
Date: Mon, 30 Jun 2003 23:49:13 -0600 (MDT)
Message-ID: <Pine.LNX.4.44.0306302347430.14298-100000@fbsd.cpsc.ucalgary.ca>

changed bootes' password using 'passwd' and didn't reboot, now nobody
can login, including bootes him(her?)self :) this must mean something...

andrey

Re: [9fans] strange things in /sys/log/auth

[andrey mirtchovski]

i fixed the problem -- the /adm/keys file was fscked somehow.

i was able to see directory entries for all users i've added in /mnt/keys,
but after the directory listing there was garbage, the same thing Geoff sent
a few messages back. when started keyfs was reporting 'bad status in key'.

i ended up doing:

	% rm /adm/keys
	% con -l /srv/fscons
	main: create /active/adm/keys bootes bootes 660
	% auth/changeuser bootes
	...

and then added all other users again. i've saved the old keys file, in case
anyone is interested in playing with it to see what's wrong. i've changed
all passwords :)

i'm a little worried, though -- i was thinking that bootes can only change
passwords on the console, but killing keyfs and restarting it again as
bootes in a local namespace gives me write privileges to /mnt/keys (though
i loose the ability to login to the machine, naturally :)

is it advisable to lock the keyfs process the same way factotum's process
is? put it in the kernel? disallow 'echo kill > /proc/xx/ctl'?

inquiring minds want to know :)


On Tue, 1 Jul 2003, David Presotto wrote:

> It means you have to:
>
> 1) kill keyfs
> 2) run auth/convkeys on the keyfile
> 3) reboot the auth server (or just give the new key to factotum on the auth server)
> 4) if you use an nvram on the auth server to store the key, use auth/wrkey
>   to rewrite it

Re: [9fans] strange things in /sys/log/auth

[david presotto]

Lots of possible problems.  If you change the bootes password, and convkeys
the file while still running keyfs with the old keys, you can easily hose
yourself.
You have to kill off the keyfs running with the old keys before you convkeys
anything.

----- Original Message -----
From: "andrey mirtchovski" <mirtchov@cpsc.ucalgary.ca>
To: <9fans@cse.psu.edu>
Sent: Wednesday, July 02, 2003 12:35 PM
Subject: Re: [9fans] strange things in /sys/log/auth


> i fixed the problem -- the /adm/keys file was fscked somehow.
>
> i was able to see directory entries for all users i've added in /mnt/keys,
> but after the directory listing there was garbage, the same thing Geoff
sent
> a few messages back. when started keyfs was reporting 'bad status in key'.
>
> i ended up doing:
>
> % rm /adm/keys
> % con -l /srv/fscons
> main: create /active/adm/keys bootes bootes 660
> % auth/changeuser bootes
> ...
>
> and then added all other users again. i've saved the old keys file, in
case
> anyone is interested in playing with it to see what's wrong. i've changed
> all passwords :)
>
> i'm a little worried, though -- i was thinking that bootes can only change
> passwords on the console, but killing keyfs and restarting it again as
> bootes in a local namespace gives me write privileges to /mnt/keys (though
> i loose the ability to login to the machine, naturally :)
>
> is it advisable to lock the keyfs process the same way factotum's process
> is? put it in the kernel? disallow 'echo kill > /proc/xx/ctl'?
>
> inquiring minds want to know :)
>
>
> On Tue, 1 Jul 2003, David Presotto wrote:
>
> > It means you have to:
> >
> > 1) kill keyfs
> > 2) run auth/convkeys on the keyfile
> > 3) reboot the auth server (or just give the new key to factotum on the
auth server)
> > 4) if you use an nvram on the auth server to store the key, use
auth/wrkey
> >   to rewrite it
>
>