Fixed! Thanks to everyone for their help. Especially JD Robinson's lxdm hint.

The problem was entirely mine. I missed the one regular file (i.e non-symlink) in /usr/sbin with the virtual terminal switching every 4 seconds so I didn't delete it. I started from scratch again, stopped lxdm and then I could see it. After rebooting and

dkms remove -m nvidia -v 346.72 -k $(uname -r)
dkms build -m nvidia -v 346.72 -k $(uname -r)
dkms install -m nvidia -v 346.72 -k $(uname -r)
modprobe -v nvidia
sv start lxdm

and I was back to a graphical desktop.