Fixed! Thanks to everyone for their help. Especially JD Robinson's lxdm 
hint.

The problem was entirely mine. I missed the one regular file (i.e 
non-symlink) in /usr/sbin with the virtual terminal switching every 4 
seconds so I didn't delete it. I started from scratch again, stopped lxdm 
and then I could see it. After rebooting and
dkms remove -m nvidia -v 346.72 -k $(uname -r)
dkms build -m nvidia -v 346.72 -k $(uname -r)
dkms install -m nvidia -v 346.72 -k $(uname -r)
modprobe -v nvidia
sv start lxdm
and I was back to a graphical desktop.