Another way could be to be able to link the kernel with device drivers before loading it, and keep the kernel just with devices used. I think that could be done by making each driver provide a (user) program to check if the device is there, and linking a second kernel depending on what's there.