I haven't actually tried that specific scenario but I don't see why the tunnel wouldn't "come up". I mean really it's up when the interface is up and the tunnel "connection" (it's UDP) isn't actually "established" (in a NAT/firewall sense) unless data is sent. You can have an interface configured for IPv6 on an "IPv4 only" network, it just won't get responses to its router solicitations so the kernel (Linux, at least--BSDs do this in userspace) won't configure routes for IPv6 traffic. If you look at your physical interfaces, you'll probably notice they all have IPv6 link-local addresses unless you've actually turned off IPv6 support in the kernel

The reason your IPv6 traffic goes out unencrypted on dual stack networks is because the default route for IPv6 traffic is not the Wireguard interface, but rather one of the physical ones. All you need to do to send your IPv6 traffic through the tunnel is configure the interface to be part of the IPv6 network you're trying to reach, and of course allow an IPv6 address from the client in the server config. In fact, if you configure your interface with IPv6 address(s) and a route pointing at the wg interface, you can even send IPv6 traffic on an IPv4 only tunnel provided your server can route IPv6 traffic. If you're using wg-quick, simply adding an IPv6 address to the interface and allowing all IPv6 traffic from the server peer would suffice.