My guess is that somehow data left in ram is causing the driver to get screwy. However, that still wouldn’t make sense as it shouldn’t be using uninitialized memory.
I guess do a full power off before switching OS. The other option is to just run a Windows VM.
X11 is used in Xwayland
It is maintained by the same people and is on life support. It has been that way for a decade