1. Remote in to your work desktop from home (via secure SSH tunnel) to check on something.
2. Forget to disconnect the remote desktop session when leaving home to go to the office.
3. Arrive at the office, and notice that your desktop seems a little laggy/sluggish. Normal windows just don't seem quite as smooth as they should be, and VirtualBox VMs have become laggy enough to be *really* annoying.
4. Spend the next 10 minutes trying to figure out what's broken. Plenty of free RAM... CPU load below 20%... disk not thrashing... WTF?!??
5. Just as you're about to give up and try a reboot, finally notice the little Remote Desktop icon on the taskbar.
D'oh!
I suspect that VMs were much more severely affected because the VM's entire display is probably treated as a raw bitmap that has to be updated across the RD connection, whereas updating of native (non-VM) windows is probably easier to optimize (e.g. forward the higher-level X-Windows primitives or something).