A co-worker and I just spent a decent chunk of the weekend chasing a bug in one of Microchip's PIC32 microcontrollers. Hardware bugs are the last thing a software engineer suspects, because bugs are usually due to software!
The nature of the bug is that reading or writing a hardware device register that has side effects beyond the register access itself can get executed twice if a hardware interrupt occurs while the machine instruction that accesses the register is executing. A good example of this would be a UART with a hardware FIFO -- writing to the Tx data register stuffs a byte into the FIFO *and* increments the FIFO pointer.
There's an existing Microchip errata that says writes to device registers may be repeated if an interrupt occurs. We now have convincing evidence that reads are affected as well, so it would seem the errata itself needs an errata.
So now we get to vet all of the code for accesses to registers with side effects, and add code to disable/enable interrupts around all such accesses.
D'oh!