So I have three main concerns
1: Power button - will i need to solder the leads from the original DC power button to properly fitting cables to wire it to my motherboard? I know you have to for NES PC builds, not sure if I will have to on a Dreamcast build
You could verify this by googling up some internal schematics, diagrams, or simply photos. All you need are two wires connected to a simple switch. It's not difficult to get working. I do suspect you will have to add a header plug thingy to the end of the wires, at least, but I have never seen a Dreamcast in person (much less an opened one) so I have no real idea.
2: Motherboard mounting - any opinions on how to rest the motherboard assuming I can't feng shui working standoffs in? Some have recommended polystyrene or similar - not sure what works and what doesn't
Anything which isn't conductive can be used. Try to avoid using materials which will trap heat, like silicone. I might just hot glue the board in place if it were me, but I love hot glue. (´・ω・｀)
3: Low profile cooling - the dreamcast has a height of 76 mm - stock AMD heatsink is 60 mm - I can buy a low profile 40mm-ish cooler, but might it be better to buy a CPU-block water cooler and just route the radiator outside of the physical case?
Using a closed-loop cooler with external radiator is certainly an option
, but I sure wouldn't recommend it. The hoses don't really flex all that easily and you'll have this little bitty device with a pair of inflexible hoses and a giant block of metal hanging off of it ...
Cooling will be a problem with such a small enclosure. I'd make sure to use a down-draft cooler (which virtualy all low-profile coolers are) with as powerful of a fan as I can manage.
Still, depending on what A6 you have, you're looking at a best-case of 65W TDP. That's going to be hard to cool at load with a low-profile cooler. I don't really know what to tell you. High-speed fans are noisy, which is bad for a HTPC-style system. Maybe you could remove all the disk drive mechanics and cut out the bay so that the drive cover opens directly into the bottom of the machine and open it when playing games for extra airflow? Haha...
Alternatively you could use a tower cooler and cut a hole in the top to let it stick out the top of the machine like some kind of old V8 muscle car's supercharger. That would be kind of neat.