You can get in to some nasty scenarios if you hand configure some stuff and leave others at auto. Some stuff when hand configured goes dumb and doesn't respond to any of the autonegotiation, leaving the stuff running at auto to conclude it is 10Mb/s half-duplex. The speed will then get picked up correctly when it sees the format on the wire, but there is nothing to correct the duplex and you end up with a duplex mismatch - the hand configured end is at full duplex so sends when it likes, the auto configured end is at half-duplex so it considers the incoming packets as collisions and results in terrible throughput. Fortunately most stuff these days rather than going dumb keeps doing autoneg, but just for the speed/duplex that it was set to.
Short version, leave it all at autoneg.
And an Ewwww for the thought of a USB GigE, I thought PCI ones were bad enough!