Try using Process Explorer
-- it has a whole page of .NET-specific counters (View->Columns) it can track. (There are other .NET performance tools but they're generally less accessible). It's possible the .NET garbage collector is wigging out periodically, and that may show up as excessive time in GC.
You're sure you're not getting excessive paging due to something else committing a lot of memory in the background? (If no security apps are installed, are you sure you've got a clean system?)