As things (texture data, primarily) exceed on-card memory, the video driver swaps with main (system) memory, not with secondary storage (HD, SSD, whatever). The OS memory manager in turn may then start paging out to secondary storage if it is overcommitted (in which case you should be buying more RAM, assuming a 64bit OS, rather than looking for an SSD), but that's outside the purview of the graphics card and its drivers which can't "see" the storage devices and have no idea what SSDs or HDs you might have attached. A game of course may be trying to juggle things between system memory and on-card memory on its own, but I would expect game code that sees constrained video card memory will seek to use less memory overall (ie automatically reduce texture size) to get optimal results. The last thing the game code wants to do is juggle with the disks during gameplay, which is why they try to load as much as they can at the start of a level.
However, to answer your question, let's assume that you are seeing system memory paging due to all of the above (ie insufficient graphics card memory resulting in thrashing through the memory hierarchy to your page file) so that the graphics card is effectively waiting on something getting paged in from the page file or a resource loaded out of the game image*. Pulling a page from the page file is effectively a 4KB random-access read. Latency for that on a recent fast SSD
is on the order of 0.03ms, which is fantastic compared to hard drives (which run into several-to-ten or more full ms at best) but that's still extremely slow as far as the GPU and its on-card memory are concerned. Modern GDDR has latencies measured in (single digit) nanoseconds, whereas .03ms is still 3000ns. So loading from SSD might be a hundred times faster than a HD, but it's still a thousand times slower than memory. You might not notice if it's just one 4KB page, but it never is, and if the GPU can't proceed without that data, you're going to see some amount of choppiness.
* Of course getting it off the disk is just the first step; once the data is in memory it has to be DMA'd through to the card. With 16 lanes of PCIe 3.0, the graphics card can pull a full 1GiB of data in 1/16th of a second, and the game probably is waiting on less than that, so (ignoring latency, OS/API overhead, and a host of other things) we'll just ignore any added latency from that stage (it'll be the same for SSD vs HD anyway).