It's probably a driver crash
- Nvidia drivers seem to do this a lot, it's never been fixed, and it's been happening for years to thousands of people.
There are good reasons a lot of software writers hate Nvidia. Linus Torvalds had good reasons to call Nvidia "the single worst company the Linux developer community has ever
You probably can't RMA your card because it'll pass testing, but it's worth saying you're dissatisfied with it.
You could return it outright and get an AMD but you may have to pay for shipping and a restocking fee again.
You could hope it's not the TDR issue and spend days/weeks/months trying random stuff to fix it instead of actually using it to play games.
In the EU there are distance selling regulations that would help you out here, not sure if the US has anything similar.