Personal computing discussed

Moderators: SecretSquirrel, notfred

 
blahsaysblah
Gerbil Elite
Topic Author
Posts: 580
Joined: Mon Oct 19, 2015 7:35 pm

GPU passthrough

Sat Aug 12, 2017 7:13 am

Has anyone switched their main PC to Linux host with QEMU/KVM so they could have isolated Win 10 VMs w/dedicated GPU for gaming/work. To blow away or archive as necessary? When i lasted googled, it led me to Windows Server(GPU pass through) so i dropped it but it seems its been semi-viable since Intel 6X/7X chipsets on Linux side. Been watching a bunch of YT videos and reading articles to get better grasp.

It would seem with Threadripper, it would be a viable alternative to buying multiple PCs for a family. Would ZFS dedup take care of little timmy and emily having duplicate copies of games in their separate VMs?

Anyone have recent first hand experience with GPU pass-through on Linux? This all came up because of the very disappointing Windows 10 for Workstations announcement/rumor had me re-checking state of things.
 
CScottG
Gerbil Elite
Posts: 958
Joined: Fri Dec 01, 2006 9:53 pm

Re: GPU passthrough

Sun Aug 13, 2017 12:06 am

Yes.. but never multiple concurrent Win 10 machines with different GPU pass-through, I just don't have that need. :oops:

I don't think a limited hardware solution is practical though when you consider peripherals unless everyone uses the same room for their computing needs. You can of course use Windows 10 Pro Remote Desktop VM (Host, not Guest) with additional hardware (..like remoting-in from laptop or tablet), but it's not the same and it still requires more hardware (and software/OS).

(..deduplication looks interesting, not unlike differencing with VHDX drives.)


..btw, none of my Win 10 machines are licensed/registered. It's not a requirement on installation (..but they do need to be allowed to update) - basically 10 is spyware, so limit its use for certain applications like games.
 
Ethyriel
Gerbil First Class
Posts: 153
Joined: Mon Feb 02, 2004 6:41 pm
Location: Tucson, AZ

Re: GPU passthrough

Sun Aug 13, 2017 8:26 am

Keep in mind that ZFS dedup is pretty hard on the ARC/L2ARC for the index, and it's capped at 25% of the ARC. So you either need a lot of RAM just for ZFS, or a very fast disk for L2ARC. I'd recommend an NVME disk for L2ARC since this is a VDI host that needs RAM for those guests. Normally I'd use NVME for ZIL and a couple of SATA SSD for L2ARC, but in this case I'd reverse that or use NVME for both.

Be wary of anti-cheat measures playing the same games simultaneously in different guests, or sometimes just being detected as a VM. You'll also be in a cat and mouse game with Nvidia consumer cards, so look at AMD.
 
just brew it!
Gold subscriber
Administrator
Posts: 49673
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: GPU passthrough

Sun Aug 13, 2017 9:34 am

I suspect that even with a beefy processor, lots of RAM, and GPU passthrough, performance and (especially) stability are going to leave something to be desired for gaming. Also, as CScottG notes, everyone will need to be physically located within video/USB cable range of the server.

Is it possible? Yeah, probably. Will the end result be worth the trouble? I doubt it. It would be an interesting experiment though.
Nostalgia isn't what it used to be.
 
CScottG
Gerbil Elite
Posts: 958
Joined: Fri Dec 01, 2006 9:53 pm

Re: GPU passthrough

Sun Aug 13, 2017 2:53 pm

Ethyriel wrote:
So you either need a lot of RAM just for ZFS, or a very fast disk for L2ARC..



I'd think it would have to be the latter with the amount of RAM used by each VM. (..Qemu is very good with balancing CPU loads, but requires fixed system RAM for each VM.)

6 (full hardware) VM's systems running simultaneously would alone need something between 8 and 16 gig each. Then the underlying OS with about 16 gig on its own, plus at least 5 gig for each terabyte of disk.. really becomes expensive. :o



-plus you'd want a good L2ARC so you don't loose performance on reboot (..which from the home systems I use happens at least every few months).
 
CScottG
Gerbil Elite
Posts: 958
Joined: Fri Dec 01, 2006 9:53 pm

Re: GPU passthrough

Sun Aug 13, 2017 3:01 pm

One area where it might be worth the cost/hassle of setup is management - even if it's a Remote Desktop affair for most of the "seats"..


I love being able to remote into my Father's system and just "replacing" systems, keeping his general activity (browsing/word-processing, etc.) limited to a Linux machine, etc..

It's just more "ordered" and "clean", and I no longer have that fear that: "oh god, he just installed something really crappy that's going to cause me to have to refresh the entire system and reinstall what good applications I can" - just to get back to a usable state.
 
blahsaysblah
Gerbil Elite
Topic Author
Posts: 580
Joined: Mon Oct 19, 2015 7:35 pm

Re: GPU passthrough

Mon Aug 14, 2017 8:08 am

Yeah, i was more thinking along the lines of for myself. To isolate the games in a VM and not worry about it, like i do with work stuff right now. Easy to carry the VM through rebuilds, simple copy to backup,... But you're right, i remember reading about some games anti-cheat protection not liking running inside a VM.

But yeah, as for the bigger machine for family, at some point, they'll each need a laptop for school work anyway.
 
Ryu Connor
Gold subscriber
Global Moderator
Posts: 4215
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: GPU passthrough

Mon Aug 14, 2017 9:02 am

All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
just brew it!
Gold subscriber
Administrator
Posts: 49673
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: GPU passthrough

Mon Aug 14, 2017 9:05 am

Ryu Connor wrote:

Don't have time to watch them now (will probably do so tonight), but the take-home seems to be "If you've got $30K to drop on a server, go for it!! :D
Nostalgia isn't what it used to be.
 
Ryu Connor
Gold subscriber
Global Moderator
Posts: 4215
Joined: Thu Dec 27, 2001 7:00 pm
Location: Marietta, GA
Contact:

Re: GPU passthrough

Mon Aug 14, 2017 9:27 am

just brew it! wrote:
Don't have time to watch them now (will probably do so tonight), but the take-home seems to be "If you've got $30K to drop on a server, go for it!! :D


From my perspective it shows the technology for this use case isn't cost effective.

VDI is a beautiful thing in the business world, but that use case doesn't require a ton of cores, RAM, and GPUs to be passed through.
All of my written content here on TR does not represent or reflect the views of my employer or any reasonable human being. All content and actions are my own.
 
just brew it!
Gold subscriber
Administrator
Posts: 49673
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: GPU passthrough

Mon Aug 14, 2017 9:44 am

Ryu Connor wrote:
just brew it! wrote:
Don't have time to watch them now (will probably do so tonight), but the take-home seems to be "If you've got $30K to drop on a server, go for it!! :D

From my perspective it shows the technology for this use case isn't cost effective.

Yes, I think that was also implied by my "if you've got $30K..." comment. :wink:

It's still interesting tech.
Nostalgia isn't what it used to be.
 
Forge
Lord High Gerbil
Posts: 8221
Joined: Wed Dec 26, 2001 7:00 pm
Location: SouthEast PA

Re: GPU passthrough

Mon Aug 14, 2017 9:47 am

Yeah, I've done it. Also, that line about "since the Intel 6X/7X chipsets" is way, way off the mark. You need to have working IOMMU/ACS for VFIO to be usable, and even then it's not likely to be very stable. Pretty much the only boards that really got ACS/IOMMU working 100% were the X79/X99 boards. Lots of others (My Z97 included) say it's working, and will work for a quick setup and a GPU-Z screenshot, but will crap the bed and take the host down five to ten minutes later.

Much like wheelies on a motorcycle, GPU passthrough is exciting and impressive, but also like wheelies, it's hard to maintain and requires a great deal of maintenance or you'll hurt yourself.

When you add in multiple GPUs and multiple USB cards to handle guest keyboards and mice, the overhead goes up and the chance of error failure does as well.

If you'd like to know any specifics, shoot them my way. I've had Windows, MacOS, and Linux guests running on my Linux host. I'm not currently doing any passthrough, for the above reasons, but I can easily reconstitute my setup if you need to see something or copy parts of a config.

I'd recommend subscribing to the vfio-users mailing list if you're serious. A month or two grazing that list and the problem reports will give you a really good idea of what you're in for.

https://www.redhat.com/mailman/listinfo/vfio-users
Behemoth - Plex server
Galactica - FreeNAS
Pegasus - R710 ESX host
fsociety - MBP 15
Rocinante - E5530 testing machine
Saratoga, Ticonderoga, BunkerHill - the Hydra desktop PC-thing.
 
blahsaysblah
Gerbil Elite
Topic Author
Posts: 580
Joined: Mon Oct 19, 2015 7:35 pm

Re: GPU passthrough

Mon Aug 14, 2017 1:04 pm

Thanks for the info. My curiosity was initially peaked by whats inferred here: GPU Passthrough for Virtualization with Ryzen: Now Working. That Intel side has been good for a short while.

I sold my GPU during mining craze, this is just research on do's and donts for next build. In case i get something during Black Friday/Holidays.

Only big standout is Nvidia adding VM check to GTX driver installation to abort install. You can work around now, but if they take even harder stance...
The HEDT platforms support IOMMU super pages for less impact besides being likely used in actual builds with the Quadro/Pro cards.

You still need new-ish kernels and critical software,.. and i guess that's not something i would want for a host OS. Bleeding edge stuff. Otherwise, there would be a lot more noise about this. Most every Linux post would say there is no excuse to not have an LTS as your host OS.
 
Forge
Lord High Gerbil
Posts: 8221
Joined: Wed Dec 26, 2001 7:00 pm
Location: SouthEast PA

Re: GPU passthrough

Mon Aug 14, 2017 1:09 pm

VFIO development is active and ongoing. If you want a polished and reliable product for a turnkey solution, we're a few years away still, IMO.

I also would not endorse Ryzen nor Threadripper for VFIO work yet. There's only a few days in so far, and many issues are already documented and being worked on. The Ryzen segfault under heavy loads issue also has me concerned. Viability not yet guaranteed.
Behemoth - Plex server
Galactica - FreeNAS
Pegasus - R710 ESX host
fsociety - MBP 15
Rocinante - E5530 testing machine
Saratoga, Ticonderoga, BunkerHill - the Hydra desktop PC-thing.
 
CScottG
Gerbil Elite
Posts: 958
Joined: Fri Dec 01, 2006 9:53 pm

Re: GPU passthrough

Mon Aug 14, 2017 1:27 pm

Forge wrote:
Yeah, I've done it. Also, that line about "since the Intel 6X/7X chipsets" is way, way off the mark. You need to have working IOMMU/ACS for VFIO to be usable, and even then it's not likely to be very stable. Pretty much the only boards that really got ACS/IOMMU working 100% were the X79/X99 boards.




With Intel systems (and a "virtual" guarantee that it *should* work) ..it's a matter of the processor having VT-D (for passthrough, recent processors all seem to have VT-x) with ECC memory support,

*and*

The motherboard supporting ECC memory AS ECC memory (and of course actual VT-D support).


..and really, you generally shouldn't be building a VM server without ECC memory (that's actually error correcting).


Both of the builds that I've done are based on the "lowly" i3-6100 *consumer* processor that ALSO use C236 (or C232) chipsets for the motherboards.. ie. server or workstation-class motherboards from the Intel line. Xeons of course are a more obvious solution. Most of the Skylake m-board chipsets provide VT-D (including consumer), but I believe that in many cases it's not implemented by the m-board manufacturer. This is not the case with boards using the server/workstation chipsets.

ex. i3-6100:
http://ark.intel.com/products/90729/Int ... e-3_70-GHz
Skylake:
http://ark.intel.com/products/codename/37572/Skylake



Ryzen (and presumably Threadripper) is far more "hit and miss" in this regard, but the processor supports both IOMMU for pass-through and ECC support - from there it's more a matter of the motherboard's IOMMU support and an updated Kernal (on the Linux-side).

It is however new hardware comparatively speaking.. so it's not surprising that it's going through "growing-pains".

https://www.youtube.com/watch?v=aLeWg11ZBn0
 
blahsaysblah
Gerbil Elite
Topic Author
Posts: 580
Joined: Mon Oct 19, 2015 7:35 pm

Re: GPU passthrough

Tue Aug 15, 2017 3:20 pm

Forge wrote:
VFIO development is active and ongoing. If you want a polished and reliable product for a turnkey solution, we're a few years away still, IMO.

I also would not endorse Ryzen nor Threadripper for VFIO work yet. There's only a few days in so far, and many issues are already documented and being worked on. The Ryzen segfault under heavy loads issue also has me concerned. Viability not yet guaranteed.

Coming to same conclusion as you. Though i would like to try it out at some point in near future. Both for gaming VM and for a couple pesky graphics apps with strict licenses(that id like to setup and leave alone)
 
weaktoss
Gerbil
Posts: 63
Joined: Fri Jun 07, 2013 1:09 pm

Re: GPU passthrough

Tue Aug 15, 2017 3:47 pm

I have an i5-6400 in a Shuttle barebones in my living room running Ubuntu as the host OS and three KVM/libvirt VMs : a Windows system with a 1050 Ti passed through (and connected to my TV), pfsense for routing, and another Ubuntu system hosting the controller software for my Unifi access points.

It was a lot of stumbling in the dark for a few weeks to get everything set up and working correctly, but it's been very stable. I don't know that I'd do it with anything mission-critical; a lot of the motivation for me was just to have something to learn and do.

No experience with running multiple gaming VMs at once, but single GPU pass-through wasn't too bad, if you have patience for a few hours of googling, forum crawling, and troubleshooting.

Who is online

Users browsing this forum: No registered users and 2 guests