Nvidia Multi-GPU support

Come join the... uh... er... fold.

Moderators: just brew it!, farmpuma

Nvidia Multi-GPU support

Postposted on Tue Jul 01, 2008 5:42 pm

JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 01, 2008 7:27 pm

That's awesome. I might grab a 2nd GPU just for folding. D:
But I don't want to do that until I see if the ATI version starts scaling better on the 4800 series.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Thu Jul 03, 2008 10:55 am

It appears to be quite a beast, too. GX2's are cleaning up on ppd.
Sony a7
Sony Zeiss 55/1.8 SSM, 24-70/4 SSM
Minolta 100/2, 100-300 APO
TheEmrys
Minister of Gerbil Affairs
Silver subscriber
 
 
Posts: 2184
Joined: Wed May 29, 2002 8:22 pm
Location: Northern Colorado

Re: Nvidia Multi-GPU support

Postposted on Sat Jul 05, 2008 2:15 pm

When folding with multiple Nvidia cards, is SLI off or on?

I have a board with two 16xPCI express slots but it's Crossfire. If they don't need to be SLI then I'd grab a 2nd GPU.
8800gts $164 after rebate
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Sat Jul 05, 2008 2:31 pm

From my understanding it looks like it isn't for SLI, just multiple GPU's in one system.
Tachyonic Karma: Future decisions traveling backwards in time to smite you now.
Convert
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3117
Joined: Fri Nov 14, 2003 6:47 am

Re: SLI

Postposted on Sat Jul 05, 2008 5:00 pm

SLI needs to be off as a you need to expose the separate GPU's to the 2 Fahcore.exe's.
Last edited by JPinTO on Sat Jul 05, 2008 7:48 pm, edited 2 times in total.
JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Sat Jul 05, 2008 7:47 pm

I've got this running properly on a 9800GX2. It involves a little messing around with shortcuts, but it is fairly straight forward.

Q6600 OC 2.9Ghz + 9800GX2 OC 675Mhz:
Linux SMP client on Vmware (Cores 0+1): 1800PPD
GPU2 client 1 (Core 2): 5200PD
GPU2 client 2 (Core 3): 5200PD
Total PPD: ~12000

Warning: the 9800GX2 is a very long and thick video board and requires a 6pin and 8pin PCIe power connectors.

I ran the clocks up to 725 mhz, and the GPU clients went to ~5500PPD, so the total PPD is closer to 13000. :D
JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Sat Jul 05, 2008 11:39 pm

JPinTO wrote:
I ran the clocks up to 725 mhz, and the GPU clients went to ~5500PPD, so the total PPD is closer to 13000. :D


Yowsers :o
Image
If you're not folding with your idle computer time you're not part of the solution.
Gerbil Jedidiah
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3292
Joined: Thu Jul 27, 2006 5:57 pm
Location: The Smoky Back Room

Re: Nvidia Multi-GPU support

Postposted on Sat Jul 05, 2008 11:54 pm

Nice setup there.
I wonder what would be better for power and heat for the points.
9800GX2 or 8800GTS SLI

Also, I thought I read that you could UNDERCLOCK the GPU's core and memory while OCing the shader clock on the Nvidia cards to gain PPD but loose the heat.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Thu Jul 10, 2008 2:08 pm

JPinTO:
Any chance you can PM me the steps involved to get your 9800GX2 to run on both cores of a single card? I've messed around with this for the last few hours and I cannot get it to work. Even when referencing the guide at XPCPUs.com
DriveEuro
Gerbil
 
Posts: 10
Joined: Tue Apr 29, 2008 10:02 pm

Re: Nvidia Multi-GPU support

Postposted on Thu Jul 10, 2008 8:19 pm

DriveEuro,

Here is my shortcut settings, if it's of any help:

[url]http://www.bimmerfest.com/forums/picture.php?albumid=45&pictureid=179[/url]
JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 11, 2008 11:12 pm

JPinTO wrote:DriveEuro,

Here is my shortcut settings, if it's of any help:

http://www.bimmerfest.com/forums/picture.php?albumid=45&pictureid=179

Thanks for the response but I was able to get it working on my own.
DriveEuro
Gerbil
 
Posts: 10
Joined: Tue Apr 29, 2008 10:02 pm

Re: Nvidia Multi-GPU support

Postposted on Sun Jul 13, 2008 8:16 am

The GPU2 client does not like to share a core with other tasks, as it appears to get no cpu cycles.

However, I assigned both GPU2 tasks to the same core and they both appear to be sharing the same core nicely without reducing the PPD. This may mean that I can run 2x SMP and 2x GPU2 on the same quad core.
JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Sun Jul 13, 2008 4:46 pm

JPinTO wrote:The GPU2 client does not like to share a core with other tasks, as it appears to get no cpu cycles.

However, I assigned both GPU2 tasks to the same core and they both appear to be sharing the same core nicely without reducing the PPD. This may mean that I can run 2x SMP and 2x GPU2 on the same quad core.
That would be awesome. I'll have to try that out when I get my cards up. Although it would be 2 GPUs and the console client rather than SMP (dual core CPU).
Running XP?
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Sun Jul 13, 2008 6:09 pm

I'm running XP.

I setup as follows:

Ubuntu SMP 1: Core 1,2 Priority Normal
Ubuntu SMP 2: Core 1,2,3 Priority Lower
GPU1: Core 4
GPU2: Core 4

It worked find for one WU, then GPU1 went back to core 3 and halted as it does not like to share with SMP2. You would have to manually reassign the GPU1 back to Core 4 on each WU. There is an "advanced" option on the Config screen that seems to be a core affinity setting but I don't have time to play with it right now.

ALso, this strategy renders the PC pretty much useless for any other uses other than folding.
JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 15, 2008 5:34 am

JPinTO wrote:I'm running XP.

I setup as follows:

Ubuntu SMP 1: Core 1,2 Priority Normal
Ubuntu SMP 2: Core 1,2,3 Priority Lower
GPU1: Core 4
GPU2: Core 4

It worked find for one WU, then GPU1 went back to core 3 and halted as it does not like to share with SMP2. You would have to manually reassign the GPU1 back to Core 4 on each WU. There is an "advanced" option on the Config screen that seems to be a core affinity setting but I don't have time to play with it right now.

ALso, this strategy renders the PC pretty much useless for any other uses other than folding.


I don't have the GPU2 client halting problem. Maybe it's because I only have one GPU2 client. Here are my settings:

Ubuntu SMP 1: All cores
Ubuntu SMP 2: All cores
GPU2 client: All cores

They all seem to be playing well together. I have Vista64. Any idea why this works on my system while other have problems?
Image
If you're not folding with your idle computer time you're not part of the solution.
Gerbil Jedidiah
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3292
Joined: Thu Jul 27, 2006 5:57 pm
Location: The Smoky Back Room

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 15, 2008 11:13 am

Well since you have Vista your GPU2 only needs like 6-7% cpu usage and VMware leaves more than that unused by itself. On my setup I can't do that because the GPU2 client needs 100% of 1 core in XP but VMware doesn't give it any more room. VMware runs on Normal or Low while GPU2 is idle priority. So when I tried that my GPU2 client got only 12% cpu usage on the quad when it needed 25% for full performance under XP.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 15, 2008 12:16 pm

Pegasus wrote:Well since you have Vista your GPU2 only needs like 6-7% cpu usage and VMware leaves more than that unused by itself. On my setup I can't do that because the GPU2 client needs 100% of 1 core in XP but VMware doesn't give it any more room. VMware runs on Normal or Low while GPU2 is idle priority. So when I tried that my GPU2 client got only 12% cpu usage on the quad when it needed 25% for full performance under XP.

Since I have one Folding VM so I just manually lower the priority of vmware-vmx.exe. I think you can specify priority in the .vmx file as well.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24493
Joined: Mon May 24, 2004 2:19 am

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 15, 2008 6:47 pm

On my setup I can't go through task manager and change priority. It's locked or something. It gives me an error (operation cannot be completed. access denied). I'm running the correct user name and with admin. Probably have to do it through the vmx file. BUT that wouldn't help under XP since the GPU2 requires 100% of one core and both VMware setups don't give up that much CPU time on their own. Like 10% each. Might as well stack them. If you let them all share CPU time under XP then the GPU2 client will suffer and that is where most of your points are. right?
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 15, 2008 7:16 pm

Pegasus wrote:On my setup I can't go through task manager and change priority. It's locked or something. It gives me an error (operation cannot be completed. access denied). I'm running the correct user name and with admin. Probably have to do it through the vmx file. BUT that wouldn't help under XP since the GPU2 requires 100% of one core and both VMware setups don't give up that much CPU time on their own. Like 10% each. Might as well stack them. If you let them all share CPU time under XP then the GPU2 client will suffer and that is where most of your points are. right?


That's probably due the fact that you have VMware running under the local system account. Right-click on the vm, choose 'Settings', choose 'Options' tab, choose 'Startup/Shutdown', and on the side of the window, there should be options under 'Run this virtual machine as:'. I believe choosing either 'User that powers on the virtual machine' or 'This user' (which requires the Windows account password), would then enable you to set the affinity in Task Manager. One more thing, you need to power down the vm first before you can change the settings.
Fold! And I don't mean your clothes!

Do you have a favorite gerbil recipe? Please share with the TR community!
flybywire
Gerbil Jedi
 
Posts: 1883
Joined: Wed Jun 16, 2004 2:28 pm
Location: Springfield, VA - USA

Re: Nvidia Multi-GPU support

Postposted on Tue Jul 15, 2008 7:21 pm

Can you change the affinity in task manager?

That's more important than changing the priority.
JPinTO
Gerbil Team Leader
 
Posts: 240
Joined: Sat Jun 30, 2007 7:02 am
Location: Toronto, Ontario

Re: Nvidia Multi-GPU support

Postposted on Wed Jul 16, 2008 2:14 pm

I can't change affinity via task manager. Probably because of the same reason. I edited the vmx files to change affinity.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Thu Jul 17, 2008 10:39 pm

I have 2 8800GTs (256MB) running on my X2 4600+ system for TR. I believe in the log file it said that it automatically downloaded the new core (1.07?) although I'm not sure how to check the version. If it doesn't have the new core this might be the problem.

Running both on 1 core gives me 3700 ppd per GPU. When I manually set affinity in task manger so they use different cores then PPD goes up to 4200 per GPU. Sometimes the affinity sticks even when I turn in a WU and it starts up a new one. But eventually affinity goes back to CPU1 while CPU0 is doing nothing and my PPD drops back to 3700 in FahMon.

I've read about environment variables (NV_FAH_CPU_AFFINITY) but I have not got it to work. Does anyone know enough about them to walk me through?
I have the -gpu 0 and -gpu 1 tag in the shortcuts. And I have them with separate folders in Program Files and App Data. I have even checked "Do NOT lock cores to specific CPU" in config for both GPU2 clients.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 12:20 pm

I'm not quite following this multi-GPU thing, since I'll never have more than one, but don't you want to UNcheck that "Do NOT lock cores to specific CPU" thing so that you can keep them from both being run from one CPU core?
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5355
Joined: Sun Jan 20, 2002 7:00 pm

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 5:33 pm

Well, I don't care if each of them run on ALL cores OR if -gpu 0 runs on CPU0 and -gpu 1 runs on CPU1. But what I don't want and what is happening is both -gpu 0 and -gpu 1 are running on CPU1 ONLY. CPU0 is 0% usage.
The reason I don't want them running on 1 CPU core is that I loose 1000 points total doing it this way. For now I'm running a console client to gain some of that back but it doesn't fill even half the points lost.

I've read that the new core (1.07) is in the install file now. I think I'm going to completely remove my GPU2 clients on that machine and do a fresh install.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 5:40 pm

Pegasus wrote:Well, I don't care if each of them run on ALL cores OR if -gpu 0 runs on CPU0 and -gpu 1 runs on CPU1. But what I don't want and what is happening is both -gpu 0 and -gpu 1 are running on CPU1 ONLY. CPU0 is 0% usage.
The reason I don't want them running on 1 CPU core is that I loose 1000 points total doing it this way. For now I'm running a console client to gain some of that back but it doesn't fill even half the points lost.

I've read that the new core (1.07) is in the install file now. I think I'm going to completely remove my GPU2 clients on that machine and do a fresh install.


Please let us know if this improves PPD :D
Image
If you're not folding with your idle computer time you're not part of the solution.
Gerbil Jedidiah
Grand Gerbil Poohbah
Gold subscriber
 
 
Posts: 3292
Joined: Thu Jul 27, 2006 5:57 pm
Location: The Smoky Back Room

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 5:47 pm

This **** PHP behavior is pissing me off. It's eaten 2 posts of mine in the last few days by making me remove quoted text from a damned PM over an arbitrary 3 quote limit and I didn't wait forever to read its error message to save/send my original submitted PM, and just now it ate my reply to Pegasus because it decided informing me of a post added before I finished submitting mine was more important than following my command and apparently gave me another idiotic decision which I failed to wait for, since I always hit "Back" twice after posting so that I don't have to wait for the back-and-forth communication to put me there.

Not that I'm pissed off or anything. Fricking morons. :evil:

Ahem. :)

What I replied with was:

The "Do NOT lock cores..." thing would seem to me to be something you do not want to check, because it thwarts the configuration change you made to specifically assign each GPU core to a specific CPU core.

Not that I know w/o having investigated, but that's how it reads to me.

**** PHP.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5355
Joined: Sun Jan 20, 2002 7:00 pm

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 5:59 pm

My assumption is that you're running the GPU clients the way I run the one I am running, which is to say sitting at a Windows desktop.

From there, one assumes you change the affinity of each Folding@Home.exe to a particular core.

I'm still thinking there's probably a loss of potential output with the CPU not doing SMP folding, but that's a lesser concern for the time being.
Ragnar Dan
Gerbil Elder
Silver subscriber
 
 
Posts: 5355
Joined: Sun Jan 20, 2002 7:00 pm

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 6:08 pm

david00214 wrote:
Pegasus wrote:Well, I don't care if each of them run on ALL cores OR if -gpu 0 runs on CPU0 and -gpu 1 runs on CPU1. But what I don't want and what is happening is both -gpu 0 and -gpu 1 are running on CPU1 ONLY. CPU0 is 0% usage.
The reason I don't want them running on 1 CPU core is that I loose 1000 points total doing it this way. For now I'm running a console client to gain some of that back but it doesn't fill even half the points lost.

I've read that the new core (1.07) is in the install file now. I think I'm going to completely remove my GPU2 clients on that machine and do a fresh install.


Please let us know if this improves PPD :D

I can manually set each GPU to it's own CPU affinity under task manger but it gets reset eventually.
Both get 4200 when they use their own CPU.
Both get 3700 when sharing a CPU core.

This is all under XP fyi.

Ragnar Dan wrote:What I replied with was:

The "Do NOT lock cores..." thing would seem to me to be something you do not want to check, because it thwarts the configuration change you made to specifically assign each GPU core to a specific CPU core.

Not that I know w/o having investigated, but that's how it reads to me.

**** PHP.

ATM I have that option unchecked. Yesterday I was trying either checked or unchecked and it seemed like they still went back to CPU1 after some time. I have not figured it out yet. hehe
Thanks for the suggestions.

ATM I'm waiting for one to turn in it's WU since it's almost done. Then I'm going to reinstall the GPU2 clients.
Ragnar Dan wrote:I'm still thinking there's probably a loss of potential output with the CPU not doing SMP folding, but that's a lesser concern for the time being.
Loss in what way?
This is running on a dual core CPU under XP btw.
i3-530 | HR-01 Plus (passive) | DH55TC | 4GB Kingston DDR3 | Toshiba 250GB 2.5" HDD | Mini P180 | picoPSU 150-xt w/102w brick | 21w idle
Pegasus
Gerbil First Class
 
Posts: 161
Joined: Fri Jun 30, 2006 1:13 am

Re: Nvidia Multi-GPU support

Postposted on Fri Jul 18, 2008 6:18 pm

Ragnar Dan wrote:From there, one assumes you change the affinity of each Folding@Home.exe to a particular core.

Fah.exe does not do much, it is those FahCore_*.exe that are the main workers. If you are manually setting things you should mess with those in Task Manager.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24493
Joined: Mon May 24, 2004 2:19 am

Next

Return to TR Distributed Computing Effort

Who is online

Users browsing this forum: No registered users and 1 guest