TR Forums

A discussion in another thread got me thinking about how TR benchmarks CPU performance in games.

These result don't appear to reflect real world gaming performance whereas these results do (full post). Thoughts?

Yea, here's a thought: games are a poor "benchmarking" tool unless used with a specific benchmarking utility which can use same (the most demanding) exact in-game level with same exact in-game actions during the same exact time period. If every reviewer just logs into the game and just randomly hops around at a random in-game location - you'll get such inconsistently useless results as in your links. That Russian tester himself admits that the results might not be reliable because he used a multplayer portion (64-player server on Kiasar Railroad) for testing, which is even worse than using a singleplayer portion of the game.

Edit: same goes for the fact that game developers tend to release updates from time to time which may have significant effect on performance.

Interesting. The Russians are testing the DLC End War and multiplayer mode whereas I believe TR were using vanilla BF3 in single player mode. Not sure if the content being used has an impact but multiplayer should certainly hit the processor harder, especially the 64 player servers they used - the results back this up too.

I have to agree with you therefore. TR should be including more multiplayer testing in their CPU tests for more relevant results. Of course it's impossible to fully control all the variables in a multiplayer test and I'm not sure the best method to do it but the differences in results seems too big to ignore.

puppetworx wrote:

TR should be including more multiplayer testing in their CPU tests

Absolutely not, for the same reasons you have stated:

puppetworx wrote:

Of course it's impossible to fully control all the variables in a multiplayer test

unless you can somehow "record" a timedemo of multiplayer game using in-game tools.

JohnC wrote:

unless used with a specific benchmarking utility

Such as:



and



and



The above results are similar to the real world gaming benchmark results I was referring to:



The difference is striking:

Hmm. I don't read Russian and haven't slogged through the article carefully. Working today, so I don't really have time. I'll offer a few responses though.

It's true our test results in the particular case you linked came from the single-player game and not a multiplayer match with 64 people. We've documented how we tested--and how we test, which continues to evolve. (You cited some old-ish results from us.) These two test scenarios in BF3 are different, but both of them happen in the real world. I don't think one is more "real" than the other, although the multiplayer test may be more relevant to some folks.

I'm happy to see folks doing the hard work of multiplayer testing with lots of players on one server. That is a very tough thing to do properly. Since the play sessions aren't repeatable at all, you'd need to do lots of sampling in order to get good results. We've not been able to dedicate the time, along with the time of 63 very dedicated friends, to making it happen.

However, I'm a little sad to see such hard work boiled down to an FPS average as the end result. Doesn't really tell you what you need to know about comparative gaming smoothness. I think we offer more granularity with our frame-by-frame plots, our latency-focused metrics like "time beyond 50 ms," and with our latest addition: slow-mo videos, like we've used in our recent GPU reviews:

http://techreport.com/review/25167/fram ... on-hd-7990

We've not done smoothness videos in CPU reviews, but we could consider it in the future.

Are those BF3 results more "real" than our old BF3 results? Nope. Are they more relvant for BF3 multiplayer? Prolly. Are they helpful? Depends on whether they sampled sufficiently while testing, in part. Still, not as much as they could be, when you have only FPS avg/min as the result.

The TR's tests also show Intel products on top, slowly turning into AMD products at the bottom. The difference in FPS might be not so dramatic but once again, it depends on the test methods (such as which exact portion of the game was tested, what your character was doing during that test and how long the test has lasted), and each site has its own.

You don't even need to test each CPU individually with each game - you can create your own graphs based on benchmarks of one game with few CPUs (or perhaps on benchmarks already made by other site), then randomly "pad" the graphs with few extra FPSes (for more "dramatic" visual difference), give them different game names, add in "results" from the CPUs you've never actually benchmarked and call it a day

Heh, just realized it's *that* set of BF3 results from us. Yeah, not my favorite set of results, given how not-really-CPU-limited that test scenario turned out to be.

FWIW, we did find that the Pentium G2120 was pretty slow *in spite* of an FPS average that's similar to everything else:



So there's that. Heh.

I mean, I guess it's useful info, knowing that most CPUs don't have a problem with that test scenario, but it isn't a very good CPU stress test. Thing is, you can do some pre-testing, but you never really know for sure how the full scope of testing will come out until it's completed. In this case, it was kind of a bust.

I think our new results capture the CPUs facing some harder problems, and they offer more insight into the differences between CPUs:



But yeah, I wish I had the time and resources to do proper mega-multiplayer testing in properly in something like BF3. Maybe I can take a crack at it at some point soon. Maybe with BF4..?

I've yet to see a StarCraft 2 CPU test with 4x4 multiplayer with motherships involved. That will stress even a theoretical 10GHz i7 4960K.

StarCraft 2 tests are very easy to reproduce as well, thanks to replay functionality. You simply follow a specific player camera around the map.

Clearly, our Tomb Raider testing must be more "real-world" than AnandTech's.

Damage wrote:

You cited some old-ish results from us.

We were chatting about BF3 performance in another thread. Results cited from your 2012 FX-8350 review were referred to by another member and that got the ball rolling.

Damage wrote:

But yeah, I wish I had the time and resources to do proper mega-multiplayer testing in properly in something like BF3. Maybe I can take a crack at it at some point soon. Maybe with BF4..?

That would be cool.

Another CPU related topic that has been discussed is what influence the next gen consoles will have on PC gaming. Will the new consoles and their multi-threaded capabilities spike CPU usage in games across the board? Will 4 Intel cores be enough for some PC games this time next year?

Keep in mind that both of the high end spec wise major consoles this/last gen have at least six threads available...

End User wrote:

Another CPU related topic that has been discussed is what influence the next gen consoles will have on PC gaming. Will the new consoles and their multi-threaded capabilities spike CPU usage in games across the board? Will 4 Intel cores be enough for some PC games this time next year?

Tough to say, but recall that a Jaguar core like one of the eight in the PS4/Xbone isn't very powerful.

Take Cinebench as an example--flexes the FPU like a lot of games can. Four Jaguar cores, in a Kabini A4-5000 APU, score 1.5. Eight at the same clock would presumably score 3.0.

The scores for today's desktop CPUs:



That's without using AVX2 on the Haswell parts, which could improve scores by ~50-100%.

I think things get tougher, relatively speaking, for the Jaguar cores when they're asked to handle nasty, branchy game code with a mix of integer and FP math.

So my big worry is that games will be dumb, with few advances in things like AI and complex physics algorithms that don't map well to GPU hardware, given the limitations of the new consoles. I have little worry about a modern PC keeping up.

Although one should never underestimate the burden of DirectX overhead in Windows.

Damage wrote:

But yeah, I wish I had the time and resources to do proper mega-multiplayer testing in properly in something like BF3. Maybe I can take a crack at it at some point soon. Maybe with BF4..?

I would love to see that. A multiplayer test would make a good follow up to the Inside the Second: Gaming Performance with Today's CPUs article. I'm not saying it would be easy (which is why no one has really done it, no one to the caliber of TR that is) but if done right it would be very valuable. Having said that you're definitely in a better position to judge the opportunity cost of such a test.

Well, whatever multiplayer test you will do - it must be done on the same map with same amount of people, preferably with small area (so you can easily cover all of it on foot) and without vehicles. I just played several rounds of BF3 on different maps (but all with 64 people), my "Average" FPS (according to FRAPS) varied from 90 (on Noshahr Canals) to 115 (on Operation Metro) on same settings ("Ultra") and same exact resolution (1920x1080), using same exact hardware (i7-2600 CPU and EVGA Titan SC). Sadly I do not have different CPUs to test.

JohnC wrote:

Yea, here's a thought: games are a poor "benchmarking" tool unless used with a specific benchmarking utility which can use same (the most demanding) exact in-game level with same exact in-game actions during the same exact time period. If every reviewer just logs into the game and just randomly hops around at a random in-game location - you'll get such inconsistently useless results as in your links. That Russian tester himself admits that the results might not be reliable because he used a multplayer portion (64-player server on Kiasar Railroad) for testing, which is even worse than using a singleplayer portion of the game.

Edit: same goes for the fact that game developers tend to release updates from time to time which may have significant effect on performance.

That really takes me back to the days of Quake2 and those massive timedemos.
Really wish modern devs had thought to include replay functionality; would be a great help to players and benchmarkers alike.

Ok, just played a couple of full rounds on "Ziba Tower" map on a 64-player server... The results seem to be pretty consistent with each round - here are FRAPS numbers:

Round 1:
Min Max Avg
49 201 99.792

Round 2:
Min Max Avg
49 198 103.565

This is with Vsync disabled, on "Ultra" settings. The map is pretty small (no vehicles) but can pack a lot of people with a lot of action (due to small size), so maps like these should be a perfect test for CPU benchmarks for "multiplayer games". Hopefully BF4 will have same "Close Quarters" type of map available from the very beginning...

JohnC wrote:

That Russian tester himself admits that the results might not be reliable because he used a multplayer portion (64-player server on Kiasar Railroad) for testing, which is even worse than using a singleplayer portion of the game.

It may be "worse" in terms of providing consistent precise results, but multiplayer is usually by far the most demanding part of games. Single-player BF3 really can't be considered "real world" either, because no one plays BF3 for the single-player.

There's no real solution to that problem though. The very nature of multiplayer is that it is unpredictable and never the same, but single-player (which can have reliable testing done) usually doesn't push the hardware as much.

End User wrote:

A discussion in another thread got me thinking about how TR benchmarks CPU performance in games.

These result don't appear to reflect real world gaming performance whereas these results do (full post). Thoughts?

TR tested using single player campaign mode in which BF3 is strongly GPU bound. The second link is obviously online play in a 64 player mission which is makes most games CPU bound. Easy as that.

The problem with any benchmark is it will necessarily favor certain features, subroutines, architectures, whatever because that's what happens to be the bottleneck for the task at hand. The trick to benchmarks is to do a whole lot of them, as diverse as possible, so that things average out. I'd love to see TR benchmarks get a bit more exotic with less focus on mainstream games simply because mainstream games have a long history of being "optimized" for AMD or nVidia in exchange for a fat wad of cash.

Games that are CPU-bounded are typically limited by IPC and clockspeed. That's why there is almost no difference between the 4770K versus the newly released 4970X. The vast majority of games only take advantage of two threads and a vanishing small number only take advantage of four threads. It is also why AMD chips are on the bottom of the list, since they are inferior to Intel's process on an IPC basis.

NovusBogus wrote:

The problem with any benchmark is it will necessarily favor certain features, subroutines, architectures, whatever because that's what happens to be the bottleneck for the task at hand. The trick to benchmarks is to do a whole lot of them, as diverse as possible, so that things average out. I'd love to see TR benchmarks get a bit more exotic with less focus on mainstream games simply because mainstream games have a long history of being "optimized" for AMD or nVidia in exchange for a fat wad of cash.

The problem with what you are suggesting is that you still want to test features that are used in common games. It's great to find some corner case that no one uses if you just want to see where GPUs or CPUs fail to perform as expected (like testing 2D features now vs older GPUs) but if it isn't used in many games does it really matter to other gamers? After all people buy a CPU/GPU for particular uses. If a CPU/GPU performs well in the things they want to use it for does it matter if it does awful in some unused area?

Krogoth wrote:

The vast majority of games only take advantage of two threads and a vanishing small number only take advantage of four threads.

A very popular 2 year old game (BF3) is pushing my CPU to 60-80% CPU usage across all cores/threads when playing in multiplayer mode. Its successor is about to come out. I'd like to know if there is a better CPU out there for BF4 in multiplayer mode. Ideally the testing would be done at real world resolutions (1920x1080+) at high settings.

Moving forward a larger sampling of popular CPU crushing multiplayer games would be ideal when new CPUs are tested. Granted that is a lot to ask so it would be cool to break that out into a separate followup post published at a later date.

As far as GPU testing goes I'd like to see GPU memory usage info when new GPUs are reviewed.

NovusBogus wrote:

The problem with any benchmark is it will necessarily favor certain features, subroutines, architectures, whatever because that's what happens to be the bottleneck for the task at hand. The trick to benchmarks is to do a whole lot of them, as diverse as possible, so that things average out. I'd love to see TR benchmarks get a bit more exotic with less focus on mainstream games simply because mainstream games have a long history of being "optimized" for AMD or nVidia in exchange for a fat wad of cash.

Why wouldn't you want to test using mainstream games? You'll get actual data of what people will be experiencing when they play.

The main idea behind benchmarking is to give people an idea of how a particular part might perform before they buy it, so testing using the actual software they'll be using is the best way to do things.

NovusBogus wrote:

I'd love to see TR benchmarks get a bit more exotic with less focus on mainstream games simply because mainstream games have a long history of being "optimized" for AMD or nVidia in exchange for a fat wad of cash.

The reason to test mainstream games is because those are the games that the majority of gamers play (at one point or another). They also have the best chance of being well coded because of the larger budget and more manpower; which is better than being UN-optimized. Besides that, AMD and Nvidia drivers give more attention to optimizing mainstream games.

I'm not sure AMD/Nvidia-specific optimizations are as major as you're thinking they are. If you look at game benchmarks*, Nvidia and AMD cards generally tend to maintain performance hierarchy regardless of whether the game is a TWIMTBP or GE title (after initial launch bugs have been worked out). As I alluded to before, it's hard to tell how much of the differences are coming from the developer's game coding or from AMD/Nvidia driver optimizations. Both AMD and Nvidia have shown that graphics drivers can have HUGE effects on performance.
*I didn't search benchmarks for this specific relationship before posting this, just commenting on my recollection. You're welcome to prove me wrong.

Star Brood wrote:

I've yet to see a StarCraft 2 CPU test with 4x4 multiplayer with motherships involved. That will stress even a theoretical 10GHz i7 4960K.

StarCraft 2 tests are very easy to reproduce as well, thanks to replay functionality. You simply follow a specific player camera around the map.

This is actually a very good idea for a test, IMO. Motherships might be too much, but those replays would allow for repeatable testing in a multiplayer environment, which, I think, is what this thread really comes down to.

Damage wrote:

Hmm. I don't read Russian and haven't slogged through the article carefully. Working today, so I don't really have time. I'll offer a few responses though.

It's true our test results in the particular case you linked came from the single-player game and not a multiplayer match with 64 people. We've documented how we tested--and how we test, which continues to evolve. (You cited some old-ish results from us.) These two test scenarios in BF3 are different, but both of them happen in the real world. I don't think one is more "real" than the other, although the multiplayer test may be more relevant to some folks.

I'm happy to see folks doing the hard work of multiplayer testing with lots of players on one server. That is a very tough thing to do properly. Since the play sessions aren't repeatable at all, you'd need to do lots of sampling in order to get good results. We've not been able to dedicate the time, along with the time of 63 very dedicated friends, to making it happen.

However, I'm a little sad to see such hard work boiled down to an FPS average as the end result. Doesn't really tell you what you need to know about comparative gaming smoothness. I think we offer more granularity with our frame-by-frame plots, our latency-focused metrics like "time beyond 50 ms," and with our latest addition: slow-mo videos, like we've used in our recent GPU reviews:

http://techreport.com/review/25167/fram ... on-hd-7990

We've not done smoothness videos in CPU reviews, but we could consider it in the future.

Are those BF3 results more "real" than our old BF3 results? Nope. Are they more relvant for BF3 multiplayer? Prolly. Are they helpful? Depends on whether they sampled sufficiently while testing, in part. Still, not as much as they could be, when you have only FPS avg/min as the result.

I agree just having the final FPS tag does not fully disclose performance and the lags and spikes that go with it... but I would 1000% love to see cpu % usage detailed out on each test in future TR articles. It would be nice to see ''core utilization metrics'' in content creation, gaming.. it would give people a little more thought into their possible purchase if they could see processor-x in maya or bf4 can only use 4 cores 90% of the time and the remainders are idle. People could mre intelligently buy a cpu to their task/needs.

End User wrote:

Another CPU related topic that has been discussed is what influence the next gen consoles will have on PC gaming. Will the new consoles and their multi-threaded capabilities spike CPU usage in games across the board? Will 4 Intel cores be enough for some PC games this time next year?

Any i7-2600k, 3770k or 4770k 8 threaded cpu's will more then be able to handle AMD 8 dedicated cores in the new console's . Even a hyperthreaded core on a i7 is probably as powerful as a dedicated Temash core. Unless this HSA computing does something amazing.

Yea. Just because consoles have so many cores doesn't mean developers must utilize all of them or utilize them equally. Not to mention the fact that the XBone/PS4 needs to be able to utilize some of these cores for various misc tasks while you're playing, like capturing and encoding the video for streaming through Twitch/Ustream.

Speaking of which - you TR guys should also consider doing a more relevant benchmarks including streaming games through, for example, Twitch.tv. While not many people are motivated enough to do that right now - the next consoles should make this service/activity more popular by bringing in more advertisers to Twitch/Ustream (which will give more financial incentive to gamers). Currently no "popular" hardware review site does such benchmarks, you might be "pioneering" something yet again