TR Forums

TR says it all right here.

Now admittedly CPU-Z was never a proper benchmarking tool, which is why TR wisely avoided using it, but that's still no excuse for "optimizing" your way to success in microbenchmarks by intentionally ignoring the program code. Especially when those optimizations could come back to bite you in subtle and unpredictable ways when running a real-world program that doesn't work with your "optimizations".

I find it highly unlikely that AMD took their sparse engineering dollars to modify silicon to cheat their way in an obscure benchmark that no one really uses. I think if they had done that they would have touted those numbers or intentionally leaked them prior to release.

Losergamer04 wrote:

I find it highly unlikely that AMD took their sparse engineering dollars to modify silicon to cheat their way in an obscure benchmark that no one really uses. I think if they had done that they would have touted those numbers or intentionally leaked them prior to release.

They didn't modify their silicon (at least I doubt they did).
However, they probably put in a microcode check that engaged "optimizations" on the particular sequence of instructions in the CPU-Z microbenchmark, which is one reason why microbenches can be so misleading.

If you recall a while back when RyZen was discovered to freeze up when given a specific sequence of instructions, the fix was to update the microcode to avoid freezing the CPU. Well, the exact same mechanism can be used to game the system.

Once again, while TR is wise enough not to use CPU-Z as a benchmark, the same cannot be said for mainstream publications. I won't post the link but if you search for PC magazine's RyZen review you'll see our pal RyZen beating an i7 7700K at single-threaded CPU-Z right at the top of their benchmark results. You can darn well bet that AMD wanted that result.

There's no cheating here. There's literally zero chance AMD spent any amount of time or money to specifically target CPU-Z (of all things) to "cheat" their way to a higher score.

This is an optimization that resulted in faster execution of this particular set of instructions - it's there to accelerate it wherever it pops up. These types of optimizations can be dangerous, which is why microcode updates are a great thing.

Disclaimer: personal, non-editor opinion.

For the sake of argument, let's assume there's some nefarious plot here. Why the heck would AMD bother working out a way to cheat at... CPU-Z? Talk about not shooting for the moon.

A bit of code that works in a certain way with Ryzen happened to be in CPU-Z. End of story.

Yeah, the case for intentional cheating is pretty thin here.

Lest we go down the rabbit hole of when an optimization of an uncommon sequence of instructions becomes a cheat let me be the first to say I don't care. In a perfect world nobody would cheat, and everyone would have all the freely available and unbiased information they needed for their own buying decisions.

... Yeah, that's not the way things are, and the PC industry is actually pretty tame about it compared to many others. That's exactly the reason why reputable sources of information like TR are so valuable.

morphine wrote:

Disclaimer: personal, non-editor opinion.

For the sake of argument, let's assume there's some nefarious plot here. Why the heck would AMD bother working out a way to cheat at... CPU-Z? Talk about not shooting for the moon.

A bit of code that works in a certain way with Ryzen happened to be in CPU-Z. End of story.

As I pointed out above, PC Magazine, which is a large mainstream publication, listed CPU-Z as the very first benchmark in its RyZen review.
Don't think for one second that AMD isn't cognizant of that fact.

Additionally, a micro-benchmark like CPU-Z is exactly the easiest thing to fool since it's relying on extremely compact and deterministic code sequences that are very easy to spot. The same tricks won't work on larger and less deterministic software, which is basically what the CPU-Z developers found upon further testing.

I have been using CPU-Z/CPUID since my Socket A days 15 years ago and I wasn't even aware there was a benchmark. I thought it was just an overclock validation and test-bench stress tool.

So, I bother looking at the bench tab and see there is in fact a benchmark in there now. How long has that been there?

Come on chuck. This one is a bit too thin to call it "caught cheating."

DancinJack wrote:

Come on chuck. This one is a bit too thin to call it "caught cheating."

I'm just applying the same standards to AMD that regularly get applied to AMD's competitors around here. If AMD is back to being some type of top-dog in the CPU/GPU world then they'll have to deal with the fallout of stuff like this just like anybody else will.

chuckula wrote:

DancinJack wrote:

Come on chuck. This one is a bit too thin to call it "caught cheating."

I'm just applying the same standards to AMD that regularly get applied to AMD's competitors around here. If AMD is back to being some type of top-dog in the CPU/GPU world then they'll have to deal with the fallout of stuff like this just like anybody else will.

I know, but no need to act like a fAnBoI to get the point across. I get it though, I do. Hell, maybe they DID cheat, but who really cares beyond the fanatics on either side?

My 6700K @ 4.5 cares not even a little bit about this.

Without making any assumptions or drawing any conclusions about what this does or does not mean for AMD's integrity, I gotta go with the others in thinking that this was not intentional. TBH if I was reading a serious CPU review that lead with a CPU-Z benchmark I would probably stop reading there. I don't mean any slight to CPUID either - I don't feel like it was ever portrayed as a robust synthetic benchmark to begin with.

And @ Chrispy_ - I've known about it for a while but I learned about it in pretty much the same way

Chrispy_ wrote:

I have been using CPU-Z/CPUID since my Socket A days 15 years ago and I wasn't even aware there was a benchmark. I thought it was just an overclock validation and test-bench stress tool.

So, I bother looking at the bench tab and see there is in fact a benchmark in there now. How long has that been there?

There wasn't always a benchmark. I can't remember when they added it, but it was after the release of Nehalem.

Good god man you have issues, I read the original thread and then saw this post and said to myself could it be...and sure enough here you are.

I only recently even noticed the cpus benchmark as I am well aware of the tried and true benchmarking utilities...but AMD chose this one to reach the masses and manipulate them.

I couldn't make this **** up if I tried.

"As I pointed out above, PC Magazine, which is a large mainstream publication, listed CPU-Z as the very first benchmark in its RyZen review.
Don't think for one second that AMD isn't cognizant of that fact."

So AFTER the release of RyZen AMD chose to alter something, is this what you are trying to imply lololol.

Well, the good news is that if I ever need some dirt on AMD for forum arguments, Chuckula will be there to help out.

In fact, I'll bookmark this thread for (much) later use.

Your own comment from that TR post:

chuckula wrote:

Well the CPU-Z benchmark was pretty much worthless before this update and if it's only using 16 year old SSE2 instructions from 2001 and couldn't even be bothered to use 6 year old AVX instructions from 2011, then it's at least consistently worthless.

End User wrote:

Your own comment from that TR post:

chuckula wrote:

Well the CPU-Z benchmark was pretty much worthless before this update and if it's only using 16 year old SSE2 instructions from 2001 and couldn't even be bothered to use 6 year old AVX instructions from 2011, then it's at least consistently worthless.

Yes, and I'm absolutely right.
But the fact that I personally don't think CPU-Z is a worthwhile benchmark has nothing to do with the suspicious behavior of AMD's chips while running that benchmark.
Once again, PC MAGAZINE which is read by a large number of people who don't have my level of technical expertise, listed CPU-Z AS THE VERY FIRST PERFORMANCE BENCHMARK THAT ANYBODY READ IN THEIR REVIEW. And that's not the only website on the internet that used CPU-Z too, it's just an easy example.

So to recap:
1. Yes, I'm right in ignoring CPU-Z as benchmarking tool, and you should too [You should see how I attack Geekbench and BTW Geekbench was not overly kind to RyZen, but that doesn't mean I'm a Geekbench fan all of the sudden]
2. The fact that I'm personally right unfortunately doesn't mean everyone else is right too.
3. The fact that I'm right is not proof that AMD didn't game the benchmark.
4. The fact that I'm right doesn't prevent major online publications from still using CPU-Z even if it is stupid.

Hmm..... Cage Match: Chuckula vs. Ronch

My money's on Chuck. He knows how to fight dirty.

Chuck, your trolling is going a bit far on this. It's at times humorous and well pointed but this is a but much, even you. There have been articles and forums that talk about how AMD did predictors in this chip debate hour it helps and hurts it. This is probably a result of that. I'm on mobile and don't feel like taking my more time away from my kids to feed this troll any longer. I suggest we all do the same. Who is more foolish, the fool out the wise man arguing with the fool?

Ordinarily, that kind of automatic optimization would be welcome, but upon further investigation, the CPU-Z team failed to replicate that behavior with Ryzen CPUs in real-world situations. Furthermore, the team says that due to the extreme unlikelihood of that specific sequence of instructions showing up in non-benchmark software, it felt it would be best to revise CPU-Z to reflect real-world results more accurately.

This is the problematic line.

This distinctly reminds me of Bob Colwell and his lecture at Stanford. He regaled the class with a tale from when the Itanium team cheated at SPEC by hand optimizing an instruction loop. Bob called the Itanium team out and told them that if they could show him a compiler reaching the same conclusion about the series of instructions he would rescind his complaint. The Itanium team couldn't meet Bob's request though, it was too early in the project for such a thing.

Unfortunately the non-engineering management in the room didn't understand what was going on and ultimately the situation didn't raise a red flag like it should have.

What Bob's story demonstrates is that we can't just pawn this off as Hanlon's razor. While there's still a possibility that this happens to be a complete coincidence. There's also a possibility that some engineer did in fact make this tweak.

On the bright side, CPU-Z has put in a more realistic bit of code. And, that the result is probably not really a good benchmark. Benchmarking with multiple specific apps gives a better view of performance. The reviewer still has to explain why some CPU's are faster or slower because of hardware optimizations. Which is why we all prefer TR's reviews.

LOL classic Chucker, AMD optimize their CPU and he/she screams "cheating", typical, if Intel did that he/she'd would be waxing lyrical about how much effort they put into their chips.

Ryu Connor wrote:

While there's still a possibility that this happens to be a complete coincidence. There's also a possibility that some engineer did in fact make this tweak.

I highly doubt this is a case of AMD optimization intending solely to accelerate a microbenchmark. There's literally zero incentive for them to do so, and they run the risk of making particular code paths unstable if it really is a "cheat".

If it's 100% stable, it's purely a case of a microbenchmark not being representative of performance...which almost all of them fall prey to eventually.

The most likely scenario is that the benchmark was doing some calculations and throwing the results away (e.g. saving the results in machine registers and not storing them to RAM before overwriting them with something else). The CPU noticed the calculations were pointless, and skipped them. This is a perfectly valid thing to do, since it can speed up poorly coded applications (or applications compiled with less-than-optimal compilers), and does not affect the output of a program that actually does something useful. No nefarious intent required.

To avoid this sort of thing, benchmark code needs to use instruction sequences which resemble those which would be used in real-world applications, instead of sequences of instructions which are "unlikely to appear in non-benchmark software".

just brew it! wrote:

The most likely scenario is that the benchmark was doing some calculations and throwing the results away (e.g. saving the results in machine registers and not storing them to RAM before overwriting them with something else). The CPU noticed the calculations were pointless, and skipped them. This is a perfectly valid thing to do, since it can speed up poorly coded applications (or applications compiled with less-than-optimal compilers), and does not affect the output of a program that actually does something useful. No nefarious intent required.

To avoid this sort of thing, benchmark code needs to use instruction sequences which resemble those which would be used in real-world applications, instead of sequences of instructions which are "unlikely to appear in non-benchmark software".

Transmeta's Crusoe did exactly this and has similar 'issues' producing erroneous benchmark results in some select cases. The 'issue' was that the benchmark wasn't doing any meaningful work and thus decides to skip executing it entirely.

chuckula wrote:

DancinJack wrote:

Come on chuck. This one is a bit too thin to call it "caught cheating."

I'm just applying the same standards to AMD that regularly get applied to AMD's competitors around here. If AMD is back to being some type of top-dog in the CPU/GPU world then they'll have to deal with the fallout of stuff like this just like anybody else will.

In this context, there's only one competitor to which some double standard could be applied with regularity, as you suggest, and that's Intel. Why bother with such transparent misdirection?

Pretending that you're speaking on behalf of so many others [competitors] to demonstrate the equity of your thoughts isn't convincing, particularity given the rest of your reasoning isn't. That is, I doubt it could convince most of the people that come here to read this type of post.

just brew it! wrote:

The most likely scenario is that the benchmark was doing some calculations and throwing the results away (e.g. saving the results in machine registers and not storing them to RAM before overwriting them with something else). The CPU noticed the calculations were pointless, and skipped them. This is a perfectly valid thing to do, since it can speed up poorly coded applications (or applications compiled with less-than-optimal compilers), and does not affect the output of a program that actually does something useful. No nefarious intent required.

To avoid this sort of thing, benchmark code needs to use instruction sequences which resemble those which would be used in real-world applications, instead of sequences of instructions which are "unlikely to appear in non-benchmark software".

Exactly. It's a useful optimization for all sorts of bad code.

Waco wrote:

There's literally zero incentive for them to do so, and they run the risk of making particular code paths unstable if it really is a "cheat".

I'm not sure why you're saying that. The Itanium engineers were purposefully cheating to present super rosy results to their bosses.

Yet you say there's zero incentive? Clearly there is an incentive. Clearly it's actually happened in the real world in the past.

You think this is a coincidence? Fine. But I wouldn't use that line of reasoning. This is a terrible hill to choose to die on.

To be honest I was oblivious to CPU-Z having a benchmark. I've never seen it mentioned in any review from any worthy site.