Personal computing discussed
MrJP wrote:Does anyone else feel even slightly apprehensive that sites are going to use a tool supplied by nvidia as their source of benchmark results?
Firestarter wrote:MrJP wrote:Does anyone else feel even slightly apprehensive that sites are going to use a tool supplied by nvidia as their source of benchmark results?
Slightly. I mean, it does highlight a problem with AMD's cards, but then again if AMD hadn't already had their own toolset to identify the problems, they sure do have one now.
CampinCarl wrote:Firestarter wrote:MrJP wrote:Does anyone else feel even slightly apprehensive that sites are going to use a tool supplied by nvidia as their source of benchmark results?
Slightly. I mean, it does highlight a problem with AMD's cards, but then again if AMD hadn't already had their own toolset to identify the problems, they sure do have one now.
Also, I'd like to think that AMD's engineers could easily reverse engineer everything in the tools suite (hardware, software) to make sure that there's no funny business. If they do find it, then they get to use it as PR against nVidia. Like I said on the front page comment thread, if this type of thing is the way forward (until we get to the point where, as Scott mentioned, both the GPU devs and game devs give benchmarkers and the public at large access into the 'chain' to take measurements), then the whole suite should be an FOSS suite.
cobalt wrote:I quite liked what they did for a little while there with cross-cutting comparisons. Like in pages 4-6 of this article:
http://www.anandtech.com/show/2745/4
They show a little higher-level analysis pretty well. For example, instead of simply showing one-game-detail-per-page, they have one page devoted to 4890 vs GTX 275 across games and resolutions, and then on another page they show what performance hit you take if you drop down $70 to the 4870, and then on another page show the $180 shootout between the 4870 and the 260 core 216. But I haven't seen that kind of higher-level summary from them in a few years.
MrJP wrote:cobalt wrote:I quite liked what they did for a little while there with cross-cutting comparisons. Like in pages 4-6 of this article:
http://www.anandtech.com/show/2745/4
They show a little higher-level analysis pretty well. For example, instead of simply showing one-game-detail-per-page, they have one page devoted to 4890 vs GTX 275 across games and resolutions, and then on another page they show what performance hit you take if you drop down $70 to the 4870, and then on another page show the $180 shootout between the 4870 and the 260 core 216. But I haven't seen that kind of higher-level summary from them in a few years.
Yes, but that article is knocking on four years old, and co-authored by Anand. I've not been nearly so impressed with anything more recent, to the point where they are probably only about fourth or fifth in the list of sites I'd look to for a GPU review. Don't get me wrong, they're still competent and unbiased (sadly not as common as you'd hope in tech sites), I just think that as they've grown they've let this area slide relative to their previous high standards.
This new FCAT study is a case in point. It was interesting to read about this over at Anandtech earlier today, but the article (and the earlier one about AMD and FRAPS) just never really seemed to get into the problem as I was expecting. I accept there's a Part 2 yet to come, but as it stands it's almost like they are happy to just parrot back what AMD and Nvidia have been telling them.
Now I check back here this evening and find Scott's new article. What a contrast. Far more incisive, and really getting into the crux of the problem without taking anything at face value. Impressive stuff.
Guru3D wrote:Mind you that Average FPS matters more then frametime measurements. It's just an additional page or two of information that from now on we'll be serving you.
MrJP wrote:Guru3D wrote:Mind you that Average FPS matters more then frametime measurements. It's just an additional page or two of information that from now on we'll be serving you.
Well they just don't get it at all.
MrJP wrote:Guru3D wrote:Mind you that Average FPS matters more then frametime measurements. It's just an additional page or two of information that from now on we'll be serving you.
Well they just don't get it at all.
cynan wrote:The only reason why FPS is considered by some to be more "important" is because A) It's a simpler concept to conceptualize and therefore more accessible and B) It's the status quo.
Captain Ned wrote:cynan wrote:The only reason why FPS is considered by some to be more "important" is because A) It's a simpler concept to conceptualize and therefore more accessible and B) It's the status quo.
And it's better when bigger instead of 99th percentile time where better is smaller. Far too many purchasers have invested far too much internetz in having the bigger number.
MrJP wrote:But if you like big numbers you can easily convert the 99th percentile frametime into an equivalent instantaneous FPS.
Damage wrote:Meet The Tech Report:
http://techreport.com/review/24553/insi ... ture-tools
Captain Ned wrote:Ugh. Yes, you can, but when you do you still exclude the huge outliers that kill the gaming experience and that Damage has so easily dealt with in the first page of any "Inside the Second" article. Averages hide extremes and the new paradigm in GFX reviews is to find the extremes.
MrJP wrote:Sorry for going further off-topic, and I appreciate this has been discussed in previous threads, I just feel that a lot of people who are against this suggestion are misinterpreting it as a desire to go back to the old average FPS analysis. Nothing could be further from the truth.
MrJP wrote:I think you misunderstand. I'm not talking about going back to the average FPS numbers which would throw away all the useful data. I'm only talking about changing the axis in the plot to frames/second rather than seconds/frame. You still plot the datapoint for every individual frame, you just present it as equivalent instantaneous frames/second simply by taking the reciprocal: instantaneous frames/second = 1/frametime. As in the table that you often see at the start of the reviews, 10ms=100fps, 16.7ms=60fps, 20ms=50fps, 33ms=30fps and so on. This is just a question of units, not changing the data that is presented.
Sorry for going further off-topic, and I appreciate this has been discussed in previous threads, I just feel that a lot of people who are against this suggestion are misinterpreting it as a desire to go back to the old average FPS analysis. Nothing could be further from the truth.
Damage wrote:
southrncomfortjm wrote:Damage wrote:
My main takeaway from that TR article is that FRAPs is pretty decent at doing what we all wish it did. Its a close enough approximation of what we want that we can use it without being too far off base. Good interpretation?
I hope that's right. I read the article late at night and haven't had a chance to get back to it.
MrJP wrote:The plots still show the same frame-to-frame variance, spikes, etc it's just done in a way where the higher-performing parts are towards the top of the graph, and people can look at it and make straightforward judgements like "this one hovers around 60fps but often spikes down to 10fps" rather than "this one hovers around 16ms frametime but often spikes up to 100ms". See how that's more easy to relate to for most gamers?
Waco wrote:People want a simple metric and they have grasped the notion of FPS pretty well by now. Presenting the same data in a different way (arguably in a non-sensical format, but oh well) that makes it more understandable to the masses is probably not a bad thing.
Captain Ned wrote:I've traded a few PMs with Damage where I support something along the lines of the way audio amplifiers are marketed, i.e. 200 watts per channel at .01% Total Harmonic Distortion. For GFX cards, it would be 90 FPS at a 99th% latency of X with the lower X being better. The bitch of my idea, though, is deciding upon the standardized test, as unless there's a standard test there's no common frame of reference.
MrJP wrote:You guys make some good points. We might just have to agree to differ.
I'll just make a last stab at it with two final arguments:
1. What are we ultimately trying to check here? Smoothness of animation. How is that usually measured in film and TV (as well as in computer monitor refresh rate)? Frames per second (or Hertz to be more technically correct).
The key thing in all of this is not how long a single individual frame takes to be processed, it's whether the overall rate of frames is high enough and the variance low enough for the illusion of movement to be convincing. Traditional "FPS" measured over a whole benchmark gives a (less than perfect) measure of the former, and the frametime analysis allows us to check the latter. Indeed, both bits of information drop out of the percentile plot, which is why I think it's the most informative bit of the new-style reviews. However ultimately since it's the rate of the image update that we're interested in, then it makes sense that the results should be presented as a rate, even when it's been derived from individual frames. So frametimes should be converted to frames/second (or better yet Hertz, but then we're back in to the unfamiliar).
2. OK so you don't buy that one. You still think that we should be talking in terms of frametimes as being the final measure. So why do other plots in the review still use frames/second? Why the inconsistency?
At the moment, the traditional FRAPS average FPS results are still plotted in the review as frames/second. The final value scatter plots are plotted with frames/second (converted instantaneous FPS for those that think that's impossible!). This straight away makes it hard to compare the results from the two types of analysis. It's trivial to convert the FRAPS average FPS into an equivalent overall average frametime (frametime=1/FPS). If frametime is the final measure we're interested in, then shouldn't everything be presented in this way?
I suspect the truth is that things have been presented this way because that's how FRAPS writes them out. Average frame rate is presented in frames/second, and the individual frame times are reported in ms. Please don't take this as an intended criticism because I think the reviews are fantastic, and ultimately the content is the draw rather than the way its presented.
And getting back on topic, TR > Anandtech.