TR Forums

G'day all,

I noticed some issues with the Frame Number charts in TR's recent GTX 1080 and RX 480 reviews, as well as the DX12 Inside The Second sneak peak. I think they detract from TR's otherwise good reputation, but they appear to have some simple fixes. I mentioned the issues in the comments thread on each post, as well as in an email to Jeff and Robert, but I haven't had any reply/acknowledgement. I couldn't find a suggestions box on the page and I'm not on Twitter, so hopefully this will reach them. I'd like to see TR go from strength to strength so figured I'd mention it again just in case it's slipped through the cracks.

With the intro done, here's the deal: the Frame Number plots in these recent posts could use some improvement. There is missing data, wasted space and various inconsistencies between tabs in a singe chart and across charts in a review.

From the GTX 1080 review:

GTA V: the GTX 1080 drew the most frames at somewhere under 4200, but the x axis ranges up to 6500 - this represents roughly 30% wasted space [assuming a chart limit at 4500 frames] with data compressed into the left hand side of the chart. Similarly, having applied the filter to the data, no card plotted a frame time above 40ms, but the y axis range goes up to 60ms. The GTX 1080 plot is only on one tab in this chart, where it is on both in other charts in this review.

Crysis 3: both x and y axis ranges could be brought back one notch, but otherwise this is a pretty good example of the style I think TR is going for: the card under test is on both tabs and gives a great reference point for comparing and contrasting cards.

Rise of the Tomb Raider: the GTX 1080 plot is not available on both tabs, and appears to be truncated [the data appears to continue past the chart boundary]. How much further did it go? A quick and nasty check against the GTX 980 Ti's ~3500 frames and ~58 FPS average suggests that the GTX 1080 might have produced ~4050 frames, but the chart cannot tell us. Similarly, just how bad are the R9 Fury and Fury X's frame time spikes? Again, they extend off the chart and are unknowable.

Fallout 4: this is possibly the worst of the lot. The y axis range goes to 120ms when no card cracks 40ms; he x axis range differs between the tabs; GTX 1080 plot appears to extend out of the chart on the Radeon R9 tab; the data is truncated on the GeForce GTX tab [both the GTX 980 Ti and GTX 1080 clearly produced more frames than their plots show] - the chart suggests very little difference in performance between GTX 980, GTX 980 Ti and GTX 1080 but every other chart proves the mistake.

The Witcher 3: GTX 1080 plot is missing from the Radeon R9 tab. Both the GTX 980 Ti and GTX 1080 plots are clearly being truncated in the x dimension. The Y axis could be brought back an increment.

Hitman: The GTX 1080 produces the most frames at something like 3000, but the x axis range goes to 6500 - less than 50% utilisation. At the same time, the Fury X's frame times clearly extend out the top of the chart - just how bad are they?

From the RX 480 review:

GTA V: the range of both x and y axes could be reduced to better fit the dataset

Crysis 3: different x axis ranges between tabs; does the GTX 980 plot finish within the chart area?

Rise of the Tomb Raider: x axis range could be reduced by one increment, but y axis range is clearly clipping the R9 380X dataset.

Fallout 4: y axis range could be reduced from 120ms to 40ms. X axis range differs between plots, appears to truncate both the RX 480 and GTX 970 plots on the RX 480/GTX 970 tab and also the GTX 980 plot on the GeForce GTX tab.

The Witcher 3: different x axis ranges between tabs. Y axis range appears to truncate data in both tabs. The R9 380X and RX 480 plots extend past the chart boundary on the RX 480/GTX 970 tab and also the GTX 980 plot on the GeForce GTX plot. Also, is the correct R9 380X dataset, or possibly GTX 970 dataset, being used for this chart? The R9 380X produces a lower average FPS, which would suggest it shouldn't produce a minimum of 200 more frames than the GTX 970 across 60s or so of game time.

Hitman: the x axis range could be brought back up to 4 increments instead of wasting ~30% of the chart width. If it hadn't been for that spike in the GTX 980 plot, the y axis could have been brought back also - as it stands, how big was that spike?

From the DX12 Inside The Second sneak peak: the x axis is great! The y axis truncates the GTX 970's dataset - just how bad are those [filtered] spikes?

Suggestion: for both axes and all tabs of a chart, set the range to be the increment above the largest value across the data set [i.e. all tabs within a given chart should have x and y axis ranges in common]. Clearly show that all datasets finish within the chart area. As far as possible, no dataset should be cut off by chart boundaries - if you do truncate data, make sure to mention what and why. Within each review, maintain consistency by selecting one card to appear as a reference on all Frame Number chart tabs across the review [should this always be the card being tested?].

Does that sound reasonable? Would it be possible for Jeff or Robert to acknowledge that they've seen this, even if they choose not to implement it?

They are trying to make all the charts comparable at a glance to have the same effective vertical scale so its easy for the reader to see when a game is really giving the video card a problem. If you expand the scales of each graph to fill the graph, you take away from the usability of the graphs for comparing across all the games.

I think they detract from TR's otherwise good reputation

lolwut?

Mr Bill wrote:

They are trying to make all the charts comparable at a glance to have the same effective vertical

That would be fine, but they are inconsistent in that. See here:

http://techreport.com/review/30281/nvid ... eviewed/10

Radeons chart ends at 4000, nvidia cards chart ends somewhere around 5000.

And this one doesn't make sense at all:

http://techreport.com/review/30281/nvid ... eviewed/12

All cards there end their lines in the middle of the graph (actually even before middle).


In the RX 480 you again have inconsistency here:

http://techreport.com/review/30328/amd- ... reviewed/7

here:

http://techreport.com/review/30328/amd- ... reviewed/9

and here:

http://techreport.com/review/30328/amd- ... eviewed/10


And this one again have almost 2 times too wide graphs for both versions:

http://techreport.com/review/30328/amd- ... eviewed/11


Now, all of that doesn't bother me that much to be honest, it's just a little error or a mistake here and there. I think in general it would be useful if TR thought a little bit about graphs presentation and perhaps moved to something more modern, a solution capable of rendering graphs from given data, so it fits to any screen and perhaps users can easily zoom in or delete/add measurements. I don't know what's out there and how much it would cost - maybe there are some freeware solutions? It's not a big deal, but I imagine it would also safe a lot of time, as the writers could just import data straight from excel or whatever and not have to deal with screenshots and pictures that you have to redo from scratch every time you decide to change something or you need to make any correction.

Mr Bill wrote:

They are trying to make all the charts comparable at a glance to have the same effective vertical scale so its easy for the reader to see when a game is really giving the video card a problem. If you expand the scales of each graph to fill the graph, you take away from the usability of the graphs for comparing across all the games.

My thought is that keeping the same y axis range would be useful if they were reviewing game engine implementations. If they're using games to review cards, I suggest that the main value is in comparisons between cards. Besides, they quite obviously have not used a common y axis range for frame number charts across the reviews I mentioned, which suggests that perhaps they aren't doing what you're thinking they are.

Ninjitsu wrote:

lolwut?

I am prepared to give the benefit of the doubt. I have been reading TR for a long while; I'd prefer to help them improve rather than move on.

Froz wrote:

Mr Bill wrote:

They are trying to make all the charts comparable at a glance to have the same effective vertical

That would be fine, but they are inconsistent in that. See here:

http://techreport.com/review/30281/nvid ... eviewed/10

Radeons chart ends at 4000, nvidia cards chart ends somewhere around 5000.

Hmmm, I see what you mean there.

Froz wrote:

And this one doesn't make sense at all:

http://techreport.com/review/30281/nvid ... eviewed/12

All cards there end their lines in the middle of the graph (actually even before middle).

Yeah, right again.

Froz wrote:

In the RX 480 you again have inconsistency here:

http://techreport.com/review/30328/amd- ... reviewed/7

hmmm yeah, the 970 reaches 4000 frames in the one but only 3500 frames in the other...
Yeah, I see what you mean and these are valid points. I think you have something here.

Its good that we get to see how many frames and how spikey they are for a run of game time. The height shows us the frame time and length shows us frames rendered during the fixed length of game time. It does look like more attention needs to happen making the graphs consistent.

Something I have wondered about is whether the game could be run for a fixed amount of game time and then plot the frame time as vertical (as they do) and game time as horizontal axis instead of number of frames rendered. It could be a horizontal axis button. So, we can see how each video card handled the same game events.

I have to say though, kudos for the graphs as they are because its pretty wonderful to be able to click a button on the graph and have it change data sets. I guess since I am using MS Office 2003, I have never gotten to fiddle with the bells and whistles of the newer Excel graphs.

Froz wrote:

Now, all of that doesn't bother me that much to be honest, it's just a little error or a mistake here and there. I think in general it would be useful if TR thought a little bit about graphs presentation and perhaps moved to something more modern, a solution capable of rendering graphs from given data, so it fits to any screen and perhaps users can easily zoom in or delete/add measurements. I don't know what's out there and how much it would cost - maybe there are some freeware solutions? It's not a big deal, but I imagine it would also safe a lot of time, as the writers could just import data straight from excel or whatever and not have to deal with screenshots and pictures that you have to redo from scratch every time you decide to change something or you need to make any correction.

My friends with Macs definitely had superior graphing applications back when I was having to makes graphs for publication. There are graphing apps that might work, but the best are very expensive and the free used to be very clumsy to alter. Maybe things are better now. My experience (being a fuddy duddy) is that all the better "free" apps have become cash cows for developers that want to be paid more than they want to be contribute to the universe of free knowledge.

Mr Bill wrote:

Something I have wondered about is whether the game could be run for a fixed amount of game time and then plot the frame time as vertical (as they do) and game time as horizontal axis instead of number of frames rendered. It could be a horizontal axis button. So, we can see how each video card handled the same game events

This might be an interesting feature to put in another chart, but it probably has limited usefulness: since few of the games offer a scripted test sequence, there's some variation from run to run. It would be almost impossible to say 'this spike on the card A plot corresponds to the spikes on the card B plot here due to specific event C'.

Fixing the existing charts would be a better idea, I think.

Mr Bill wrote:

I have to say though, kudos for the graphs as they are because its pretty wonderful to be able to click a button on the graph and have it change data sets. I guess since I am using MS Office 2003, I have never gotten to fiddle with the bells and whistles of the newer Excel graphs.

That's just simple javascript.

OK, so is it safe to say that after all these views, most people agree that there is an issue here? If so, how would one go about bringing this to the attention of the TR gang, so that they can see whether the want to fix it?

Apologies for the double post.

I took a few minutes over lunch today to have a look back through TR's previous GPU reviews - the R9 380X and R9 Fury Strix ones specifically. These reviews have Frame Number plots that appear to be a little more consistent than the three mentioned earlier in the OP, and while they sometimes have wasted vertical space and marginal horizontal waste I haven't seen evidence of truncated data sets.

This leads me to wonder whether the problem stems from something as simple as an inherited Excel spreadsheet - if the various parameters were left largely unchanged from previous reviews, we might expect to find errors of the sort that we have seen.

If that's the case, a quick wander through the spreadsheet to update the various cells should result in Frame Number plots at least on par with the historical ones - a win for everyone!

I don't know if you've seen this thread, Robert, but I'd like to thank you for tightening up the Frame Number charts in your recent RX 480 + 16.7.1 driver update - the charts look much better! The only minor gripe I have is that the the 16.7.1 CM plot appears to head off chart in the Crysis 3 results - we can't tell whether it hit max at 35ms or continued. Apart from that, everything is visible and there's basically a minimum of wasted space.

As an aside, where is your breakover point for reducing the scale increments on the y axis? The Crysis 3 plot has 5ms increments, while the GTA V chart has 10ms increments. Was there a particular reason for not using 5ms increments for both? That would probably have let you reduce y axis maximum to 25ms for the GTA V chart, if you thought that might be worthwhile [I am just curious].

Well, the GTX 1070 review is in! There have been some improvements on the Frame Number chart front; a few more tweaks could see them being pretty much spot on:

- Ensure that the Device Under Test [or another suitable reference point] is present across all tabs of each chart. Some of the results, usually where the Fury was not available, already do this; make this consistent.

- Set the Y axis range to the same value for all tabs of any given plot [see the Fallout 4 charts: no card broaches 30ms so use that for the range on both plots; only the GTX 980 exceeds 25ms so why use 35ms for the GeForce plot?] and try to keep all data inside the chart boundaries [limited by the least "performant" card].

- Set the X axis range to the same value for all tabs of any given plot [see the GTA V charts] and try to keep all data inside the chart boundaries [limited by the most "performant" card].

- Decide on whether or not to set the x axis maximum to a scale increment and consistently use that; the y axis on most [if not all] charts ends on a scale increment and labels it [both with a value and a mark on the axis]. I realise that if you end the x axis on a scale increment and label it that the label may extend to the right, but for the label lengths that we are looking at it would seem that the labels would fit within the column width [by comparing against the screen shot width]. At minimum I would suggest showing the scale mark, and preferably show the label as well.

I look forward to the next review!

Suggestion, since I've been told this is Javascript. How about the reviewed unit in the graph with whatever else you think is needed for illustration, and then having buttons for a few comparison cards that result in that cards data being plotted in the same graph. Click on click off would be even better.

That's not bad, I like that. I know it'd be a tonne more work for them, so I don't know how likely it would be, but they could keep the same sort of tab buttons they currently have which would switch between certain preset options. Then you could tailor your view.

Again, I really can't see if happening any time soon. Simple fixes first, TR, then the bigger stuff.