Why not display frame time data in a familiar format?

From the pixels, bits, and shaders to the graphic cards that power them. Discuss the latest from AMD and NVIDIA here.

Moderators: morphine, SecretSquirrel

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:44 am

You could call it "instantaneous frame rate." This would be equivalent to the frame time and will also have a name that means exactly what it is.
Carpe diem quam minimum credula postero
sid1089
Gerbil Team Leader
 
Posts: 290
Joined: Wed Jul 26, 2006 4:56 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:52 am

Or people could understand that the graphs would mean the same things if converted to FPS. When converted it is a "velocity measurement" per frame and that is different than average/max/min FPS that was shown in the old reviews.

It's the difference between in saying in baseball a pitcher throws on average 70 Mph, graphing each pitch during a game showing the balls actual speed, or showing the same graph with the time it takes each pitch to get to the plate. With the second and third graphs you are really showing the same data.

To me the real advancement was measuring down to a point you can't take any more samples, not what units the data is shown in. In the past people had graphs showing averages down to the second at best but nothing that really gets to the individual frame velocities.
tfp
Grand Gerbil Poohbah
 
Posts: 3070
Joined: Wed Sep 24, 2003 11:09 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 4:42 am

The funny thing is that frame times have been the standard performance metric for game development for as long as I can remember. Which translates into 10 years give or take one.

And now it is 2013 and the "gamer crowd" is suddenly amazed and horrified at this revelation! Kudos to TR for job evangelizing more scientific methodologies and props for sticking to your guns in the face of angry nerds.

That said, the "this is teh revolution" and "get used to it" marketing spiel *is* getting old and tired, when we have already been measuring "inside the second" for the last decade or so. Just to put things into perspective.

Over and out.
BlackStar
Gerbil
 
Posts: 92
Joined: Fri May 11, 2007 3:38 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 4:43 am

mkenyon wrote:No, I'm not wedded to it in any way. I'm talking about the clear difficulty in getting people to understand what any of it means. Yeah, it's clearly written, but there's a very visible block that people are having in understanding that this is a more accurate measure of performance rather than an alternate measure of performance. Using the different measurement I think contributes to this.

mkenyon wrote:So far, that hasn't panned out too well. I'd say an overwhelming majority of folks that I talk to about this view it as a way to capture something else entirely which isn't quite as important as FPS.

mkenyon wrote:I think this makes sense, but the Inside the Second articles are a lot to digest. There's so many points and revelations in there that some finer details might be obscured or fail to hit home.

I think what Ned meant was that you kept wanting some "new term" but with "FPS" in it. And as others point out it is somewhat backwards. The audience should still do the proper homework, read up on Damage or other people's articles on the subject to understand the new concepts anyway. Changing to some new term with the 3 letters in it does not alleviate the need to go do the proper reading. Your argument that just including the 3 letters in there will suddenly make people understand with no need to do the background reading does not seem to hold here. And this new methodology does require people to go read, can't encourage laziness that's what I think Ned was not happy about.

I am kind of a more visual person. I do look at the 99th percentile, but I usually can get what I need to know by the actual frame time plot. If I see the line wildly swinging up and down, then I know this is bad. With the buttons below the graph for different card comparisons, I can immediate draw conclusions as to which card is "smoother". Then of course I still look at the min FPS to see if the gameplay is smooth and fast enough for the eyes. Once all the frame times pass a certain point, it will be the smoothness of the curve/line that separates the good from the bad. To me at least, it is easy to understand, just need to adjust a little bit.

Granted I stay on TR during most days, but I am not really seeing this so-called "overwhelming majority of folks" that don't understand this. At least I am not seeing a whole bunch of comments from each of TR's review that claim "this sucks I don't understand all this". Perhaps it is the average level of knowledge/intelligence/comprehension/critical thinking of the TR readership is higher? :P It is not like most of the gerbils are like <15 year old "g4m3Rz!"?

I am bad with analogies, but I think of something while typing all this: if Intel can switch the entire industry from GHz obsessed to performance per watt, why can't the enthusiast community and gaming industry do a paradigm shift? Similar to the "claimed wattage" to "amps delivered on 12V rails" in the PSU debate, we have changed our mindset successfully. So why can't we do the same with this?
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24360
Joined: Mon May 24, 2004 2:19 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 8:15 am

The problem with "Frames Per Second" is that it implies an evenness that isn't necessarily there. A pitchers fastball will essentially have the same velocity from the moment it leaves his hand to the moment it ends up in the catcher's mitt, with little deceleration in the forward direction. A car moving at 50 mph will travel 50 miles in an hour, and if that is his average speed, he will travel 50 miles in an hour even with short periods of acceleration and braking. A monitor at 60 Hz will refresh every pixel 60 times in a second.

But 60 FPS on a 60 Hz monitor does not mean that the animation is smooth. Having an instance of a certain FPS is meaningless because, as PC Perspective is exploring with the idea of "runt" frames, those frames might only be displayed as a sliver on the screen, causing massive tearing in the image. Or, the screen might appear "stuck" on a certain image because the next frame hasn't been rendered. This is not represented by FPS measurements.

And as I said before, this is a discrete time system, which means we have a finite number of snapshots (frames). A fastball, a car's speed, and the air speed velocity of a swallow is a continuous time system, which means that the speeds won't change in an instant. Speeding up and slowing down is always smooth unless they run into something (and having something with high velocity suddenly stop results in massive damage). Since computers are discrete time systems, "frame rates" CAN change from frame to frame, and the metrics Tech Report and others are working on are trying to analyze THAT. Using terminology that implies evenness that is not necessarily there confuses the issue more than forcing gamers to learn about time sensitive metrics.

EDIT:
Meadows wrote:Superjawes also had an idea about a "smoothness rating" formula, that sounds like something TR might look into.

There are some people looking into it. When I first saw PC Perspective's stuff showing the results on screen, I thought it was beyond excellent. There are two problems with coming up with this rating, however.

1. Although PC Perspective is doing great work showing the problem, it's very hard to turn that into a usable metric. Sure, you can SEE the problems, but unless you want to show video for every card and every game, that doesn't quite quantify what we want to see.

2. That work doesn't necessarily identify where the problem is. Scott's work with Fraps and timestamps definitely shows some of the variance in smoothness on the back end, but as PCP pointed out, other processing might go into the image after Fraps stamps it, and that might vary by game engine. On the other foot, PCP can measure the images at the monitor, but that doesn't identify where the hiccup happened. It might have happened before Fraps would have stamped the image, but it could also have happened between stamping and displaying. You need to identify where the problem occurred to fix it.

So yeah...like I said, we've got more work to do, but we're still way ahead of where we were a couple years ago, and I am much happier moving forward with better metrics than trying to explain them in terms of old ones.
Last edited by superjawes on Thu Feb 28, 2013 8:32 am, edited 1 time in total.
Damage wrote:Don't try to game the requirements by posting everywhere, guys, or I'll nuke you from space.

-Probably the best Damage quote ever.
superjawes
Graphmaster Gerbil
Gold subscriber
 
 
Posts: 1061
Joined: Thu May 28, 2009 9:49 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 8:28 am

Well it's specious to claim that frame rate is by definitionaveraged over a time period. That's simply not true: FPS is literally frames/second, i.e. the inverse units of seconds/frame, and the values are thus the simple inverse as well. There is absolutely no reason frame rate has to be averaged over a time, so I don't think anyone is convinced by that alone as an argument. I also agree most people are used to frames/second units more than seconds/frame units, and I've argued in past articles for this as well.

However, another gerbil convinced me s/frame might a more useful metric, in the same way L/100km is more useful, in that proportional improvements are linearized. E.g. the 5MPG improvement from 15MPG to 20MPG is 14.1 to 11.3 L/100km, but the much less significant 5MPG improvement from 35 to 40MPG is more readily visible as the numerically smaller improvement from 8.1 to 7.1 L/100km. The same is mostly true with frame times.

So I gave up arguing that we need to use frame rates instead of frame times. We're smart people, we can switch to a new metric. I was then irritated that frame time wasn't used throughout the article; frame rate was used to represent the average, then everything else on the page in frame times. So I said if you're going to switch, then switch! But as Damage responded, he only has one frame rate chart per page. (It's the average one, which alas I think simply perpetuates the illusion that frame rate has to be over a time period.) So he has one chart to give people an approximate reference in the units they're used to, then everything else is actually frame times.

Between those two things, I felt good enough about it that I stopped arguing that we should use frame rates.
cobalt
Gerbil
Gold subscriber
 
 
Posts: 36
Joined: Mon Oct 30, 2006 11:28 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 10:08 am

Why do you want to show "frames per second the last 17 milliseconds"?

The "time spent past" graphs give a good corellation to FPS already, with the usual values being the equivalent of 60 and 30 FPS(time spent past 17,2ms means the time spend at LOWER than 60fps. Time spent past higher values means time spent below 30fps, etc)

It's clear as day, and if you can't tell someone "this is the amount of time you spend below x fps" then converting frametime to FPS will just make it even more confusing.

There's a reason we don't show cars' speeds in metres per second or kilometres per second, because it's supposed to show the average distance you'll travel. It doesn't matter if you have 300fps in a game for an hour, as higher FPS doesn't make you travel faster in the game or give you the ability to do more. What matters in games is how well it's going RIGHT NOW, which the frame latency method shows perfectly.

Cars don't go 170kph and then 20 and then 130 either(if a car does that, it's in serious trouble)
RandomGamer342
Gerbil In Training
 
Posts: 2
Joined: Thu Oct 18, 2012 4:07 pm

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 10:10 am

Why do you want to show "frames per second the last 17 milliseconds"?

The "time spent past" graphs give a good corellation to FPS already, with the usual values being the equivalent of 60 and 30 FPS(time spent past 17,2ms means the time spend at LOWER than 60fps. Time spent past higher values means time spent below 30fps, etc)

It's clear as day, and if you can't tell someone "this is the amount of time you spend below x fps" then converting frametime to FPS will just make it even more confusing.

There's a reason we don't show cars' speeds in metres per second or kilometres per second, because it's supposed to show the average distance you'll travel. It doesn't matter if you have 300fps in a game for an hour, as higher FPS doesn't make you travel faster in the game or give you the ability to do more. What matters in games is how well it's going RIGHT NOW, which the frame latency method shows perfectly.

Cars don't go 170kph and then 20 and then 130 either(if a car does that, it's in serious trouble)
RandomGamer342
Gerbil In Training
 
Posts: 2
Joined: Thu Oct 18, 2012 4:07 pm

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 10:17 am

Regardless of whether "framerate" is the proper term to use for the new method, consumers playing games at home with FRAPS running in the corner of the screen are going to see FPS (which is of course a running average). Is the "99th percentile frame time" not an average in it's own regard anyway? Frame-time and frame-rate are mathematically interchangeable as given in the conversion table at the beginning of each GPU review. Just pick one or the other to present: average framerate and 99th percentile framerate OR average frame time and 99th percentile frame time.
i5-3570K, ASRock Z77 Pro4-m, Asus GTX660 TOP, 120 GB Vertex 3 Max IOPS, 2 TB Samsung EcoGreen F4, 8GB G-Skill @1.25V, Silverstone PS07B
DPete27
Gerbil Jedi
Silver subscriber
 
 
Posts: 1632
Joined: Wed Jan 26, 2011 12:50 pm
Location: Madison, Wisconsin

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 12:33 pm

tfp wrote:To me the real advancement was measuring down to a point you can't take any more samples, not what units the data is shown in. In the past people had graphs showing averages down to the second at best but nothing that really gets to the individual frame velocities.

This. So much this. Agreed 100%
Flying Fox wrote:Granted I stay on TR during most days, but I am not really seeing this so-called "overwhelming majority of folks" that don't understand this. At least I am not seeing a whole bunch of comments from each of TR's review that claim "this sucks I don't understand all this". Perhaps it is the average level of knowledge/intelligence/comprehension/critical thinking of the TR readership is higher? :P It is not like most of the gerbils are like <15 year old "g4m3Rz!"?

I am bad with analogies, but I think of something while typing all this: if Intel can switch the entire industry from GHz obsessed to performance per watt, why can't the enthusiast community and gaming industry do a paradigm shift? Similar to the "claimed wattage" to "amps delivered on 12V rails" in the PSU debate, we have changed our mindset successfully. So why can't we do the same with this?

I do think the TR community is a bit isolated in terms of understanding performance. I spend most my internet forum time on NeoGAF, OCN, and then various esports centered places. When people THINK they understand frame time, they think it's an alternate measurement to Min/Max/Avg FPS that needs to be used in conjunction with the second based polling to get a good idea of performance. They don't understand it's a replacement. There are people who do get it, but it's a very small portion.

I'd really like to think that we can switch it, but...

DPete27 wrote:Regardless of whether "framerate" is the proper term to use for the new method, consumers playing games at home with FRAPS running in the corner of the screen are going to see FPS

This right here is the major roadblock. People who care about performance do this through fraps/afterburner/precision and this is how they judge their performance. They don't look at frame time, even if they understand that frame time testing is superior.

There will *never* be a switch where people say "I want to buy a computer and am looking to have consistent 8.3ms frame times for my 120Hz monitor, what parts should I buy?" That person will be saying, "I want 120Hz/FPS."

I think the attitude of "well they just need to stop being idiots, read closer, and figure out what all this means" doesn't really get anyone anywhere. This is ultimately about providing tools to allow end users and reviewers to get a better idea of how our consumer products perform. I think looking at where the roadblocks are in getting these tools and information out there is just as important as the tools themselves. The cryptic (to others) nature of the millisecond data points is a huge road block, IMO.
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 1:05 pm

Frame time is basically the minimum frame rate.
What matters is the amount of time spent at that minimum, which is harder to quantify in terms of FPS.
If we go by averages then the good frames tend to smother out the bad frames, making it a bit hard to represent as a simple graph.

HardOCP's style of plotting the frame rate over time is a pretty good method, IMO. You can clearly see where the FPS line graph go into those single digit dips, and how often and how long those dips last.
jihadjoe
Gerbil Team Leader
 
Posts: 251
Joined: Mon Dec 06, 2010 11:34 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 1:25 pm

jihadjoe wrote:Frame time is basically the minimum frame rate.
What matters is the amount of time spent at that minimum, which is harder to quantify in terms of FPS.
If we go by averages then the good frames tend to smother out the bad frames, making it a bit hard to represent as a simple graph.

HardOCP's style of plotting the frame rate over time is a pretty good method, IMO. You can clearly see where the FPS line graph go into those single digit dips, and how often and how long those dips last.

Frame time is not the minimum frame rate. Frame time displays all of the detail.

HardOCP's frame rate over time is lacking because it's still second-based polling.
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 1:43 pm

mkenyon wrote:So many people I talk to on other forums and even with my friends just don't understand that these numbers are essentially measuring the same exact thing ...

Nope. The two numbers are not measuring the exact same thing, and until both the speaker and listener understand that, no actual information is communicated.

There are many statistical measures that are used to convey numerical data for purposes of comparison. The "average" is a common measure, and FPS is an average. It tells you how many frames you can expect to see over a period of time. "Apples per tree" might be one that an apple grower might use to compare yields. It's important to note, though, that "apples per tree" says nothing about the SIZE of those individual apples. So two trees with identical apples-per-tree numbers might have very different measures of "goodness".

A measure that closely aligns with TR's frame latency testing is "standard deviation". Standard deviation is a measure of clustering. If you have a large number of frames with very similar rendering times and a very low number of frames with longer rendering times, then you will have a low standard deviation and a good gaming experience. The more "anomalous" data points you have (in terms of frame times), and the farther these data points are from the overall average, the larger your standard deviation and the worse your gaming experience gets. Standard deviation, then, matches up pretty well with what the latency charts try to communicate.

A person going up a flight of stairs and a person going up an escalator may arrive at the upper floor at the same time, but they've had entirely different experiences. The escalator offers steady, smooth motion, while the stairs yield wide swings in velocity in a given time slice. The latency charts try to show this difference.
This problem was caused by Windows, which was created by Microsoft Corporation.
sluggo
Gerbil Jedi
Gold subscriber
 
 
Posts: 1541
Joined: Wed Feb 16, 2005 8:44 pm
Location: under the table and dreaming

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 1:49 pm

You're only referring to the line chart.

Frame time testing can be used to measure a huge array of things from percentile latency, to smoothness, to focusing on outliers and where things can go wrong.

This is exactly what min/max/avg is trying to capture, but fails to do so accurately.

*edit*

To use an analogy, they're both measuring the same thing. It just so happens that frame time testing measures mm with a mm ruler. FPS testing measures mm with a cm based ruler. Though one is clearly better, they are in fact attempting to measure the same thing.
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 1:56 pm

So as a graphics engineer in AAA game development, I don't really care how the frame time data gets displayed. Internally, we use both frame latencies and instantaneous frame rates to measure performance and aggregate performance trends as changes are being made, because, as others have noted, the two values are just inverses of each other and some people find it more intuitive to work with the "bigger is better" paradigm that FPS numbers offer. What really needs to die is the concept that min/avg/max FPS is a useful measure of performance and gameplay experience. We care about average FPS insofar as a rough measure of whether a user will have an unplayable experience so we can gauge how much optimization work still needs to be done for a given tier of hardware. Min/max FPS is literally meaningless.

@OP: Frankly, if you have the time to spend trying to convince people to still show FPS values in performance measurements that are latency based, I would rather you spend that time educating the apparently "ignorant" populace in the communities you frequent about why still trying to use min/avg/max FPS measurements is pointless.
Zoomastigophora
Gerbil Elite
 
Posts: 611
Joined: Tue Nov 11, 2008 7:10 pm

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 1:58 pm

mkenyon wrote:Frame time is not the minimum frame rate. Frame time displays all of the detail.
HardOCP's frame rate over time is lacking because it's still second-based polling.


Sorry I mean the "99th percentile frame time" basically refers to the minimum frame rate. The frame rate over time thing can easily be fixed, just expand the X-axis, or emphasize the spikes/dips when compressing it along the X-axis.
jihadjoe
Gerbil Team Leader
 
Posts: 251
Joined: Mon Dec 06, 2010 11:34 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:02 pm

I'm coming in a little late to this but my opinion is that a conversion to FPS is a good idea.

Call it "instantaneous FPS" (iFPS if you need a shorthand) or "equivalent FPS" (eFPS).

This is really helpful, IMO, for three reasons:

1)It's the common language used all over the place: on other sites and with various measuring tools
2)It's what people are familiar with and it is very easy for people to think about it in those terms
3)It speaks to the REAL PROBLEM which is achieving CONSISTENT FPS.

And there's one argument against it that I think is bogus: that it's innaccurate. This isn't true if you just call it iFPS or eFPS as mentioned above. If you explain what the conversion means, then there's no inaccuracy.

For people to get married to "frame latency" is no less silly than getting married to "FPS" in my opinion. They are easily converted from one to the other. The difference that needs to be made clear is whether you're talking about the average or the instantaneous.

Everything ties back to getting a consistent FPS, so I think it's silly not to keep it in those terms.

But I don't care, really. Jennifer Lawrence or Emma Watson?
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:05 pm

Zoomastigophora wrote:@OP: Frankly, if you have the time to spend trying to convince people to still show FPS values in performance measurements that are latency based, I would rather you spend that time educating the apparently "ignorant" populace in the communities you frequent about why still trying to use min/avg/max FPS measurements is pointless.

I do that very thing, actually :P

http://www.neogaf.com/forum/showthread.php?t=512976
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:22 pm

Some of this reminds me of people giving suggestions to any opensource project. Often the responses from surly community members are, "get the code and implement it yourself. Good luck!"

i.e. the onus is on everyone else in the world to fix whatever is wrong with the item.

I get the same feeling from the "everyone else should be smarter" vibe getting flung at the OP. As educators, you have a responsibility to convey the information in the way it'll do the most good. Getting pedantic about it and missing the opportunity to expand your audience is just sad.

OP has been trying to evangelize this to the people who it would most serve, the esports players, and has documented the issues that he's run into. Instead of "too bad for them" the response really should be about finding a way to constructively deal with the problem. I also think Scott should jump in on this, too, as while it's all well and good to work with other tech editors about the testing methodology, this is an example of how the current results are being interpreted and misintrepreted in the real world. I'd think he'd welcome this opportunity with open arms to finetune how his results are displayed in a way that makes more sense to a layman.

Because I don't want to be attacked about my opinion here, just let me say that I practice what I preach and try to give back what I learn, be it with random posts in TR's forums or doing things like writing up "how to compile" how-to's for opensource projects. I hope something constructive comes of this thread and discussion! :)
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:30 pm

mkenyon wrote:You're only referring to the line chart.

Because that's what I was talking about.

mkenyon wrote:To use an analogy, they're both measuring the same thing. It just so happens that frame time testing measures mm with a mm ruler. FPS testing measures mm with a cm based ruler. Though one is clearly better, they are in fact attempting to measure the same thing.

This is a horrible analogy, and it's wrong on both points.

If, as you suggest, TR is "measuring the same thing", just calling it something else, then you must be suggesting that all TR is doing is taking time/frame, because this is indeed the reciprocal of frame/time. But that's not what they're doing, because they're not interested in taking the reciprocal of data that's already proven to be not all that informative. The percentile figure does NOT represent just a way of stating frame/time as time/frame. The percentile figure is a bounds measure (single-ended, in this case), NOT a measure of mean value over time. So no, they are NOT measuring the same thing.

Second, it's not a question of higher precision (as your mm/cm example claims). You can have very precise and accurate FPS measurement and very precise and accurate percentile measurements and they will tell you very different things. The percentile measurements will NOT tell you FPS, and the FPS measurements will NOT give you frame time percentiles.

Lastly, and think about this: why would TR spend all this time generating this data if it were simply a way of restating what they could easily get from FRAPS? They know what they're doing, and this data is informative in ways that FPS data is not. If your friends on other forums have trouble grasping that, giving them bad analogies like mm/cm is doing them no favors.
This problem was caused by Windows, which was created by Microsoft Corporation.
sluggo
Gerbil Jedi
Gold subscriber
 
 
Posts: 1541
Joined: Wed Feb 16, 2005 8:44 pm
Location: under the table and dreaming

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:45 pm

Scrotos wrote:Some of this reminds me of people giving suggestions to any opensource project. Often the responses from surly community members are, "get the code and implement it yourself. Good luck!"

i.e. the onus is on everyone else in the world to fix whatever is wrong with the item.

I get the same feeling from the "everyone else should be smarter" vibe getting flung at the OP. As educators, you have a responsibility to convey the information in the way it'll do the most good. Getting pedantic about it and missing the opportunity to expand your audience is just sad.

OP has been trying to evangelize this to the people who it would most serve, the esports players, and has documented the issues that he's run into. Instead of "too bad for them" the response really should be about finding a way to constructively deal with the problem. I also think Scott should jump in on this, too, as while it's all well and good to work with other tech editors about the testing methodology, this is an example of how the current results are being interpreted and misintrepreted in the real world. I'd think he'd welcome this opportunity with open arms to finetune how his results are displayed in a way that makes more sense to a layman.

Because I don't want to be attacked about my opinion here, just let me say that I practice what I preach and try to give back what I learn, be it with random posts in TR's forums or doing things like writing up "how to compile" how-to's for opensource projects. I hope something constructive comes of this thread and discussion! :)

Thank you so much for getting it.
sluggo wrote:This is a horrible analogy, and it's wrong on both points.

I mean that they are both trying to measure performance and smoothness.

How something is measured and what is being measured are two things, and I think you're getting hung up on the former, which we both agree on. I totally agree with what you are saying, you are absolutely right. I think I just didn't convey my above statement accurately.
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 2:48 pm


Uh... Well... Maybe... Or not... I feel like Buridan's ass right now :-? Can I choose both? :(
My subscription allows you people to exist on this site and makes me a better human being than you'll ever be
JohnC
Gerbil Jedi
Gold subscriber
 
 
Posts: 1881
Joined: Fri Jan 28, 2011 2:08 pm
Location: NY/NJ/FL

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 3:10 pm

I did not read the entirety of this thread, but I would prefer that we have less stupid/uninformed people that can't understand a two-metric system (some form of framtime + fps), rather than try to dumb things down for them.

Educate people about frame timing until it is the norm and common knowledge.
I am jaded. I am cynical.
lilbuddhaman
Gerbil First Class
Silver subscriber
 
 
Posts: 131
Joined: Sat May 10, 2008 8:23 pm

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 3:46 pm

lilbuddhaman wrote:I did not read the entirety of this thread, but I would prefer that we have less stupid/uninformed people that can't understand a two-metric system (some form of framtime + fps), rather than try to dumb things down for them.

Educate people about frame timing until it is the norm and common knowledge.


What bothers me more than settling on either metric is the insinuation that people who prefer the FPS metric are stupid. Please don't go there. It's bad form in every way. That's the absolute worst turn this discussion can take, so I ask you: don't take that turn.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 3:50 pm

lilbuddhaman wrote:I did not read the entirety of this thread, but I would prefer that we have less stupid/uninformed people that can't understand a two-metric system (some form of framtime + fps), rather than try to dumb things down for them.

Educate people about frame timing until it is the norm and common knowledge.


Sure, like we can use wavelength and frequency somewhat interchangeably. But the question raised here is as if every article ever written about the visible spectrum of light used wavelength as its metric, and so people generally knew how many nm red is and how many nm blue is, why should we switch to frequency? There have to be some good arguments that wavelength is a clearly better metric, or at least that is has value which is clearly not captured in the original metric.
cobalt
Gerbil
Gold subscriber
 
 
Posts: 36
Joined: Mon Oct 30, 2006 11:28 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 4:51 pm

Looking back over my posts here I can see where I might have come off as a bit pedantic. I could try to pin on the years I spent teaching math, but in truth it's just my nature. So my apologies to mkenyon and may I say welcome to the board and we're not all pricks here :)

If I was quick with my keys, it's because this "inside the second" work by the TR staff has been very useful and enlightening. If it is to become one of the standard metrics (and I hope it does) it will be because it's been rigorously and clearly defined, not because it's easily quotable by the larger base of users. The fact that GPU manufacturers are taking it seriously is a testament to its value. If a clever and accurate three-letter-acronym comes along that catches on, so much the better, but let's not force anything just yet.
This problem was caused by Windows, which was created by Microsoft Corporation.
sluggo
Gerbil Jedi
Gold subscriber
 
 
Posts: 1541
Joined: Wed Feb 16, 2005 8:44 pm
Location: under the table and dreaming

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 5:23 pm

No worries. This stuff has been under attack from fanboys and closed minded enthusiasts alike since the day it was first talked about. That 7950 vs 660Ti review definitely made things a bit more heated too. It's natural to go into defensive mode about it, as that's exactly what I've been doing for months now. I get it.

I'm just finding that this data is really really hard for a lot of people to grasp, even though it is really simple. It's a total paradigm shift, and when you're dealing with modern internet culture (which everyone who is passionate about tech is a part of), it can be extremely hard to get people to change gears or even care about the data.

This is pretty much what I'm talking about though:
sluggo wrote:If a clever and accurate three-letter-acronym comes along that catches on, so much the better, but let's not force anything just yet.


I'm not looking to force anything, but I just wanted to open up discussion on it. From someone who doesn't frequent the TR forums and spends most of his forum time elsewhere, I thought a discussion on what I've found might be helpful to further refine this nascent methodology.

Disclaimer: I get that graphic engineers have been doing it for ages, but it's new for the rest of us.
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 6:03 pm

While someone might say that there is a distinction between instantaneous rates and average rates, in reality all rates are averages - it's just that when the time interval is small enough, depending on the situation, whether are not there is variance within the interval ceases to be important. The velocity of a car moving at an average of 50 mph for a short period - an instantaneous, erm, instant - is completely described by saying it is travelling at 50 mph (for most intents and purposes). A car that travels an average of 50 mph for, say, an hour, could be travelling 50 mph for most of it, or it could be travelling 30 mph for 15 min due to construction, 20 mph for another 15 min and 75 mph for the remainder, is not always so. And sometimes this variation is important.

There is nothing to indicate that a card cranking out 60 FPS is delivering each frame every 1/60th of a second. Because the degree to which this fluctuates can impact user experience, it is of utility to describe this variance in a GPU review. Hence this whole business. And now we are lead even further down the rabbit whole with the discovery that all delivered frames are not necessarily of equal merit...

Whether or not one describes this variance using FPS or "frame time" is irrelevant. As I said on the previous page, the major drawback to using 99th percentile FPS to describe latency is to have it confused with the more traditional averaged FPS. To me, this just risks the perpetuation of further ignorance on the subject. So why change to it now?

The argument that it may make some people feel more comfortable is a hollow one in my opinion. The lack of discomfort comes from a lack of understanding of what is being measured. Changing the name of the metric won't help that one bit. Or are you advocating the substitution of discomfort with ignorance? If ignorance really is bliss, why bother reading these reviews in the first place?
cynan
Gerbil Elite
Gold subscriber
 
 
Posts: 812
Joined: Thu Feb 05, 2004 2:30 pm

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 7:16 pm

I'm stepping into serious speculative land here, but the way that I've seen it works something like this:

FPS = Performance
Frame time testing in ms = level of stutter
level of stutter =/= absolute performance
Therefore, frame time is a supplemental testing to FPS which is still king for measuring performance.

I don't think it's discomfort, but I think it's people not realizing that Min/Max/Avg FPS has always existed to try and show how smooth gameplay is, albeit very inadequately.

This is even more true for people who are interested in esports. For these folks, (myself included) latency is EVERYTHING. The frame time data only matters insofar as to point out areas where that latency spikes. We don't really care if it varies as long as it's staying around 8.3ms or lower. So since they kind of fall into that category I outlined above, they write this off as unimportant.
mkenyon
Gerbil
 
Posts: 25
Joined: Tue Oct 02, 2012 11:50 am

Re: Why not display frame time data in a familiar format?

Postposted on Thu Feb 28, 2013 7:25 pm

cynan wrote:While someone might say that there is a distinction between instantaneous rates and average rates, in reality all rates are averages

No, no, no, you're not getting what I'm saying. An "instantaneous frame rate" is just another name for "frame latency converted to frame rate". If you don't like the term "instantaneous frame rate" then use the term "converted frame rate" or cFR or cFPS or whatever term you pick. But it's a mathematical fact that the delivery time of a single frame can be extrapolated and expressed as a theoretical frame rate.

However, our wonderful debate here probably matters not one bit. Scott doesn't like using the term "frame rate" at all because, as I read it, he feels the term is completely tainted and he needs to distance his work from that taint. Maybe he's right, but I just don't think it's necessary and I think that expressing frame latency as instantaneous frame rate would be much more helpful. Again, the whole point, the whole entire point, is consistent frame rate. Why bother leaving the metric that really matters and that he always has to come back to anyway?
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

PreviousNext

Return to Graphics

Who is online

Users browsing this forum: No registered users and 3 guests