Where minimum-FPS figures mislead, frame-time analysis shines

In the wake of AMD’s Ryzen launch, I’ve been reading a lot of comments around the web about the inclusion or usefulness of the “minimum frame rate” in many reviewers’ results. Readers seem to think folks who report a minimum frame rate alongside an average are providing a more complete picture of gaming performance from these new CPUs. That may be true in a way, but the picture is still about as shallow as reporting a frames-per-second average on its own.

The problem is that a minimum frame rate—or any instantaneous frame-rate value—doesn’t tell us much at all about the overall gaming experience. A minimum frame rate indicates that somewhere in a test run, the graphics card rendered a frame in a certain number of milliseconds. It doesn’t tell us how many frames were produced in a similar amount of time, or how closely together they occurred. The way I see it, focusing on what might be one frame in a set of thousands places undue value in an outlier.

That said, a minimum frame rate value (or maximum frame time) and its friends might occur more than once in a test run. Anybody familiar with how The Tech Report presents frame-time data knows that we already have better ways of dealing with this problem. We set a number of crucial thresholds in our data-processing tools—50 ms, 33.3 ms, 16.7 ms, and 8.3 ms—and determine how long the graphics card spent on frames that took longer than those values to render. Those values correspond to instantaneous frame rates of 20 FPS, 30 FPS, 60 FPS, and 120 FPS.

If even a handful of milliseconds start pouring into our 50-ms bucket, we know that the system is struggling to run a game at times, and it’s likely that the end user will notice severe roughness in their gameplay experience if lots of frames take longer than that to render. Too much time spent on frames that take more than 33.3 ms to render means that a system running with traditional vsync on will start running into equally ugly hitches and stutters. Ideally, we want to see a system spend as little time as possible past 16.7 ms rendering frames, and too much time spent past 8.3 ms is starting to become an important consideration for gamers with high-refresh-rate monitors and powerful graphics cards.


In tandem with these buckets, our frame-time plots tell us where and how often major slowdowns occurred, and whether they occurred in close successsion. On these charts, low and flat-looking lines are best. Lots of humps, spikes, or too much “furriness” is an indicator of a sub-optimal experience. If we see just one or two spikes over an entire test run, it’s likely that the experience was still enjoyable, but flat lines are still best. In our recent Ryzen review, all of the CPUs delivered a solid gaming experience for the most part—some were just faster while doing it.

Enter the histogram with Crysis 3

Even with all of those data-visualization methods at our disposal, there may still be useful ways to plot frame times that we haven’t yet considered when thinking about minimum frame rates. While I was musing on the characteristics of these rates on Twitter, a TR reader asked me to consider whether a histogram would be a good visualization of the minimum-frame-rate (or maximum-frame-time, if you will) problem. Histograms are a natural fit for this type of analysis, since my primary complaint about a minimum frame rate value alone is that it doesn’t tell us how many times such an event occurred during a test run. I figured it was worth a shot, so I started crunching numbers and inserting graphs in Excel.

Here’s a couple histograms of test runs from Crysis 3 in our Ryzen review, from the Ryzen 7 1800X and the Core i7-7700K. These chips were closely matched in Crysis 3. I’ve converted raw frame times into instantaneous frame rate values (not averages over a second) so the left-to-right ordering makes sense.


In both cases, we can see that only a couple frames out of thousands would qualify as “minimum frame rate” frames, and it seems highly unlikely that a gamer would care about their contribution to the overall picture. Our “time-spent-beyond-X” graphs for Crysis 3 back up that analysis.

We can pick over these histograms quite a bit, but they tell the same basic story as our 99th-percentile and average-FPS charts from Crysis 3. I’m not actually sure how valuable it is to present this histogram information, because it seems easy to draw the wrong idea from these charts without familiarity with frame-time testing.

Still, it’s clear that we want more and taller bars toward the middle or right of the chart, as demonstrated by the FX-8370’s Crysis 3 performance.

 

Another look at Grand Theft Auto V
Crysis 3 proved good for both the Core i7-7700K and the Ryzen 7 1800X, but Grand Theft Auto V wasn’t so kind. This less-threaded game opened up a considerable performance gap between the two chips, so we wanted to re-examine its performance with histograms to see what that looked like.

That’s interesting, we think. The Core i7-7700K produces a nice, fat dromedarian hump with most of its frames clustered to the right of the chart, while the bactrian Ryzen 7 1800X exhibits a seemingly-more-bimodal distribution. Not only is the Core i7-7700K faster, but its frame delivery is more consistent—and our frame-time data bears that out.

At the end of the day, I don’t think it’s worth putting too much stock in minimum frame rates. Our histogram analysis lets us see that they’re extreme outliers that might not contribute more than a couple frames (if that) to the overall picture. We already have much better tools to make conclusions about component performance in the 99th-percentile frame time, frame-times-by-percentile, and plots of frame-time data that we present. We might start including frame-rate or frame-time histograms in our future reviews, however, because hey, they’re interesting. Let me know what you think.

Oh, and yeah. Average FPS continues to be terrible. You can take that to the bank.

Feature image originally by Jjron on WikipediaCC BY-SA 3.0

Comments closed
    • saiAOLH
    • 3 years ago
    • mcarson09
    • 3 years ago

    Min, avg, and max FPS are still good to have because their numbers only apply to each tester and test setup and they have been the reviewing standard for years. By leaving out min and max fps it really makes reading your review of the 1080 ti pointless to me without a full picture of the kind of benchmarks and the data you were receiving. Another thing that has become toxic is the fanatical AMD can do no wrong here after hiring Scott. Ryzen has issues and has yet to break into the workstation/server market and I would like those issues to be objectively explored.

    • guruMarkB
    • 3 years ago

    I left 2 emails to Jeff last week that have links to a report that reveals that Windows 10 doesn’t identify that the Ryzen is hyperthreaded and thus isn’t scheduling processes properly and thus by treating the new Ryzen as a 16 core cpu without HT is incorrectly assigning a 2nd task to a cpu core instead of putting it on another of the 8 cores. This in turn means that any game performance article that is posted here such as this one does not have valid AMD Ryzen graphs and we can’t compare Ryzen head to head against Intel in Crysis 3 (or any game). If Jeff would be upfront about this and let us know that until Windows 10 ‘s scheduler is properly patched that he will conduct all tests using Windows 7 Pro.

    In another article it was stated by AMD’s CEO that no Windows 7 optimized drivers will be coming from AMD and only Windows 10 will be officially supported. I sure hope AMD and Microsoft can get Win 10 patched asap.

      • Cannonaire
      • 3 years ago

      AMD has made official statements debunking that rumor. PCPerspective has also done tests which corroborate as much.
      [url<]https://www.pcper.com/reviews/Processors/AMD-Ryzen-and-Windows-10-Scheduler-No-Silver-Bullet[/url<] *Edit* grammar

      • Jeff Kampman
      • 3 years ago

      [url<]https://techreport.com/news/31579/amd-says-ryzen-cpus-are-performing-as-expected-after-launch[/url<] Quit spamming my inbox and articles with this dreck.

    • JoeKiller
    • 3 years ago

    +1 to histograms

    • calken
    • 3 years ago

    Hello everyone (new to techreport).

    Totally agree with the histograms but believe they should go a step further. I would have a normal distribution curve applied and the charts overlaid.

    Ideally the benchmark tested will be known to create a normal distribution under controlled circumstances e.g. no looking round at the sky for a few minutes. I would then plot r*r (goodness of fit), mean and standard deviation as I think it would provide a better description of the playability of the game.

    – Poor r*r shows odd behaviour in either the games/CPU/GPU which could be critically analysed (memory bandwidth, scheduler issues, game design, vram etc)
    – The mean of the fitted curve is a better indicator of gaming experience (see GTA example)
    – The SD shows the overall spread, the tighter the better. Do gamers locking the FPS wants a lower SD for their experience?

    Badly designed games might have right skewed non normal distributions which return a high average fps with which medians should be the plotted value and they should be called out on it.

    I think the tech press is reaching a point of paradigm shift in their consumers/readers where the data being presented isn’t a reasonable representation of events and the audience has matured to a level where the data needs evolving to meet expectations.

    It’s so refreshing to see results presented in such a way, accompanied with some fantastically thought provoking commentary. Keep up the good work.

    Chris.

      • dikowexeyu
      • 3 years ago

      The distribution clearly are not a normal distribution, so, it would be misleading.

    • rex
    • 3 years ago

    Here is a simpler/similar test.

    1. Define the lowest number that is acceptable for gaming at a given value of fps. So for gaming on a 60Hz monitor, ideally, you’d want each frame to be rendered at around 16ms time intervals. Shorter or longer than that leads to nonoptimal performance.

    2. Find the number of occurrences/frames that have a higher number than that. The device that has the lowest value “wins.” This quantity measures the number of hiccups that the GPU/CPU/etc. has had during the testing procedure.

    3. You should also define a quantity that describes how smooth, in general, the gameplay is. In other words, every frame should be rendered at 16ms intervals (60Hz refresh rate), everything that takes longer or shorter time than that is again nonoptimal. For that measure the FWHM of the Gaussian or the percentiles of the histogram for the centered values. The smaller the FWHM the smoother the gameplay.

    4. Repeat the tests N number of times to get an average value of the two quantities. Done.

      • cynan
      • 3 years ago

      While great in theory, I don’t know how feasible it would be to define a definitive frame time threshold. And I’m not sure why shorter frame times are an issue. Only longer ones cause lags that become detectable as stutters in smooth play.

      I think when it comes to something like frame time, relative performance (vs other systems) vs absolute (a definitive frame time to beat) is most useful.

    • Kretschmer
    • 3 years ago

    Suggestions:

    1) Instead of presenting the absolute number of frames that fall below a certain frametime value, display a graph of the percentage of frames that meet or improve that frametime goal.

    Product X – 99.8%
    Product Y – 98.1%
    Product Z – 83.3%

    This aligns with the usual “bigger = better” tech review bar chart convention and empowers the reader with the relative magnitude of smoothness. Knowing that 1,234 frames from your benchmark fell below the 16.7ms threshold isn’t as actionable as knowing that 2% of frames delivered do not meet the 16.7ms target.

    2) Instead of referring to the frame time values, translate this into “smooth FPS” values for the reader. So “97% of frames meet the 8.3ms target” becomes “97% of the frames are consistent with smooth 120FPS gameplay.” This is much more intuitive and may entice more readers who aren’t familiar with your methodology.

      • cygnus1
      • 3 years ago

      I like both of these suggestions a lot.

    • psuedonymous
    • 3 years ago

    I really like the histograms, though they could do with vertical dividers at common rates – e.g. dashed at 30, solid at 60, (90 for VR), 120, and 144 – to make it more clear when frame delivery time is actually going to impact visual output (rather than merely an I-discard-more-frames-than-you pissing match).

    Is there any chance we could get a ‘sample dataset’ to play with, see if we can come up with some interesting visualisations?

      • MrJP
      • 3 years ago

      It’s very easy to generate your own dataset by logging frame times using FRAPS while playing a game, just a bit harder to do find something to do on a single system that will generate interestingly-different profiles to compare.

        • Cannonaire
        • 3 years ago

        You can set Affinity for a game to generate different and comparable datasets.

        As a real example I did the other night, Overwatch on all 4 cores was running at about 137fps according to the in-game counter, standing still in the training area. I changed the affinity so that it could only use 3 cores, and the framerate dropped to ~129. Giving it only 2 cores to work with moved the consistent fps all the way down to about 70-90, and when I finally started moving it felt very stuttery as compared to the near-perfect, consisted smoothness I get with 4 cores and framerate capped at 115. (144hz monitor, vsync off)

      • Jeff Kampman
      • 3 years ago

      We don’t share sample data but you can generate your own pretty easily with PresentMon: [url<]https://github.com/GameTechDev/PresentMon[/url<] The column that you'll want from the output is the MsBetweenPresents data. Alternately, AMD has been working to produce a capture tool of its own called OCAT that's a bit more user-friendly: [url<]https://github.com/GPUOpen-Tools/OCAT[/url<] However, some games don't like the way it hooks into .exes. The data you'll want from the output file is the same as that from PresentMon.

        • psuedonymous
        • 3 years ago

        Capturing our own data is fine, I was more thinking of having one ‘sample’ dataset for everyone to work from, from a ‘known good’ capture (to rule out “I did something dumb” or “my system is really weird” issues). A bonus if it’s a dataset you’ve worked on before so results can be compared directly.

    • Cannonaire
    • 3 years ago

    After reading most or all of the comments here, I have formulated my thoughts on what I personally think would be the clearest way to present the recorded data.

    Converting the frame counts back into time would put a disproportionate emphasis on the lower framerates/higher frametimes. Converting it back into a time metric defeats the purpose of using frametimes over average framerate.

    What matters most to animation smoothness and game feel is [i<]consistency[/i<]. If you're getting discrete frames which take longer to render than your desired frametime, those are the problem - not total time spent beyond that target threshhold. I think the perfect solution would be presenting three graphs: 1. 99th Percentile Frame Times chart as currently used. It's a simple number indicating relative performance, but it's superior to FPS. 2. The "Frame Number" graph as currently used. It shows [i<]where[/i<] you have problems in a particular benchmark run. (1 and 2 both account for and convey time) 3. Instead of "Time Spent Beyond X", use the histograms as shown in this article, with buttons to switch between competing hardware. There can be a note on the X axis with "SLOWER FRAMES" or just "Worse" on the left and "FASTER FRAMES" or just "Better" on the right, which can be understood intuitively. The way it is now, readers get an incomplete picture of how steadily a product is maintaining a particular frametime, and the order of the products sometimes even [i<]changes position[/i<] on the chart at different steps, which adds to the confusion - particularly when many of them are the same color. If you use the histograms, you can directly compare the data between products on the same normalized scale, with the full context, and with all of the steps that are currently shown by "Time Spent Beyond X". It will give you a better picture of [i<]how steadily a given product will sustain[/i<] those frametimes rather than [i<]how long it will not.[/i<] A product with a 'hump' further right will be objectively better than one with a 'hump' further left. I don't know what's simpler than that. Thank you for exploring new methods and writing this excellent article, Jeff! You've given your readers something new to think about and discuss. This is why TR is the best PC Hardware site there is.

      • TheMonkeyKing
      • 3 years ago

      While this is fantastic for GPU references, I am sorta hesitant to apply it to CPUs.

      I understand the need to compare the new with current offerings and older but unless it is a complete SOC, trying to find and fit a bottleneck is just a corner case and not a real life situation. Oh i know everyone uses a reference GPU and the ability to push the workload back, as much as possible, to the CPU but I see it failing as a future standard. AMD is a different beast than nVidia and Intel in that both CPU and GPU demand full attention; the others use the CPU/GPU pretty much as an afterthought or better yet, a market need.

      I like that you are delving into the analysis of what a person might see of experience and try to find similarities than then can be judged likely…but it still speaks to creative hobbling to produce a result.

        • Cannonaire
        • 3 years ago

        Great response! I see what you mean.

        I do believe that CPU comparisons at very low resolutions to remove any GPU bottleneck is still very useful. If a CPU is choking, that means the game simulation will have problems – whether that ends up causing game affecting problems or just smoothness-related problems depends on the game and the engine. Physics are very often problematic with too-variable simulation rates.

        For a more real-world case, competitive games in particular benefit from consistent simulation times (refering to the time when a frame is started to the time it is fully rendered), which to be fair are determined by CPU and GPU. In order to make your SIM times consistent, you need to cap your framerate (in-engine) at the correct value, and to do that the first thing you need to know is how well your CPU will perform. Capping your framerate like this is something that gamers actually do, for real benefits.

        I play Overwatch. I have a 144hz monitor, and I always play with vsync off. Now my graphics card can push 144 a lot of the time, and sometimes even range as high as about 220 (instantaneous fps), but I’ve found that even when I crank the graphics and resolution down (960×540), my framerate will sometimes dip as low as 115~120 (sometimes lower, but those are momentary and don’t seem to be caused by what’s going on in-game). Even though my monitor supports 144, it is best to have consistent frametimes for input and gameplay smoothness. Now, with some graphics options turned up, I have nearly-unwavering 8.7 SIM time, as compared to what I had before when the number was swinging wildly between 5 and 30. It makes a real difference because your input consistency is completely related to full simulation time consistency.

        Sorry for the wall of text.

    • Luminair
    • 3 years ago

    I don’t think you need to present things in a way more complex than street lights. If I said Intel’s chip produced sufficient frame rates 99% of the time, and AMD 98% of the time, everyone would get that.

    I’ve mentioned in the past that my preferred system would be a red box for percentage of frame times below 30fps equivalent, a yellow box for 30-60, and a green box for 60+. That’s the standard 3DFX used since the 1990s — 60fps has long been the aspiration of the high end gamer. And today it might be 144hz!

    I personally don’t see why anyone needs to know how much time is spent 31fps, 32fps, 33fps, etc, which is what the histogram tells you. I think I’m repeating what I told TR and PCper when this originally came up.

    • Wonders
    • 3 years ago

    Re: Damage’s suggestion — How about simply converting the Y axis to time, instead of using frame count? Or equivalently, use percent (proportion) of the test run’s duration.

    In other words, the distinction here is instantaneous [u<]weighted[/u<] FPS, meaning each data sample is weighted for the amount of time it takes up. I am a massive, longtime fan of all Scott's fantastic work, and he got so much right, but I think he really missed a key aspect way back in his original analysis and presentation of frame data. These histograms do bring us a step closer to the future. When Scott claims that "time spent beyond X" is simpler to interpret than histograms, recall that TR had to rely on JavaScript switching to show multiple frame rate targets. How is that simpler than one familiar graph style everybody studied in 6th grade science class, which allows for easy comparison of all frame rate targets at a glance, with no need for flipping back and forth between multiple graphs? If I want to compare "badness" AND "goodness" between multiple devices in a single graph, in a purely visual way, then overlaying multiple line graph histograms seems like a really strong avenue for further experiments. We read Tech Report because want the whole story. I want to know why Intel is one kind of camel, and Ryzen is another. This is how breakthroughs are made. If I just wanted to know "Is it smooth?" I'd just need a one word answer, not a graph. Instead, I want to know how, and why, where, and under precisely which conditions every piece of hardware both is and is not smooth. The whole story. In other words, I crave deeper scientific analysis of familiar gaming benchmarks, not NewRelic server-style "health dashboard" inspired charts.

      • derFunkenstein
      • 3 years ago

      That would be kind of fun. No idea how to do it in a chart, but the SUMIF equations that ImSpartacus suggested would probably be involved. Then you could see [b<]how long[/b<] the game ran at any given instantaneous framerate. Additionally, doing this wouldn't marginalize the really bad outliers like Scott was concerned about.

        • superjawes
        • 3 years ago

        Sounds like a side-project that some gerbil can take on. If they could get the raw data, they could play around with some of these ideas and maybe find a new way to display it all.

        ::cough::hint::cough

      • MrJP
      • 3 years ago

      Yes. Simply multiply the number of frames in each bin by the mean frame time for that bin, and you get the time spent in each bin. This would add weighting to the slower frames and make them stand out rather more. I think this has some merit.

      However to replicate the “time beyond x” plots in this format would require the data to be presented in cumulative form, so each bar would include the total time of all frames slower than the current bin. As you can imagine this would result in the largest bars on the RHS of the plot (the rightmost by definition will represent the entire benchmark run length), and the slower frames on the LHS will again be very small and lost on the plot.

      As Andrew pointed out in comments above, this is really then just the current percentile frame time plot in a harder-to-read form. I do wonder though if the barrier here for some people is the use of frame times in these plots, and I still think switching the axis to instantaneous frame rate would make this easier to understand. Andrew also raised an important point about linearity of these measures though, so perhaps the axis needs to be non-linear if labelled in frame rates. 140Hz isn’t as big a gain over 100Hz as 100Hz is over 60Hz.

        • Wonders
        • 3 years ago

        The cumulative form (“time beyond X”) loses valuable data features for no good reason because data “overflows” into multiple buckets — it’s too narrowly tailored to show “badness” and misses out on illustrating the whole play experience.

        What’s the most intuitive way for me to get a highly precise picture of [b<]both[/b<] the performance strengths [u<]and[/u<] weaknesses of a new card or chip? Clearly, it's something along the lines of the histograms (probably with lines instead of bars for overlaying multiple cards) using instantaneous weighted FPS (weighted for amount of time each sample takes up). Note that using "percent/proportion of test run time" on the Y-axis would produce an identical graph. Might be even easier to understand than using time -- "The 1080 Ti was at 140 FPS 25% of the time," etc. Even a 12-year-old gamer who just learned to read these graphs in science class would find more value and [i<]coolness[/i<] in reading "Oh hey! It's at 30 FPS just 12% of the time. And it's at 90 FPS 60% of the time, not bad! So... why the heck is it at 20 FPS 17% of the time?!?"

          • superjawes
          • 3 years ago

          There might be a benefit of adding time weights, but the end goal is to maximize attention to the slowdowns because they are going to have the biggest impact on perceived smoothness. This is why Scott likes the “time beyond X” graphs, because they focus almost exclusively on the slowdowns.

          Back in 2012, Scott actually captured this by using a high speed camera ([url=https://techreport.com/review/24051/geforce-versus-radeon-captured-on-high-speed-video<]here[/url<]). I would be curious to see how those cards would look on a histogram, but the video highlights what Scott was trying to say in all of the data he was presenting. Average FPS showed little difference between the two cards, but time sensitive metrics showed the stuttering making it to the display (in fact, the Radeon looked like it had a higher "regular" frame rate, but the severe spikes in frame times stood out like a sore thumb). Anyway...adding time weight to the histograms might help to bridge this gap, but at the end of the day, the metrics need to show what the experience is like at its worst, because that is what will determine the overall experience.

    • Tirk
    • 3 years ago

    Thanks, I like articles like this looking into finding better ways at displaying the information.

    Have you considered changing the 99th percentile graph in terms of Hz. That would allow someone looking at the numbers to easily compare them to the target monitor’s refresh rate that they wish to use and show the strongest setup with the higher number.

    10ms 99th percentile would score 1000/10=100Hz meaning that the tested setup would confidently produce fluid frames on a 100Hz monitor or lower at tested resolution.

    16ms 99th percentile would score 1000/16=62.5Hz meaning that the tested setup would confidently produce fluid frames on a 62.5Hz monitor or lower at tested resolution.

    Even for VRR monitor comparison that has a 90Hz-30Hz range getting a setup that scored 60Hz would reliably produce 99% of its frames within the VRR range.

    It could also help make the argument against mismatching a super high costing and performance setup that will not display any difference on the monitor that its targeted at using. E.G. spending substantially more on a setup that scored 100Hz over one that scored 60Hz would provide negligible benefit to someone looking to use a 60Hz monitor. In either case, these numbers seem easier to discuss than whether a 10ms score or 16ms score is worth it for someone to consider. Hope this suggestion helps.

    • sophisticles
    • 3 years ago

    What a complete waste of time, sorry guys but all this B.S. you have devised with graphs and analyzing comes across as just that, total B.S. designed for the sake of taking the exact same data every other site has and making it seem like something special.

    All the work into analyzing frame rates and time spent in rendering a frame and all that crap is a waste of time for one simple reason: the rate at which frames can be rendered does not correspond to the rate at which frames can be displayed.

    When a video card renders a frame internally it gets stored in video ram, which is segmented into a back buffer and a front buffer, it stays in queue waiting to be displayed after the current frame is being displayed BUT sometimes it ends up not being needed and gets flushed out of memory (it’s thrown away), meaning your analysis is useless because you don’t know whether or not not the frame was displayed or not.

    Even minimum fps doesn’t tell you much, because even if the minimum fps is 90fps, meaning that the frame rate never drops below 90 fps, if your refresh rate is 75Hz then the fastest that frames can be displayed at is 75fps, meaning that every second that passes you have a surplus of at least 15 frames, meaning that within a few minutes all the video cards buffer will be used up and need flushing, meaning it’s likely you will notice a slight hiccup as the old frames are thrown out to make room for new ones.

    More importantly, all one has to do is look at console gaming, where a game like Halo, running at less than 30fps, on a PS or XBox gives a smooth and enjoyable gaming experience.

    So basically you wasted your time analyzing a problem that doesn’t exist.

    Edited because I typed the wrong word and everyone is having a spasm over it.

      • cygnus1
      • 3 years ago

      Troll, please go away.

        • sophisticles
        • 3 years ago

        Excellent rebuttal.

          • cygnus1
          • 3 years ago

          I thought so.

          • Jigar
          • 3 years ago

          Unfortunately we couldn’t come up with anything better than this for your nonsense.

      • YukaKun
      • 3 years ago

      Where to begin even…

      It’s like… When you… And then if…

      No, I don’t even. I’ll just go talk to a wall. Seems like a better time investment.

      Cheers!

      • Jeff Kampman
      • 3 years ago

      ok

      • Chrispy_
      • 3 years ago

      [quote=”sophisticles”<]the rate at which frames can be displayed does not correspond to the rate that frames can be displayed.[/quote<] Mind. Blown.

        • ahmedabdo
        • 3 years ago

        Believe it or not, I think I read it more than ten times trying to make something out of this line!

        • sophisticles
        • 3 years ago

        I mistyped, obviously, what I meant to say was “the rate at which frames can be rendered does not correspond to the rate at which frames can be displayed”.

        But if you guys want to keep pretending that something unique is being done with this testing far be it from me to interfere with your Reality Distortion Field.

      • Mr Bill
      • 3 years ago

      TR actually measures what gets to the monitor.

      Interesting name…

      kind a reminds me of…

      “Is is solipsistic in here or is it just me?”

        • sophisticles
        • 3 years ago

        No, it doesn’t, by definition a monitor’s refresh rate is:

        [url<]http://www.windowscentral.com/what-does-refresh-rate-mean-your-pc-monitor[/url<] "The refresh rate of your monitor refers to how many times per second the screen refreshes the image on it. It's measured in hertz (Hz), and the higher the number the more times per second your monitor refreshes." It is impossible to display a number of frames greater than the refresh rate, what TR and all their fancy B.S. is measuring is how many frames are rendered internally by the graphics card, which means jack squat when the rate of rendered frames is greater than the refresh rate. It's no different than someone taking the exact same hamburger that everyone else serves but putting it in a fancy bag, still fattening, still greasy, still bad for you.

          • cygnus1
          • 3 years ago

          Have you heard of tearing? That’s the monitor displaying more frames per second than the refresh rate…

          • Mr Bill
          • 3 years ago

          No, its [url=https://techreport.com/review/24553/inside-the-second-with-nvidia-frame-capture-tools<] more like this.[/url<] I think there might be a later article that showed the setup and the latest rig better.

            • Jeff Kampman
            • 3 years ago

            We actually don’t use FCAT to gather frame-time data and haven’t for a long time. Scott explained why in this piece: [url<]https://techreport.com/blog/28679/is-fcat-more-accurate-than-fraps-for-frame-time-measurements[/url<]

            • Mr Bill
            • 3 years ago

            Oh! Thank you for correcting my error. 😉

          • Freon
          • 3 years ago

          Many of us are running variable sync monitors these days, or very high refresh monitors. I’m not sure average or flaky minimums are any better even if you are on a fixed 60hz monitor. Averaging something like 65fps with vsync off still doesn’t tell you much about slow downs. Or even with Vsync on, doesn’t tell you much about how much jutter or 30fps play you’ll actually see. Frametime data DOES give you a really good idea about how that may apply to your 60hz fixed monitor.

          Also, TR specifically calls out the 8.3ms, 16.67ms, 33.3ms. and 50ms frame time thresholds in their reviews which would give you a good idea the threshold for a standard 60hz fixed monitor, and these thresholds are visible in this article as well.

      • Andrew Lauritzen
      • 3 years ago

      Frankly, your mental model of what’s going on is too simple, and that’s why you’re not really understanding the core issues here. If you’re willing to learn I’d recommend you review some of TR’s earlier articles on this topic. Particularly focus on the discussion related to game simulation loops, back-pressure and update timers. Your view seems to be entirely focused on the display end and while things can certainly go wrong there as well, they often go wrong much earlier in the pipeline!

      • drfish
      • 3 years ago

      You got down-voted because of your inflammatory opening paragraph and final sentence. I’d wager that your post would have largely been ignored vote-wise if you had taken a different approach (even with that gem of a typo).

      For the most part, we’re a considerate and thoughtful bunch that’s fine with constructive criticism. Tweak your approach a little bit and we may have had a nice discussion instead of this dogpile.

        • derFunkenstein
        • 3 years ago

        Agreed. “Don’t be a dick” goes a long way around here, for sure, even if I ignore that advice myself from time to time.

          • Redocbew
          • 3 years ago

          DICK CONFIRMED!

          Hoping for another WTF notification here… 🙂

          And yes, I meant the other guy. I often have to remember the same thing sometimes. Helps to avoid catching a case of footinmouth disease. 🙂

      • Ifalna
      • 3 years ago

      “More importantly, all one has to do is look at console gaming, where a game like Halo, running at less than 30fps, on a PS or XBox gives a smooth and enjoyable gaming experience.”

      Once you get used to a constant 60+, you won’t enjoy a sub 30FPS anymore.

      It has to be seen/felt to be believed. I was firmly in the camp of “30 FPS is more than enough” until I got my 1070, that enabled me to run at an almost constant 60. Now, I get cranky if I drop below 55 (darn VSync).

      Yes, obviously these analysis’ only make sense if you have a proper Monitor. No sense fussing about 90 FPS drops if you’re a poor sod like me that games on a crappy 60Hz TN. >.<

        • sophisticles
        • 3 years ago

        Simple question, do you enjoy watching movies? I do a lot of video work, in the past on a professional level and I can tell you with certainty that movies are shot in and played back at 24 fps (usually), PAL frame rate is 25 fps, NTSC is either 23.97 or 29.97 fps, and to this day there is an ongoing debate as to whether or not Peter Jackson screwed up when he shot The Hobbit in 48 fps.

        Why is it a movie like the free open source Tears of Steel, with live action footage, CGI, special effects, close ups, etc, can play back smooth as silk at 4k24p (you can download the nearly 200GB uncompressed y4m source and see it for yourself) but for some reason you guys seem to think that you need to maintain at least 60fps in order to enjoy a game? Why is it you can watch the SB being broadcast at 30i and being displayed at 30p just fine on a 50inch TV yet on a 21″ monitor with a max refresh rate of 60Hz you guys seem to think you need to have the game rendered internally at 120+fps?

        I’ll tell you why, because there has been an ongoing brainwashing campaign promoted by video card companies and tech review sites that have convinced an entire generations worth of gamers that you must have more than a constant 60fps in order to enjoy a game and I can even pinpoint when this all started: back in the day of Quake 3 and Nvidia’s Geforce cards.

        If you will recall back then you still had print tech review magazines, like MaximumPC’s mag, and somehow the test of a video card became if it could run Quake 3 at 1600×1200 at 60fps; I still remember MaximumPC making a big deal (and of course Nvidia jumped all over it) when they found that the Geforce2 Ultra 64mb was the first and only card at the time to accomplish this feat, people went crazy, and Nvidia and resellers sold a ton of the then $500 Geforce2 Ultras.

        That’s when ATI and Nvidia realized they were onto something, create a stupid artificial meaningless goal, 60fps and similar meaningless resolution goal, namely 1600×1200, and boast about how their cards could do it in some game and it’s juts escalated since then.

        Today we have guys and tech sites that go on and on about being able to game at 4k and 1440p at some ridiculous frame rate, when tests have shown that the average persons eyes are able to resolve less than 60 fps and even elite Air Force pilots can only see about 120fps under ideal conditions, when most gamers are lucky the have 1080p monitors with refresh rates of 60fps.

        But this is the world we live in now, manufactured news created with the sole purpose of driving up ratings, in this case in the form of page views and some silly analysis.

        Let me know if you guys start thinking critically, until then I leave you to your ridiculous down voting, which is even sadder when you consider that there are guys with lives so empty they actually paid for the “privilege” of being able to vote with more than one “+1”.

          • Waco
          • 3 years ago

          You have no idea what you’re talking about.

          • Firestarter
          • 3 years ago

          wait, you’re a “the eye can only see 24fps” guy?

          • MrJP
          • 3 years ago

          Movies are not interactive and have no feedback loop (unless you count walking out and asking for a refund…). Games are different.

          • Andrew Lauritzen
          • 3 years ago

          As the other commentators have replied, this is a very old but profoundly flawed comparison.

          Incidentally some pretty directed research has actually happened in this space around VR recently though so it’s no longer necessary to speculate quite as much. Suffice it to say if you really want to trick the brain into perceiving something as smooth motion you need both low persistence *and* a refresh rate somewhere in the 90-120Hz range (varies slightly person to person, but anything below that will be either very blurry or vomit territory in VR :).

          Movies have a bunch of factors that make them very different:
          a) They are non-interactive, which is very different from perceiving something like a quick mouse movement as being “1:1” with something in a game.
          b) They have a large amount of motion blur – we’ve all gotten used to this but it would be highly incorrect to claim that anyone would mistake 24fps movies for reality, even if it covered their entire field of view (see VR research).

          Suffice it to say the so-called “tests” you are referring to are generally not very principled or in a lot of cases simply don’t apply, particularly compared to the much more direct research around VR HMDs in the past few years.

          • WOOFLE
          • 3 years ago

          DUDE! Seriously. If you did professional video work you must know that frames =/= fields. games render frames. Movies display fields. Not the same thing.

          • f0d
          • 3 years ago

          movies/tv/video have so much motion blur its not funny
          when the camera pans left or right at speed its just a blurry mess
          [url<]http://i.imgur.com/xRxhXgX.jpg[/url<] thats not something i want in my fps games - EVER.! there is also this [url<]https://boallen.com/fps-compare.html[/url<] just see for yourself if you can notice the difference and this [url<]https://techreport.com/news/25051/blind-test-suggests-gamers-overwhelmingly-prefer-120hz-refresh-rates[/url<] more fps (and a monitor that can match it) = better and more fluid experience

          • Ifalna
          • 3 years ago

          “Simple question, do you enjoy watching movies?”
          While I still can enjoy them, I do miss the fluidity and dislike the blur.

          I recorded my FF-XIV wedding at 60FPS and rendered it at 30 in the first go. Couldn’t stand it. Had to re-render it at 60.

          Seeing the same clip in 60 and 30 really emphasizes the difference.

          Also, do not forget the “haptic” component of video games. It is much easier to FEEL the difference between a constantly high framerate and a constantly low one than it is to see it.

          Our eyes may perceive 24fps as “basically fluid motion” but that doesn’t mean that they can’t tell the difference to higher FPS. After all, many people can see the flickering of a 50-75Hz Tube screen and it annoys them. Once you push 100Hz (Or 100FPS) I would agree that further increases are barely perceptible.

          But 24/30 vs 60? HELL YEAH, I can see that.

      • MrJP
      • 3 years ago

      [url=https://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking<]Required reading[/url<] Once you've finished that, search for other "Inside the second" articles on TR and read those as well. Then you can come back here and apologise for being rude and aggressive when you weren't in full possession of the facts.

        • sophisticles
        • 3 years ago

        So you want me to read a self serving article full of B.S. that lays the ground work for all the rest of the B.S. articles that followed? No thanks.

          • superjawes
          • 3 years ago

          Well if you aren’t going to read the articles…then you can’t determine whether they are “B.S.” or not.

          Sounds like you just want to sling criticism without consequences. How’s that working out for you?

          • Andrew Lauritzen
          • 3 years ago

          In case you’re not aware, some very smart folks who do this stuff for a living are trying to help you out here. Cut the attitude and show a little humility in an area that you are clearly not an expert in.

      • f0d
      • 3 years ago

      lulwot?!?
      i can easily tell 30fps on my consoles (xbox one and ps3) with action and fps games 30fps is quite horrible

      why do you think many fps games are now coming out at 60fps? its because 30fps blows for shooters

      • Froz
      • 3 years ago

      I still can’t decide if it’s a troll or an honest person who is just clueless. I’m leaning toward troll now. Maybe we could have a quick poll on the subject?

        • albundy
        • 3 years ago

        did you make a decision yet? lol

        • Ifalna
        • 3 years ago

        I’d vote troll. Difficult to imagine someone being THAT resistant against knowledge and arguments.

          • MOSFET
          • 3 years ago

          sophisticles is a name I recognize well around here, so despite the bad day, I think we can dismiss the troll label. I’ve had a bad post or two and owned up to them. That probably needs to happen here too, soph.

          edit: in an effort to be helpful, soph I do think you should check out that 580Ti review linked above. For explanations like this from Damage:

          [quote<]Whoops. The Gyro comes out looking better in that chart, even though it's obviously doing a poorer job of delivering fluid motion. The solution we've devised? Rather than counting the number of frames above 50 ms, we can add up all of the time spent working on frames beyond our 50-ms threshold.[/quote<]

            • Ifalna
            • 3 years ago

            Bad posts / being wrong happens to all of us.
            Being called out on it, reading up and changing our views is how we learn.

            His reactions, however, neither display reflective thinking nor willingness to listen to new information.

        • CuttinHobo
        • 3 years ago

        A troll poll? How droll, lol.

          • Redocbew
          • 3 years ago

          What would be the toll of this droll troll poll? Will the majority vote troll? I think so.

            • CuttinHobo
            • 3 years ago

            How much troll could a droll troll poll toll if a droll troll poll could toll troll?

    • cmrcmk
    • 3 years ago

    I have two related ideas/requests that are really beyond the scope of this article but definitely related.
    1) Could these images come in a larger format for those of use using the wide layout? Some of these are very dense and a little extra spread on the horizontal axis would help (particularly the currently standard frame times graphs that show thousands of data points).
    2) Could there be a selector box for which CPUs/GPUs/SSDs/Whatever you want to compare? The existing frame time graphs are very difficult to read even though they’re split into groups of three or four products per image. Additionally trying to compare different products I’m interested in that happen to be on different images involves lots of clicking between images as a try to read both together in my mind’s eye. Using javascript to overlay the data into a single image for the products I care about would be awesome.

    • DrDominodog51
    • 3 years ago

    I really like the histograms with the same axes. Is there any chance you will include these in the 1080 Ti review?

    • dikowexeyu
    • 3 years ago

    Show lines instead of bars, so they can be superposed on the same chart.

    • XTF
    • 3 years ago

    A minimum FPS of 60 (or 144) would tell you the game is perfectly smooth, wouldn’t it?
    Minimum FPS might cause false negatives, but so might the 99th percentile FPS.

    Another thing that would be nice for future reviews: an interactive compare tool that lets you pitch any two cards against eachother and show all results for those two cards.

      • cygnus1
      • 3 years ago

      [quote<] A minimum FPS of 60 (or 144) would tell you the game is perfectly smooth, wouldn't it? [/quote<] No. Smoothness = consistent frame delivery. If the frame rate jumps around all over the place, even if it's all above 60 FPS, it will not feel perfectly smooth. I think that's best seen in the fury frame time graphs. Like Jeff said, minimum FPS is about as useful as avg. FPS. At best, it can tell you the game visual experience is or isn't total garbage. But it can't really tell you if it is kind of garbage, or in other words it can't quantify for you the level of garbageness you'd experience.

        • superjawes
        • 3 years ago

        Yeah, the problem with ANY measure of FPS is that it is taken over some nonzero time interval (usually a whole second). For example, getting 59 frames at 8 ms each + 1 frame at 528 ms , the average would still come out to 60 FPS, but you would spend half a second staring at the same frame (very not smooth).

        The only way to get around this is to use a purely time-based metric. “Instantaneous FPS” is almost there, and as Scott pointed out, “time spent beyond X ms” also captures stuttering like this.

        • XTF
        • 3 years ago

        > No. Smoothness = consistent frame delivery.

        A max frame time of 16.6 ms (min FPS of 60) guarantees you can do consistent frame delivery doesn’t it?
        And 99 percentile numbers don’t tell the whole story either.

          • cygnus1
          • 3 years ago

          [quote<] A max frame time of 16.6 ms (min FPS of 60) guarantees you can do consistent frame delivery doesn't it? [/quote<] No. A max frame time of 16.6 ms only says your minimum FPS was 60. If it fluctuates wildly between 60 and 144 (between 7 and 16 ms), it won't be smooth. Consistent frame delivery means all the frames are delivered with about the same time between them. Also, even if you have a consistent rendering of frames over 60 FPS, your monitor's capabilities and v-sync settings will determine if those frames are [b<][u<] delivered [/b<][/u<] consistently.

    • Jeff Kampman
    • 3 years ago

    All the graphs should have the same axes now so that we’re not committing a crime against statistics. Let me know if anything is still amiss.

      • Meadows
      • 3 years ago

      Looks fantastic now.

      If you do decide to use this in the future, it will require some serious explanation for readers to interpret it right. Mostly on the optimal balance between how “far to the right” the hump is (average framerate) and how “tall and/or sharp” the hump is (framerate stability).
      One possible visual aid would be coloured vertical rulers at notable fps values, similar to how you provide a cheat card for your “time spent beyond X” graphs using fps analogues.

        • Firestarter
        • 3 years ago

        agreed, looks fantastic now. Switching back and forth between the two graphs really highlights that the 7700K still produces a slightly more stable framerate as well as higher average in this benchmark. I wish it were easy to condense this information in one graph without the result being too confusing, luckily the 99th percentile fps value already does a great job at that

      • cygnus1
      • 3 years ago

      Looking a lot better. Coming up with a good way to combine data from multiple GPUs or CPUs is your next challenge. And… GO!!

        • DPete27
        • 3 years ago

        I think that’s the strength of the “Frame Times by Percentile” graphs that are currently in use at TR. They’re in the form of line graphs so you can overlay many results onto one graph. Is one better than the other? I think that’s subject to personal opinion. I think the histogram speaks more clearly to me, albeit with it’s limitation in ease of comparing many data sets.

          • DPete27
          • 3 years ago

          I wonder if the histogram data points could be plotted as a line graph. That would provide the user with the same general shape of each histogram, but with the ability to combine multiple data sets on the same graph.

      • Wonders
      • 3 years ago

      Heads up, these new graphs are not watermarked (but the originals were).

      • _ppi
      • 3 years ago

      I really like the histograms with aligned axes. I believe it tells generally better story than the frametimes chart (though that one is good for identifying “wonky” behaviour, like the old cross-fire, or spikes behaviour).

      Perhaps try using % of number of total frames on Y axis as well.

      Of all the presentations, I also like PCPer’s chart of 50-100 percentile sorted fps.

      • Anovoca
      • 3 years ago

      Thank you for taking the extra time and getting these right. I hope what started as a toss away experiment may become a regular fixture to further distinguish the TR reviews from the rest of the buffalo herd.

    • TallestJon96
    • 3 years ago

    I love that you continue to try to represent frame data in the best possible way. Tech Report, Digital Foundry, Gamersnexus, and Tom’s are right to recognize the importance of frame pacing, consistency, frame times, etc.

    As a Gsync owner, the “times spent past X” is largely irrelevant. All I care about is the time spent past 33.3ms, as that is the lower limit of gsync.

    Minimum fps is bad for the same reason average fps is bad. If you have a game cranking out 60fps, and then experience a 200ms spike, your minim frame rate is about 50, when there was a huge lag spike. If this happens over and over again, you may get a minimum frame rate of say 45 fps, but a nearly unplayable experience.

    I’ve played games at 100+ fps that are unplayable because of this.

    • Meadows
    • 3 years ago

    [quote<]"too much "furriness" is an indicator of a sub-optimal experience"[/quote<] Now that's just discrimination.

      • Firestarter
      • 3 years ago

      Well we could use choppiness as an indicator but I’m not sure if that’s ok with our attack helicopter aficionados

        • Meadows
        • 3 years ago

        I didn’t know we had any of those, but I suppose it was folly to expect otherwise.

        • DrCR
        • 3 years ago

        Helicopter. helo or heli, well, OK on those too. But any real helicopter aficionado does not accept chopper as a synonym for helicopter.

      • DrCR
      • 3 years ago

      How discriminatory of you to call out their discrimination. Such intolerance of the intolerant.

      • Redocbew
      • 3 years ago

      It’s true though. Removing furriness makes you faster. Just ask a swimmer.

    • kvndoom
    • 3 years ago

    Mikemikemikemikemike!

    • Damage
    • 3 years ago

    Jeff’s been doing good work, so I don’t want to say too much here. Still, I’m going to respectfully suggest that these histograms aren’t the best option for understanding relative animation smoothness. I tried histograms here and explained the issues:

    [url<]https://techreport.com/review/22151/nvidia-geforce-gtx-560-ti-448-graphics-card/2[/url<] Essentially, the problem is that frame counts aren't nearly as important as total time spent on long frame-to-frame delays. By counting frames only, these graphs miss the time element. A lot of time spent waiting can hide in a single frame. Histogram plots are rather complicated graphs and are harder to read than, say, a simple plot of the frame-time distribution itself. Nice words about the shape of a histogram sound smart, but one is still complicating matters needlessly. The metric that does the work one wants on this front, I think, is "time spent beyond X." It looks at time, not just frame counts, and it's simple to read and compare. It also tracks well with smoothness or stuttery animation, based on my play-testing experience while gathering results. One could perhaps construct a histogram based on time spent between different intervals based on the shape of the distribution. But I dunno. Seems complicated and not as easy to read as "time beyond X" still. I'm not sure what insights one would gain from such an exercise. Anyhow, that's my two cents.

      • ImSpartacus
      • 3 years ago

      [quote<]The metric that does the work one wants on this front, I think, is "time spent beyond X." It looks at time, not just frame counts, and it's simple to read and compare. It also tracks well with smoothness or stuttery animation, based on my play-testing experience while gathering.[/quote<] Do all gpus use the exact same amount of time to run the benchmark? It feels like the best metric for that would be "percentage of total time spent beyond X" in order to normalize for total time variances. And if you do it that way, you can summarize the results across multiple games (one game's benchmark might have a lot more total benchmark time than another). Right now, the only possibilities for "clean" summarization are avg fps (or 99th %tile equivalent fps, etc).

      • Anovoca
      • 3 years ago

      I agree that on their own, these types of charts don’t tell the whole story; however, I do believe that they work great in conjunction with the Seismograph(?) style framerate chart. When you have 2 or 3 cards you are benchmarking on these types of graphs at once, the lines can often look as thick as chalk and do a poor job of telling you how much of that line is actual time spent at that frame rate and how much of that line is just Excel connecting two scattered data points.

      • Cannonaire
      • 3 years ago

      [quote<]Essentially, the problem is that frame counts aren't nearly as important as total time spent on long frame-to-frame delays. By counting frames only, these graphs miss the time element. A lot of time spent waiting can hide in a single frame.[/quote<] Since the graphs are counting how many frames are taking certain ranges of time, you will still be able to see if there's a frame that takes, say, 500ms to render. [quote<]Seems complicated and not as easy to read as "time beyond X" still. I'm not sure what insights one would gain from such an exercise.[/quote<] You will be able to see which frametime range the majority of the frames are rendered, as well as the dropoff - or often the general area - at which there are very few frames being rendered at a slower rate. Having clear visual access to these both values in the larger context at the same time can help you figure out how well the GPU can sustain any particular framerate, with the goal being consistency. That's the reason I skip the other graphs and read "time beyond X" first - because I want to know how sustainable a specific framerate is. Those are just my thoughts on it.

      • derFunkenstein
      • 3 years ago

      I guess it depends on the actual data being presented. I’m way more forgiving of a handful 200ms spikes early on in gameplay or when walking into a new area in a game because it’s probably loading something.

      I’m less forgiving of when the action gets fierce and the action takes a prolonged dive even 2-3 seconds into the 50ms range, which might be caused by visual effects and therefore something that is likely to happen over and over. Both those scenarios could have roughly the same “time spent beyond X” values.

      If anything, what we need more than additional graphs is a narrative. Why did this spike occur? What was happening in-game when this prolonged dip occurred? In my case, though, I’m probably better served by the actual frame-time graphs than I am by a histogram, the 99th percentile, or frame time beyond X.

        • Mr Bill
        • 3 years ago

        Agreed and apropos to this it would be interesting if in addition to showing total frame count as the axis, it could also be plotted to show the time interval of progress in the game. Then peaks that occur in an interval can be directly compared between the systems. Toggling the X-axis between frame count and game interval would be easy. Although, it doubles the amount of work for the reviewer.

      • Firestarter
      • 3 years ago

      I disagree, the histograms give a great overview of the kind of performance you can expect. With proper (equal) axis you can choose your own threshold (say 16.6ms/60fps) and easily see how many of the captured frames are on the wrong side and how bad it actually gets. One giant peak in the fast bit of the graph means great performance while having bars smeared all over the graph means highly variable performance that may or may not actually be bad depending on what threshold you set. If there’s a lot of variation for example but all of it is beyond 60fps then many players won’t care.

      The “time spent beyond” graphs can illustrate the same badness, but throw away a lot of information and can as such exaggerate a problem. I think they’re still very useful to identify hitches/spikes though

      BTW, I think the histograms would integrate really nicely with the 99 percentile figure, since that represents a vertical line on the histogram where 1% of the frames observed lie on one side and 99% on the other. The 99 percentile figure would be best to compare many different setups, while histograms would be best to compare a small number of setups in depth

        • Cannonaire
        • 3 years ago

        THIS. Yes. I agree with your post completely.

        • Andrew Lauritzen
        • 3 years ago

        The problem is the way the data is displayed is still not proportional to the actual thing that matters perceptually: time. I agree with Scott that you really do want to penalize excessively long frames more than ones slightly over the threshold as those you *will* notice much more severely in practice. If a single frame takes 2 seconds (can definitely happen for various reasons), it doesn’t matter if every other frame is under 16ms, that’s not okay.

        In terms of the 99th percentile figure, we already have a graph in the reviews that it pairs perfectly with: the percentile one 🙂 It’s the same data, but presented in a more useful way that doesn’t cause people to try and integrate in their heads – which they invariably do by looking at areas which in this case is absolutely wrong 🙂

          • Jeff Kampman
          • 3 years ago

          I have nothing but the highest respect for the methods that Scott developed in his work at TR and elsewhere, and I’m not proposing to do away with any of the extremely hard work that we did to get to this point. Everything we currently do is hugely valuable.

          The way I see it as someone who had to learn this stuff from the ground up, however, is that frame times in general can be an extremely difficult concept to get a grasp of. Lower is not usually better in hardware benchmarking, and the “time-spent-beyond-X” graphs can require a lot of mental exercise to interpret for the average reader. First, we’re talking about an aggregate of time built up over the course of a test run that’s sliced off each offending frame, which is one level of abstraction; second, that time above a given threshold corresponds to time spent below an instantaneous frame rate of some importance, which is a difficult leap for most readers to make; and third, that longer bars on those graphs are actually worse, which has led to a number of misunderstandings when our reviews make their way to Reddit et al. It’s a beautiful presentation of data to anybody who’s steeped in frame times, and we try to explain it in every review, but people still end up confused.

          The nice thing about histograms, as I see it, is that they can offer an easier jumping-off point for understanding our existing methods. As I’ve presented the histogram data here, it’s evident that a buildup of frames toward the left of the chart is bad, because that’s where lower frame rates can be found, and the counting of discrete frames is at less of a remove from the raw data than the time-spent-beyond-X graphs. The fact that a frame ends up in a bad “bucket” like an instantaneous frame rate of 20-24 FPS offers an immediate value judgment, because we know that figure is “bad” from learned experience. Taller bars clustered toward the right of the chart is good; even small bars toward the left of the chart are bad. That’s not immediately obvious from the data, but I don’t think it’s an insurmountable prerequisite to instill.

          We’ve had a good back-and-forth about this, and nothing is going to be replaced in or removed from our existing presentation. It’s also unlikely that anything will change in the immediate future. However, the surprising response to this piece suggests that this presentation has struck a chord for a lot of people.

            • Andrew Lauritzen
            • 3 years ago

            > Lower is not usually better in hardware benchmarking

            I understand where you’re coming from for the rest of your post, but this bit is kinda silly. It absolutely is completely normal for “lower to be better” in a lot of benchmarking! And I refuse to believe that people can’t understand that one way or another, otherwise basically all CPU benchmarking would be impossible to comprehend 🙂

            The bigger issue with FPS is that it is *non-linear*. If it were simply flipped (say, plot -ms/frame :P) it would be strange, but fine. Because it’s non-linear though, people’s intuition about it are actually provably flawed on a pretty fundamental level. This is exactly the same phenomenon as why miles per gallon is a bad measurement and most people are moving to non-inverted measures (L/100km or similar). (One example discussion: [url<]https://carboncounter.wordpress.com/2015/06/04/it-is-time-to-stop-measuring-fuel-efficiency-in-miles-per-gallon/[/url<]). The problem with the histogram is that it doesn't actually offer any more useful data at a glance than your other metrics. It has the illusion of more subtlety without actually providing good intuition for people and thus folks will come to incorrect conclusions. I'd argue it's not only strictly less useful than the percentile graph (which contains exactly the same data, just formatted in a better way), but it's actually less useful than just your raw frame time plot as well. What can you really see from the bucketing that isn't immediately more visible by just looking at how flat, fuzzy, and high the raw frame time numbers are? The most I would do - which I also suggested to Scott at the time - is rescale the X axis on the frame time graph by time (for most benchmarks, except frame-locked timedemos) so that the lines are more comparable between different GPUs. But that's gravy. Anyways don't take any of this as a huge concern about any of this stuff - I still think you're doing some good work here 🙂 Just relaying experience as this is now something that we have looked at in the industry in some depth. I'd encourage you to be really critical on each new proposed piece of data as to whether it actually adds any understanding that the ones you already present don't. As you noted, the more things you present the more likely people will get confused and come to incorrect conclusions. Or worse, start data-mining to support pre-disposed opinions 🙂 More data isn't always better! And I'll still argue that if you just want a single number to tell you how "good" something is, 99th percentile is pretty good. The percentile graph lets you pick a different point in the curve of how much jitter you individually are willing to tolerate if you don't think 99% is strict enough. The "time beyond X" are more useful if you are locking frame-rates, using v-sync, etc: they are effectively a measure of dropped frames. The frame time graph itself is the third bit you can look at if the other data seems to conflict. I think the sequence here isn't actually that hard to understand to be honest, but I'm obviously biased 🙂

            • Jeff Kampman
            • 3 years ago

            I should say that lower isn’t usually better in the context of [i<]game[/i<] benchmarking, to be clear. Thank you for your perspective.

            • superjawes
            • 3 years ago

            We gamers love our big numbers 😉

            • tsk
            • 3 years ago

            I agree with this, I’ve read all the comments but I don’t understand why we need the histogram when we have the frame time plot?

        • Freon
        • 3 years ago

        It is true that “time beyond #ms” is very context specific given the particular system and graphics settings. Its may look very different if the user is willing to change one setting, or if the CPU used is not a screaming 7700K, or if the user has an overclocked CPU, lower or faster RAM, etc. Some minor tweak could make a big difference in a system’s ability to hit, say, that magic 16.67ms mark for a significant amount of gameplay time. Or even benchmark run specific, etc.

        Also it is overly specified if you have a variable sync monitor. The cut off becomes very arbitrary.

      • Meadows
      • 3 years ago

      I think the fact that Jeff used frametime ranges as the X axis is an important difference, and one that might make it work. Since every single column of the histogram is a certain “bucket” of frametime, this is essentially taking TR’s current approach, turning it vertical, and placing all the different time ranges side by side instead of requiring you to click a button to switch between the ranges.

      All that remains is to make it uniform by specifying an exact number of columns to use, which should ideally be constant across reviews and products. If done properly, no information should be lost compared to the current approach, but one could gain new insight in the form of a visible “smoothness profile”, similar to the visual message of the 99th percentile curve that’s also used by TR. It might even be more informative to the average reader than the 99th percentile curve.

        • Damage
        • 3 years ago

        I used a range, too.

          • Meadows
          • 3 years ago

          My mistake, you are correct. However, the idea you tried out was probably too coarse for this. I mean, there were 5 useful columns in your test.
          If we subdivide the histogram further like we see with Jeff’s try, and have it show a single product in a single colour only, we can approach a graph profile that actually looks like it means something. Which was my point. With an optimal amount of subdivision, and perhaps colour-coding (by having certain ranges of the graph shaded darker), we get a good enough sense of time out of this, which was the sense you’re worried about.

            • Andrew Lauritzen
            • 3 years ago

            The fine grained data is already included in the reviews though in the form of the percentile graph. I don’t see any compelling argument for why the PDF is more intuitive here, do you?

            • ImSpartacus
            • 3 years ago

            I think an instantaneous frame rate [i<]CDF[/i<] might be easier for most folks to understand compared to the existing frame time CDF, but that's probably a discussion for another time.

        • ahmedabdo
        • 3 years ago

        THIS!
        What Jeff eventually did, was that he rotated the (time beyond X) graphs 90 degrees CCW, put them together side by side for a single processor, and sub-divided the resulting columns into even more then our usual (8.3, 16.7, 33.3, and >50) ms categories.
        However, for me, the best graph is the raw frame time in ms/frame number([i<]or[/i<] time), though it's always very small and condensate, relatively.

          • jessterman21
          • 3 years ago

          I agree – the raw frametime graphs are my favorite, with the curves coming in a close second.

        • puppetworx
        • 3 years ago

        I immediately liked the histogram presentation because FPS is still the dominant metric and this helps newbies understand.

        Damage’s point about smoothness is spot on however. Some other sites have tried ‘rate of change of framerates’ or ‘deviation from moving average framerate’. It is possible there is a better solution, but so far I haven’t seen anything I prefer to TR’s raw data and percentile charts.

      • Mr Bill
      • 3 years ago

      You make a good point. The histogram can be used show a smooth distribution but it tends to increasingly smooth out spikes as bucket size is increased.

        • derFunkenstein
        • 3 years ago

        Maybe we could combine the two, where the histogram is “time spent on frames that take x milliseconds to render”.

      • MrJP
      • 3 years ago

      I think there’s something to be gained from both ways of looking at the data. The “time beyond x” graph is a very good measure of badness as you demonstrated previously. However the density-type plots are also useful to show the overall shape of the frame rate distribution as well. I think these just need sufficient resolution on the x-axis to avoid burying the fine detail, hence the reference to a density plot rather than just a histogram. As you mentioned, potentially you could try multiplying the frequency by the frame time at each point to get it to be a distribution of how much time gets spent at each framerate as well: a thick tail in this plot would correspond to a big bar in the “time beyond x” plot as well.

      I certainly wouldn’t advocate taking away anything from the current well-established set of plots, but it would be good to try adding something new to see how it works out. There is some risk in overloading readers with too much information, but I think as this discussion shows there’s quite a few of us who will likely lap it up. It’s good to see the excellent work of the past being taken forwards anyway, so thanks to both Scott and Jeff for your efforts.

      • cygnus1
      • 3 years ago

      Scott, I really give you a lot of credit for getting the focus of game benchmarks to be on frame render times and not on avg FPS or even minimum FPS.

      Jeff having used like 4 or 5 times the number of buckets, keeping them much smaller, than what you used when you tried histograms is what I think makes these useful. I think it’s a very good way to coarsely (but not too coarsely) visualize the same data that goes into the super dense and fuzzy frame time graphs.

      Honestly, the time spent beyond X graphs have always seemed overly summarizing to me and conveying very little about how smooth a game ran on a particular card. I guess that’s ok for comparing different cards “overall badness” but they only give you a single number with little to no real world context to compare. The time spent beyond X data is just an amount of time it was rendering slower than X.

      The way I see it is, before TR was giving us a fully zoomed out 10,000 ft view in the fuzzy frame time graphs and then gave us a fully zoomed in electron microscope view in the time spent beyong X graphs. With the histograms done as Jeff is doing them, I just get a better feel for the smoothness of the benchmark run than either of the other graphs.

      Obviously I really do like the histograms, but will also agree with what others have said: data visualization is hard. They definitely have room to be iterated and improved, especially for combining/comparing different cards/CPUs data.

      • YukaKun
      • 3 years ago

      I like the smoothed curve distance against the real curve, myself.

      I’m pretty sure you could get a function describing the raw curve data by segmenting it and then using a Fourier Transformation to smooth it out, then compare point by point.

      A smooth curve governing the time plot is the best way to see a “smooth transition” and then make a hard cut for frames under X or Y value and aggregate.

      Cheers!

      • superjawes
      • 3 years ago

      I think that there is a benefit to the histogram, and I think it could be especially powerful at pulling some better/more useful “Average” FPS numbers because the data is already presented in a statistical model. You can generally see where the component “likes” to be during the workload, or how consistently it performs overall.

      That being said, you do make a very good point about “time spent beyond X”. Hiccups have a disproportionate impact on “smoothness”, and “time spent beyond X” is the only way I have seen this detail captured (at least so far).

      • ptsant
      • 3 years ago

      I agree, there are two types of badness: transient slower framerate (bad) and frametime spikes (worse).

      Having 10sec at 30 fps is not as bad as having 1 sec in 10 fps, for example, because the 100ms frames are perceived as jaggy.

      Your proposal for time-spend-beyond X corresponds to what I described below (cumulative distribution) and is just a way of extending the current bar plots to all possible values of X.

      Btw, for those who don’t know how it looks like, read [url<]https://en.wikipedia.org/wiki/Cumulative_distribution_function[/url<] It's an equivalent representation to the histogram, but easier to read when the question is "sum of measurements over X".

      • DPete27
      • 3 years ago

      I think the purpose of this article was a suggestion to replace the “Frame Times by Percentile” graphs with histograms.

      I would agree that the “Time Spent Beyond” graphs are arguably more important than either of those other two though.

      • Freon
      • 3 years ago

      I think you’re spot on. It’s sort of interesting to see the histogram, but it ultimately is not a better way to present the data that the previous distribution and “time beyond” metrics. It definitely should not replace them. Percentile vs. frame time 2D graph, particularly between the 50% 100% worst frames, is really what I feel as a consumer I care most about.

    • DragonDaddyBear
    • 3 years ago

    I LOVE this metric. Let’s take 60 FPS for example. If 30 of those frames are generated at 8.3 (1/120th of a second) milliseconds and the other 30 frames are generated at 33.3 (1/30th of a second) and then get 60 frames of 16.7 (160th of a frame FPS) you get 60 frames a second over 2 seconds.. But those 120 frames are not very consistent or smooth. This is the issue with mean vs median. These large outliers are why homes prices are usually expressed in median vs mean (average) price.

    Keep up the great work.

    EDIT: My example wasn’t very good, but I think the mean vs median point is valid.

    • ptsant
    • 3 years ago

    Hey, thanks for constantly trying to improve.

    Given that most people simply pick an fps target and want to know how often the system meets (or not) that target, the cumulative distribution–which you already provide (time spent over ZZ)–is probably better than the histogram. You could improve upon that graph by adding some tick marks at the 30fps (tolerable for short passage), 60 fps, 90fps, 120fps and 144fps points.

    You could make nice overlaid density graphs using R and ggplot2. Importing CSV from excel is not that hard and the process can be completely automated.

      • Cannonaire
      • 3 years ago

      One advantage the histograms as presented have over time-spent-above-x graphs is that you can see the volume of frames which are close to your minimum desired rate, which is useful to know in order to see how much ‘breathing room’ a GPU has at that target instantaneous framerate.

        • ptsant
        • 3 years ago

        The volume of frames is exactly equal to the “time-spent-over-x” in a cumulative distribution plot, which examines all values of x. That’s the whole point of what I’m suggesting: you take the histogram and sum the bars for all values of X, thus being able to say, for example 99% of time over 30 fps, 90% of time over 60 fps etc, for any value of desired fps.

          • Cannonaire
          • 3 years ago

          No, you are misunderstanding. You’re suggesting the the volume of [i<]individual frames[/i<] is the same as their [i<]cumulative times[/i<]. The histogram lets you see how many frames are problematic and how problematic they are instead of just how much of the time there is a problem. Having a single frame at 2000ms is much worse than 40 frames at 50ms (20 instantaneous fps), but these would both show up as 2 seconds spent on frames taking 50ms or longer to render. You lose important detail when you put too much emphasis on time. The example is a bit extreme, but it's also realistic in many circumstances.

            • ptsant
            • 3 years ago

            Both “time over X” and “number of frames over X” are examples of cumulative distributions, which is what I suggested above.

            Anyway, using your example: the first situation would figure as having 2sec over 2000ms (or below 0.1 fps) and the second would figure as having 0 sec over 2000ms. If you plot the graph for all values of X you can still recover the information as the cumulative distribution and the density (approximated by the histogram) can be mapped one to another.

            What TR is currently doing is plotting bars for given values of X (16ms, 8ms etc). What I am saying is that there should be a curve over all values of X. So, the information of very high latency frames is still there.

            • Cannonaire
            • 3 years ago

            What is the benefit to combining the data? Ease of comprehension? In order to find specific values, you would have to subtract lower values from the higher ones.

            I concede that both approaches would provide access to the same information, but I believe that showing discrete, uncombined (quantized, but not cumulatively combined with all lower values), unweighted datapoints on a consistent scale and in context provides clearer information, and it is easier, or at least no more difficult, to understand and interpret.

            The histograms in the article – aside from labeling legibility and a ‘left side bad/right side good’ note – provide a perfectly clear, easy to understand picture of the entire dataset.

            • ptsant
            • 3 years ago

            I’m not sure you understand exactly what is a cumulative distribution plot.

            Anyway, it answers exactly the question “what proportion of samples are above (or below) X”, for any X from min(x) to max(x). That is a general definition, not specific to frame times, and it happens to correspond quite nicely to what I want, ie choose a cutoff and look how much time (or how many frames) don’t meet it. The advantage compared with what TR provides now is that you can use any cutoff you want, not limited to 8ms and 16ms, for example.

            To answer that question using a histogram you have to choose the cutoff then sum the height of the bars below (or above) the cutoff. So, it takes more work to extract that answer from a histogram. The value of the histogram is that it shows additional info, like bimodality (the case of the RyZen here!), skewness or kurtosis.

            If I have time I will generate random frame time data and plot these in several ways so that people can see what I mean.

            • Cannonaire
            • 3 years ago

            This is what I think when you say cumulative distribution plot:
            [url<]https://en.wikipedia.org/wiki/Cumulative_distribution_function[/url<] (Example picture on the right side of the page) Correct me if that is not what you mean. A histogram will allow you to use any cutoff you want, just as you could with the cumulative distribution plot. You don't need to sum the values - the peaks aren't important to smooth animation, but the valleys are. I don't see a need for the peaks unless you are trying to find average FPS or Max. FPS. And as you said, a histogram will also allow you to see bimodality, skewness, and kurtosis, which can be useful for finding issues, as with the Ryzen 7 1800X results. It also has a practical use when you want to cap your framerate at a sustainable value with the goal of consistent input and animation. *EDIT* Addressed your comment about summing values. Also, I am really enjoying our discussion. 🙂

    • TheEldest
    • 3 years ago

    There’s no reason to not align your x-axis on your histograms or to use the same bins. Excel lets you specify the bins you want to use.

    [quote<]I'm not actually sure how valuable it is to present this histogram information, because it seems easy to draw the wrong idea from these charts without familiarity with frame-time testing.[/quote<] Given the amount of content TechReport has produced discussing frame-time testing, I think it's fair to assume you have a large contingent of readers that ARE familiar with the process and can properly interpret the results. [quote<]In both cases, we can see that only a couple frames out of thousands would qualify as "minimum frame rate" frames, and it seems highly unlikely that a gamer would care about their contribution to the overall picture.[/quote<] Two things: 1. It matters a lot for VR. 'Minor' hiccups can have a large impact. 2. Maybe introduce a metric to quantify how frequently a game 'stutters'. (Maybe number of 6-second intervals that have a frame in the top 5% of times?). The point would be to show whether a setup had all of it's slow frames in a single scene or if they are sporadically spread out in a way that causes stuttering issues.

      • Jeff Kampman
      • 3 years ago

      Sorry, I’m learning that Excel 2016 makes some awful decisions about what it does and doesn’t allow you to customize about histograms. I’m working on a new set of graphs and I’ll edit them into the article when I can.

        • Firestarter
        • 3 years ago

        awesome, having equal X-axes will make those graphs so much easier to interpret

    • Bauxite
    • 3 years ago

    This mindset behind this article and similar is why I keep reading TR, nice to see a scientist approach is still out there over the usual puff and flair journalist spin on everything.

    • Wonders
    • 3 years ago

    A) Super helpful and informative analysis.
    B) I’d like to see the X axes normalized, as DPete27 suggested (crush Excel into dust, Jeff!)
    C) I’d also LOVE to see a simple line instead of bars, to enable super-slick overlay comparisons between chips. (Suggested by Den2, ImSpartacus and others)
    D) Instantaneous frame rate is great. It’s just like Instantaneous MPG shown by cars. Plus, it makes sense that because [b<]animation smoothness[/b<] is the true substrate of real-time graphics, frame rate is always going to be on people's minds. I agree with ImSpartacus: [quote<]it's a fantastic compromise between measuring what you need to be measuring (that is, frame times) and providing something that people can understand (that is, fps equivalents).[/quote<] E) E is for Encore! Encore! I am totally heading straight over to renew my Gold subscription after hitting "Add post" on this. P.S. How can I buy the TR Library a copy of [url=https://www.amazon.com/Visual-Display-Quantitative-Information/dp/1930824130<]The Visual Display of Quantitative Information[/url<]? I am excited about these new developments. I would love to see animation smoothness analyses grow in both sophistication and clarity in the years to come.

      • Jeff Kampman
      • 3 years ago

      I have all the Tufte books on my bookshelf already. Great stuff. Thanks for subscribing.

      • Cannonaire
      • 3 years ago

      Response to C):
      As far as I’m concerned, the bars convey the information more clearly than lines – especially multiple lines – would. The frame count numbers at the top of the bars along with a visual depiction for the volume of frames, directly connected to the range labels makes all of the information intuitive and clearly quantized into understandable chunks.

      Lines would look nice, but they would ultimately be much more difficult to read since they wouldn’t show counts or direct connections to the ranges.

        • ImSpartacus
        • 3 years ago

        Personally, I don’t think the exact “numbers” matter. It’s the shape of the curve.

        Honestly, if I was doing it, I’d renormalize into percentages of a whole and treat it like a probability distribution function (or maybe technically probability mass functions). When you’re working on PDFs (or PMFs), the exact probability in any one spot rarely matters. It’s all about the shape as compared to a benchmark.

        And actually, you could totally calculate moments on that mother fucker and just go whole hog.

        “The 1280 seems great, but the 690 has better average kurtisis scores.”

        That would go over great, amirite?

          • Cannonaire
          • 3 years ago

          The most important performance considerations for a game are a high enough framerate, low input lag, correlation with the simulation, and [b<][i<]consistent[/i<][/b<] frametimes (in no particular order). If you can clearly see the precise instantaneous framerates (in hz) for both the peak range at which the most frames are, and also the lower range at which there is a sharp dropoff in the frame count, you can make the best interpretations of how well a GPU can give you a good experience. The bar graphs give you the ranges and the numbers so you can easily identify what you need.

            • ImSpartacus
            • 3 years ago

            I think the problem you run into is that you’re not judging one gpu in a vacuum.

            You need a methodology that scales to at least a dozen scenarios (e.g. GPUs) being compared all at once.

            With a traditional histogram labeled with data points, everything would get busy in a hurry.

            I think it’s best to cleanly approach each of the considerations that you brought up one at a time. There are no bonus points for cramming together too much into one single data visualization.

            • Cannonaire
            • 3 years ago

            Those are some good points. I wouldn’t want to put multiple GPUs on the same graph, as that would destroy everything intuitive about it.

            I think a good solution would be to have buttons, like with the time-spent-beyond-x, but to switch between GPUs. You would be able to see all of the useful information a histogram conveys and compare it on the same scale, in the same visual space.

            • ImSpartacus
            • 3 years ago

            I agree that “switcher” buttons would be your best bet for attempting to visualize that sort of thing.

            But that’s like the last resort of data visualization. You’re supposed to avoid doing that if at all possible.

            The only time I’ve ever seen the “switcher” parameter used to switch between competitors in a hardware review is in Anandtech’s SSD consistency graphs. And that was only done because it’s literally not possible to simplify those graphs. They are what they are.

            In the context of GPU frame time testing, I feel like it IS possible to simplify and still “tell the story”. I’m an Occam’s Razer kind of guy.

            • Cannonaire
            • 3 years ago

            Yes, you’re probably right. 🙂
            I’m just putting my thoughts out there, and I probably don’t have the best solution.

            But I can say one thing about it with absolute certainty: To me at least, the histograms in the article are intuitive and convey all of the information I would want to know about a GPU’s results in a benchmark, all in one graph and at a glance.

      • Cannonaire
      • 3 years ago

      Response to D):
      For the most part, I agree with you. There are two other major factors to consider, though.

      1. Input lag/consistency. I’ll use SLI as an example here. Say you have 3 cards in SLI, and they are indeed working perfectly in AFR, each consistently outputting a frame in about 33ms for ~90fps. It seems like an ideal situation for SLI, but you will still have the same amount of input lag as if you were playing at 30fps – what you see on the screen will always be a few frames behind the simulation. Which leads into the second point:

      2. Simulation. Frames should be directly connected to the simulation. If you have perfectly smooth animation, say a new frame for every 8.3ms interval, but they don’t correspond to specific moments in the simulation, it will look smooth but it will have inconsistent motion. I don’t know how I can word that better, but hopefully it was clear enough.

      *EDIT*
      Changed my wording a bit on 2., but it’s still unclear. I don’t know how to say what I mean on that one.

    • djayjp
    • 3 years ago

    “Time spent beyond x” is the absolute best way to categorise (meaningful) performance imo. Great job, TR!

    Now if only you guys would benchmark GPUs with less than the fastest CPUs available….

    • 1sh
    • 3 years ago

    One would think an open world game like GTA5 with an heavy AI load would run better with more threads but that’s just wishful thinking.

    • illusio13
    • 3 years ago

    @ Jeff Kampman

    Excellent analysis sir, far beyond me, but presented in a manner which even my old brain can understand.

    Love the histogram, great visual aid.

    Moar please!

    • ET3D
    • 3 years ago

    To convince me that average FPS is terrible you’d have to provide some examples where it doesn’t perfectly correspond to 99th percentile results. (Unless of course you’re saying that 99th percentile results are terrible too.)

      • ImSpartacus
      • 3 years ago

      Wasson did a decent job explaining several real world examples here.

      [url<]https://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking[/url<]

        • ET3D
        • 3 years ago

        Thanks. And although it convinced me that they aren’t perfectly correlated, I still think the correlation is quite high.

          • ImSpartacus
          • 3 years ago

          The correlation is definitely high.

          It’s just that even if the two had a very high correlation coefficient of, say, 90%, it would mean that average fps is still failing roughly 10% of the time, which would be inexcusable.

          There are absolutely “diminishing returns” with this kind of thing. But that’s how statistics work. You spend half of your time ensuring that the weird edge cases all work out properly.

            • Cannonaire
            • 3 years ago

            Nothing beats looking at the entire data set, and these histograms do the best job of it I’ve ever seen.

            You can clearly see the outliers and how they compare to the bulk of the data.

            • ImSpartacus
            • 3 years ago

            I can’t argue with that. I was one of the original people that whined that Wasson should display the entire frame time distribution, not just the 90-99th (might’ve been 95-99th) percentile frames to “zoom in” on the “important” part.

            And yet, one critical part of data visualization is knowing your audience. TR’s audience needs something more digestible. To be blunt, many of them are pretty lacking when it comes to stats and data. Something like a 99th percentile frame time converted to an instantaneous frame rate is very understandable. It’s not perfect, but it captures most of the “story” in a very simple package.

          • cynan
          • 3 years ago

          As I recall, the benefits of the “Inside the Second” frame time data presentations over average FPS really became apparent when examining multi GPU setups. In particular, AMD’s crossfire commonly had scenarios where the average frame rate (relatively high) and slowest frame times (slow and not that infrequent) were not very well correlated at all, leading to the “microstutter” phenomenon.

            • Firestarter
            • 3 years ago

            microstutter would lead to histograms with 2 obvious peaks

            • Redocbew
            • 3 years ago

            Would it though? Or would it just flatten out and skew the graph to the left? I suppose it’s possible, but I can’t imagine microstutters being so consistent as to create something we might call a bimodal distribution. I’m not sure the histogram would show you how long the stutters lasted either, but just how many of them there were.

      • Cannonaire
      • 3 years ago

      I’ve never been a fan of 99th percentile results. I usually read the testing methods then scroll down past average and 99th percentile to look at the frames-beyond-x measure, then read the author’s interpretation.

        • ET3D
        • 3 years ago

        Actually, I do that too. 99th percentile doesn’t represent smoothness the way the frames-beyond-x does.

      • MrJP
      • 3 years ago

      Any review with multi-GPU results would be a good starting point.

        • Andrew Lauritzen
        • 3 years ago

        Yeah, this x 100. I don’t think it’s going too far to say that multi-GPU FPS numbers are basically manufactured marketing propaganda in a lot of cases. I could probably just completely throw out every second frame in a driver and your experience wouldn’t be a whole lot different…

    • Krogoth
    • 3 years ago

    Smug camel is smug.

      • Anovoca
      • 3 years ago

      lol, I said that already 😉

      • Redocbew
      • 3 years ago

      Camel is not impressed.

      • morphine
      • 3 years ago

      Before anyone asks, no, we’re not sponsored by camelcamelcamel.com.

        • Inkling
        • 3 years ago

        How do you know?

          • drfish
          • 3 years ago

          We’re clearly more about gerbils and buffalo.

    • Anovoca
    • 3 years ago

    A good read Jeff, but I think you give the histogram too little credit. Try swapping out the 99th percentile graph and instead show the histogram side by side with the individual performance of each chip in the frame time in ms/frame number chart. When shown side by side, the histogram should read like a summary of the frame-time chart.

      • derFunkenstein
      • 3 years ago

      I like the histogram at least like it’s presented here because once you get 2-3 sets of data on the same frame-time chart, it becomes very hard to read.

    • Den2
    • 3 years ago

    Have you tried making a single histogram with both CPUs being compared?

    Something like this:
    [url<]http://support.sas.com/kb/43/addl/fusion_43365_1_sgplot.gif[/url<] For more than 2, you could use 3D histograms to show each set, but not sure it would be a good fit. A heat map would be more presentable I suspect. Examples: [url<]http://journals.plos.org/plosone/article/figure/image?id=10.1371/journal.pone.0119563.g005&size=inline[/url<]

      • Cannonaire
      • 3 years ago

      Just an opinion: Those 3D graphs are hideous for conveying information clearly.

      The first link with two sets is really nice, though.

        • Den2
        • 3 years ago

        Agreed. Which is why I said heat maps would be better… don’t think 3D histograms are a fit for any data.

      • ImSpartacus
      • 3 years ago

      I agree. Use a line in place of the histogram bars.

      I never do a “true” histogram. Just connect a line across every data point and you’re golden. So much easier to overlap several of them without busying up the graph.

      Data visualization isn’t easy, that’s for sure.

        • drfish
        • 3 years ago

        [quote<]Data visualization isn't easy, that's for sure.[/quote<] But so much fun!

          • ImSpartacus
          • 3 years ago

          Amen, it’s sad how bad people are at it.

          [url<]https://www.reddit.com/r/dataisugly/[/url<] I die every time I go there.

        • Cannonaire
        • 3 years ago

        One of the things I like most about the histograms is how they clearly show how many frames are in each category, with simple cut-offs. A continual line would make specific information unavailable.

          • ImSpartacus
          • 3 years ago

          You’d have points connected by line segments.

          [url<]http://www.stata.com/support/faqs/graphics/gph/graphdocs/multiple-overlaid-connected-line-graph/line6.png[/url<] It seems crystal clear to me. Best of both worlds.

        • jihadjoe
        • 3 years ago

        But frametimes aren’t a line graph, but an actual representation of discrete render times, mapped to discrete time values that correspond to the refresh rate.

        It’s not like you have a slow progression from 8.3ms to 16.7ms. It’s either you make the target and display the frame in time or miss it entirely and have judder. Plotting a line is useless and misleading.

          • ImSpartacus
          • 3 years ago

          You can easily use line segments to connect the various data points.

          See below for an example of how that might look when you’re comparing several gpus head to head.

          [url<]http://www.stata.com/support/faqs/graphics/gph/graphdocs/multiple-overlaid-connected-line-graph/line6.png[/url<]

            • jihadjoe
            • 3 years ago

            And that’s great for some data, but not all data.

            Connecting the dots makes it seem like the price is gradually increasing or decreasing over the time range, but in reality the actual prices are more of a step pattern where it stays at a certain level until a date where a price drop happens, then it sharply drops down ON THAT SPECIFIC DATE.

            The same is true for the frame time graphs. It’s not like you have a series of in-between frames that slowly progress from one frametime to the next. No, it’ one frame rendered at x ms, and the next frame rendered at y ms.

            • ImSpartacus
            • 3 years ago

            That would be true if you used much larger data bins, but Kampman is using 4ms wide bins. The effect you describe is effectively non-existent with that bin size.

            Furthermore, if you [i<]don't[/i<] forgo bars, you can't effectively compare multiple result iterations. That's simply a no-go when you need to compare nearly a dozen GPUs at once. Finally, this is literally a standard practice in statistics. You jump on wikipedia and search for discrete distributions like the [url=https://en.wikipedia.org/wiki/Poisson_distribution<]Poisson Distribution[/url<] or [url=https://en.wikipedia.org/wiki/Skellam_distribution<]Skellam Distribution[/url<] or [url=https://en.wikipedia.org/wiki/Geometric_distribution<]Geometric Distribution[/url<] or [url=https://en.wikipedia.org/wiki/Hermite_distribution<]Hermite Distribution[/url<] or [url=https://en.wikipedia.org/wiki/Logarithmic_distribution<]Logarithmic Distribution[/url<] and you see that they use connected line graphs to demonstrate them. And guess what? [b<]Frame time is a discrete random variable[/b<]. It deserves to be treated just like any standard discrete distribution being compared in several scenarios.

            • MOSFET
            • 3 years ago

            @jihadjoe, nice comments here.

      • Firestarter
      • 3 years ago

      my favourite graphs are interactive ones where you can add or remove the datasets to show the comparison you’re most interested in, which also helps to keep a good overview. I really like the static plots of TR as well though, them being images makes it really easy to link to them to settle an internet debate

      edit: like these on computerbase.de for example: [url<]http://i.imgur.com/LZcjSMd.png[/url<]

      • MrJP
      • 3 years ago

      Remember these plots?

      [url<]https://techreport.com/r.x/core2-qx9650/cm3d-x2-6400.gif[/url<] As everyone else has said, overlaid line graphs are probably easier to read than 3D bar charts.

    • superjawes
    • 3 years ago

    I like the histograms. That’s quite a savvy way to improve on FPS figures that people are used to.

    Is there an easy way in Excel to mark where 30 FPS, 60 FPS, etc. are on the histograms? That might offer a quick aid to translate the graph, and it would include some of the “time spent beyond X ms” information.

      • Cannonaire
      • 3 years ago

      I agree. My biggest problem with the graphs in the article is the text for the ranges – it’s difficult to read them and know what you’re looking at quickly, even when you understand what is being shown.

    • Cannonaire
    • 3 years ago

    I would love to see histogram data for different RAM frequencies in some games. I don’t have any objective evidence or the means to gather it, but I think that maybe higher frequency system memory would give more consistent low frame times in a not-insignificant variety of games.

      • ImSpartacus
      • 3 years ago

      I’d love to see a deep dive on that sort of thing.

      And if it doesn’t appear up affect performance at realistic settings, then create a custom benchmark to do a couple things in the back ground.

      Play music, shuffle through a couple web pages, maybe watch YouTube. All, while the game is being benched.

      You always hear people claim that highly cored CPUs or extra/faster RAM or larger/faster SSDs improve a “holistic” gaming experience but not necessarily an isolated “gaming-only” experience.

      I say test it. We’re here to learn, right?

    • MrJP
    • 3 years ago

    [quote<] I've converted raw frame times into instantaneous frame rate values (not averages over a second) so the left-to-right ordering makes sense.[/quote<] Bravo. I proposed the idea of presenting instantaneous frame rates rather than frame times to Scott back at the start of the "Inside the second" analysis, but he definitely wasn't a fan. I think his objection hinged around the potential for confusion in people's minds with average FPS. However to me instantaneous frame rate makes far more sense as it ranks everything in a logical order and has a direct link to something that everyone understands from experience: you know what particular frame rates feel like, not frame times, and so end up doing the conversion in your head while looking at the results. Any chance you'd consider switching to instantaneous frame rates rather than frame times for the existing standard plots as well? As someone else commented, the histograms are nice, but you do need to get consistent bins between different components to make the comparison valid. I'm pretty sure this is possible in Excel, so I'll pass it on if I can remember/work it out.

      • ImSpartacus
      • 3 years ago

      I agree.

      And when you show a histogram, it’s 100% clear what’s going on.

    • Anovoca
    • 3 years ago

    That is one smug looking camel!

    • geniekid
    • 3 years ago

    I don’t understand “instantaneous frame-rate range”. Is that the inverse of time-to-render? It feels unintuitive to me that a single frame is produced with an instantaneous frame rate of ~60.

      • Jeff Kampman
      • 3 years ago

      1000/frame time in ms

        • Mr Bill
        • 3 years ago

        So, to be completely pedantic, is this correct?
        1000/8.3ms = 120 and that is bottom of the 120 fps histogram box
        1000/16.7ms = 59.9 and that is the bottom of the 60 fps histogram box
        1000/33.3ms = 30 and that is the bottom of the 30 fps histogram box
        1000/20ms = 20 and that is the bottom of the 20 fps histogram box
        Thus, the histogram boxes are the actual instantaneous fps intervals. We don’t mind skew to the right but very much mind significant skew to the left.

        spelling 🙂

      • ImSpartacus
      • 3 years ago

      I think it’s reasonable.

      The whole idea is that if you had any number of 16.6 ms frames, then you’d always average 60 fps.

      Seems clear to me.

      I think it’s a fantastic compromise between measuring what you need to be measuring (that is, frame times) and providing something that people can understand (that is, fps equivalents).

      • TheEldest
      • 3 years ago

      If a frame takes 17 ms to show after the one prior you could say that you achieved 1frame per 17ms (1f / 17ms). If you want to normalize your time to a 1 second interval it’s basically a unit conversion:

      (1 frame / 17 ms) = (x frames / 1s)

      x = 1 frame * 1s / 17ms * (1000ms / 1s) = 58.8 frames (58.8 fps)

      • MrJP
      • 3 years ago

      It’s like the speedometer on a car. It shows your current speed in mph without having to wait for an hour of data to measure it.

      That’s why I’m not a big fan of the raw frame times: it’s like your car displaying speed as ms/inch (or whatever one encoder tick is worth in wheel circumference…).

      • geniekid
      • 3 years ago

      I understand now. Thanks for the explanations all.

      • Redocbew
      • 3 years ago

      It confused me at first also. Now I can see how it correlates with the “time spent beyond x” graphs, but I’m still wondering what I can get out of the histograms that I can’t get from the frame times. I suppose the histograms focus more on performance while the frame times focus more on latency.

        • Beahmont
        • 3 years ago

        My understanding is that the Histograms show a very visual way of both absolute and relative ‘goodness’.

        So the other numbers and graphs [b<]show[/b<] much more about absolute performance than relative performance. But take a look at that GTA V histogram. Both chips are 'good' because they are far enough to the right of the graph, but (in this particular game) they also [b<]show[/b<] clear and unambiguous proof that the i7-7700k is just better at GTA V right now. And it's a very easy to understand visual effect in the GTA V case presented. The Histograms [b<]show[/b<] the data results in a form that can and will in certain cases be understood much more intuitively without having to do your own "Okay, now what do those number really mean" comparison.

    • TwoEars
    • 3 years ago

    And this is why I keep coming back to TR. Nice analysis Jeff.

    • DPete27
    • 3 years ago

    [quote<]These are rather chaotic graphs, and they require extremely careful reading for comparison-making purposes (be sure to note that the x-axis range is not the same between the two chips).[/quote<] You stated the solution to the problem in your article. Simply normalize the X and Y axis to cover the range of all CPUs tested in the article. That way a reader can make easy and direct comparisons between graphs. 2) Worth noting that a "perfect" situation on the histogram would be a single bar (100% of frames delivered at exactly the same rate). In the spirit of TR, where consistency is rewarded vs "bursteyness", the shape of the histogram is more important than the location along the X-axis.

      • Jeff Kampman
      • 3 years ago

      Doesn’t really work for histograms, at least not so far as I’ve found in Excel.

        • derFunkenstein
        • 3 years ago

        Can’t you pick your bins when you create the histogram? Granted I’m just playing with random numbers in an Excel 2016 document, but the data analysis tools want me to pick a cell range with my data and another cell range with my bins. I created a range that is just x numbers apart (I did 5 but I’m sure you can do others) and it bundles everything up for me and drops it in a new worksheet.

        [url<]http://imgur.com/a/HteSc[/url<]

          • Jeff Kampman
          • 3 years ago

          Yeah, I probably need to do that. Thanks for figuring it out.

            • derFunkenstein
            • 3 years ago

            Any time!

          • ImSpartacus
          • 3 years ago

          You don’t even need to use the histogram tool. It’s clunky imo.

          Just use COUNTIFS and SUMIFS to build the table manually. I do it for work all the time.

          You can combine that with some clever helper columns on your frame time data table and you could automate the entire graph creation process from the moment you paste data into excel.

            • derFunkenstein
            • 3 years ago

            I didn’t even think about that. I’ve used the histogram tool before so I’m used to it, but that would be handy if you’re not!

        • ImSpartacus
        • 3 years ago

        Use COUNTIFS and SUMIFS to build a consistent table from which your graph is “fueled” from.

        Then the axis can be consistent and, as a bonus, you can skip a few steps in the data manipulation process. Win win.

        • Mr Bill
        • 3 years ago

        I really like these histograms. Believe it or not, I was coincidentally also reading the RyZen review and wondering if histograms could be used as a display aid. I did a lot of them to display elemental distributions in rocks. Creating a consistent binning algorithm is key for comparing elements. You don’t want to arbitrarily select the number of bins. Its been rather a long time since grad school back in the 1980’s and early 90’s. But elements have a natural affinity for logarithmic distributions. I think (and you have shown) this will work with frame times also. So, I used to select the bins by taking the log of the total range which gave a number that could be used via a formula to pick the bin interval, this split the range into an equal number of logarithmic bins. When tallied up, it gave a consistent easily comparable display. I’ll see if I can find that formula. I did this both on a mainframe and with spreadsheets.

        • hugovinicius
        • 3 years ago

        Try using GNU Octave or MATLAB for plotting those histograms. Yeah, I know that they’re overkill for this task, but you can do several graphical adjustments.

      • Cannonaire
      • 3 years ago

      Response to 2)
      That, precisely, is why competitive gamers often limit their FPS (in-engine, mind you) to the highest FPS they can maintain at all times in order to keep their input as consistent as possible.

      *EDIT* Fixed my wording – meant highest, not lowest.

      • Jeff Kampman
      • 3 years ago

      After overcoming some difficulties with Excel 2016, all of the axes are normalized.

    • Unknown-Error
    • 3 years ago

    Damn! Some excellent stuff Jeff.

    And must not forget the grand-master himself the great Scott Wasson for bringing in new ways of evaluating the gaming experience.

    • ImSpartacus
    • 3 years ago

    Good to see an analysis of this issue.

    I detest how imprecise the term “minimum fps” is. Is it literally the longest frame time being covered to an instantaneous frame rate? Or maybe it’s literally just the average fps from the arbitrarily bucketed “second” that had the fewest frames? You rarely can figure out exactly what is meant by “minimum fps”. Tech review bloggers just don’t have the training/background in properly communicating their methodology.

    I actually [url=https://forum.beyond3d.com/posts/1964422<]wrote up a rather lengthy comment on my interpretation of GamerNexus's methodology[/url<] and how Scott Wasson appeared to give it his blessing in an interview, but I believe the interviewer may have accidentally mislead Wasson. I still need to validate my interpretation with GamerNexus (again, even when they put together a fantastic methodology video, it's still unfortunately imprecise). At least they seem to try to do the right thing. EDIT It looks like I misinterpreted the article. This article appears to assume that "minimum fps" is always measured as the lowest instantaneous frame rate. [quote<]The problem is that a minimum frame rate—or any instantaneous frame-rate value—doesn't tell us much at all about the overall gaming experience. [/quote<] It's undeniable that such a metric is pitifully useless in measuring the real world experience of the give benchmark, but I think the issue is more complex because I believe that "minimum fps" can also be used to describe the the sequential 1-second bucket of frames that has the fewest frames. And that sequence would be arbitrarily chosen add the benchmark runs. I'll call it "minimum average fps" to attempt to differentiate. Obviously, "minimum average fps" is also marred by high benchmark-to-benchmark variance, but it's a profoundly different metric from the lowest instantaneous frame rate. It's different in that it isn't uselessly low. For example, I believe TechSpot uses a "minimum average fps" when they use the term "minimum fps". And you can see that their minimum fps figures aren't that much lower than their average fps figures. [url<]http://www.techspot.com/review/1345-amd-ryzen-7-1800x-1700x/page4.html[/url<] That tells me they couldn't possibly be using a "lowest instantaneous frame rate" for their "minimum fps". Otherwise it would be much much lower than their average fps. That's just one example, but hopefully it's clear that this is a complex issue and this article might actually confuse readers if it bluntly assumes that all "minimum fps" metrics use a "lowest instantaneous frame rate" methodology.

      • Airmantharp
      • 3 years ago

      To use statistical terms, try replacing ‘minimum FPS’ with ‘median low FPS’.

      But that is too imprecise and would require intelligent and broad implementation to be useful. ‘Time spent beyond X’ is really the best metric, and the 99 percentile graphs and frametime plots help explain problems revealed by a lot of slowly rendered frames.

      • psuedonymous
      • 3 years ago

      “because I believe that “minimum fps” can also be used to describe the the sequential 1-second bucket of frames that has the fewest frames. ”

      Not just a one-second bucket: [b<]ANY[/b<] non-instantaneous (i.e. not just the inverted frame-time) measurement of 'FPS' will be a sliding window of [i<]arbitrary[/i<] size. The size of your sliding window will determine the 'smoothing' of the metric, and its (in)sensitivity to momentary frame delivery hitches.

        • ImSpartacus
        • 3 years ago

        Oh yeah, it’s a shit show. A second is an agonizing amount of time for something like this.

        But personally, I think that’s how most “minimum fps” implementations work.

        If they used the “minimum fps” interpretation that Kampman describes, then the “minimum fps” would almost always be uselessly low, like 0.

        And yet, it’s not. It’s often like 80% of the average fps. That tells me that there’s some thing else going on. My best guess is that it’s the “bucketing” methodology that I mention in the above comment.

      • Andrew Lauritzen
      • 3 years ago

      Yeah “minimum FPS” is a garbage term that we need to just entirely avoid at this point. As best I can tell *most* people are referring to what FRAPS reports as the “minimum FPS” which incidentally is *not* the longest frame “converted to FPS” (why “convert”… so silly) but rather than minimum of the average FPS values over ~1 second intervals as you note.

      I suspect FRAPS does it this way because if you actually report the longest frame it’s fairly useless in a lot of cases as it’s not abnormal for there to be a small number of spikes in any run. Neither measurement is particularly useful though, and 99th percentile (or even make it a bit stricter if you want) is pretty clearly better.

        • ImSpartacus
        • 3 years ago

        That’s good to know. I don’t use fraps much anymore, so I wasn’t aware of that.

        Though if that’s true (and I see no reason why it wouldn’t be), it really complicates this article. Any readers utilizing “minimum fps” are going to go, “that’s not my minimum fps. My minimum fps is relatively close to average fps. It doesn’t have the problem you pointed out.” When, of course, it’s still very flawed, just in it’s own special way that isn’t addressed in this article.

        And then that brand this article isn’t as successful/influential since it’s talking about the “wrong” minimum fps throughout.

    • JosiahBradley
    • 3 years ago

    How do I thumbs up an article? These are the reasons I come back here. Great scientific analysis of performance.

      • morphine
      • 3 years ago

      [quote<]How do I thumbs up an article?[/quote<] In the subscriptions page? 🙂 (Joking, you're already a Gold subscriber)

        • superjawes
        • 3 years ago

        That was smooth 😉

          • morphine
          • 3 years ago

          I’m a smooth operator.

      • Anovoca
      • 3 years ago

      You could always retweet Jeff’s link to the article

        • RAGEPRO
        • 3 years ago

        Absolutely. We appreciate any and all attempts to spread our message of consistent frame delivery.

    • Cannonaire
    • 3 years ago

    Hey Jeff – would it be better to use time spent beyond 6.94ms instead of 8.3ms? 144hz Monitors are far more common these days, and it’s difficult to even find 120hz monitors now.

    *EDIT* Addendum
    I love the histograms. I feel that they would be especially well-suited for competitive games in which you want to maintain a specific high instantaneous framerate at all times for the sake of input consistency.

    (For example, my system can maintain 115 FPS constantly in Overwatch, so I use the built-in framerate cap in order to keep my frametimes and input consistent, rather than having it range from 115 to 180+.)

      • Jeff Kampman
      • 3 years ago

      I’ve been thinking about this for a while; it’s probably something we need to do in our next review.

        • Visigoth
        • 3 years ago

        I concur. As CPU’s and GPU’s become more powerful, differences don’t show up as easily anymore. A more stringent 6.94ms option would be very welcomed by those of us running powerful rigs and need to know if the latest and greatest is worth the upgrade.

      • meerkt
      • 3 years ago

      Could be good to add 90Hz to fill the large gap above 60Hz. Variable refresh rate monitors are getting more common nowadays.

      And if 144Hz, I think it would be better to have in addition to 120Hz rather than instead.

      • npore
      • 3 years ago

      There are some of us, admittedly a minority, with ~144Hz monitors that would be more interested in 8.3ms because of ULMB. Haven’t been checking lately, but at least when I was looking for a new monitor ULMB topped out at 120Hz. Focusing on keeping above a frame time target is super important with ULMB.

      I imagine a lot (most?) 144Hz+ monitors have g/free-sync, which makes the distinction between 6.9 and 8.3ms less important anyway. Would be an interesting follow up poll to the current one – do you have a VRR capable monitor

        • Cannonaire
        • 3 years ago

        Regarding ULMB, that is a good point. It’s more important to sustain consistently low frametimes for ULMB at 120hz than anything at 144hz. Having finally used a 144hz monitor, it’s difficult to ever see tearing with vsync off, so it’s fine to have framerate values between 100 and 144. You have convinced me.

    • Waco
    • 3 years ago

    That bimodal distribution seems to indicate a scheduling problem, no? I’d love to see a revisit when AMD claims to have things sorted out in terms of scheduling, or with SMT disabled on current platforms.

      • chuckula
      • 3 years ago

      I think the bimodal distribution is related to scheduling and that SMT likely exacerbates the issue, but I don’t think SMT is the fundamental underlying cause.

      Instead, I think it’s because every time you reschedule between core complexes in that chip the amount of overhead for copying all the data with cache coherency between core complexes is pretty high. That’s why the right kinds of workloads that rarely invoke that type of scheduling seem to work great with RyZen but other kinds of workloads that are less deterministic tend to exhibit some issues. Turning SMT on gives the scheduler more opportunities to foul up, so it’s probably not helping, but I don’t think SMT by itself is the root cause.

        • Waco
        • 3 years ago

        If the scheduler is putting two heavy threads (that can utilize most of the resources) onto a single physical core it could create this kind of situation as well (and disabling SMT could reinforce that hypothesis if the shape of the distribution changes). IIRC that was what AMD suggested for Bulldozer/Excavator/etc chips, since it allowed the pairs of cores to run at higher clocks. That seems to be the opposite of what they’d want in an SMT system with shared resources.

        I think more testing is needed, and I’d love to see follow-ups based on any patches AMD gets into the kernel as well as mobo/firmware updates over time.

          • chuckula
          • 3 years ago

          The question then becomes, why would the scheduler do that to an AMD chip but not to an Intel chip that actually has fewer physical cores (assuming Hyperthreading isn’t having a big negative impact on the 7700K).

            • Waco
            • 3 years ago

            Because AMD’s prior chips all preferred that, and the scheduler might interpret a pair of threads on a core as a Bulldozer-like module.

            Just a WAG, but it wouldn’t surprise me if it treats Ryzen chips like a Bulldozer in terms of scheduling instead of like an Intel chip with HT…

            • ptsant
            • 3 years ago

            The OS is presented with a number of logical (not physical) CPUs which appear identical but may suffer penalties in different conditions, for example when putting two threads on the same core or when changing the affinity of a thread for a given core or when mixing FPU thread with FPU thread on the same core (bulldozer had a single FPU for a module).

            Simply put, MS does take into account the specific features of the 7700K architecture when scheduling, but we don’t know if it does as well with the Ryzen architecture. Linux support for RyZen is functional, but scheduler improvements may take several months and may have to be tuned/tested for many different workloads.

            Should it have been done for day-0? Maybe. Then again, it’s a new architecture, not the 7th iteration of a given architecture, so I suppose the work (testing, validation) is significant and the hardware had to get out.

            There should be a further 5-10% gain to be had in the situations where scheduling does seem to be problematic, such as games that improve with HT off. Situations that don’t improve with HT off probably won’t benefit much.

      • ptsant
      • 3 years ago

      Actually, mathematically speaking, a bimodal distribution means that we are sampling two different processes. So, when things go “OK”, you get the same hump as the 7700K, but when things don’t go OK, you get a hump to the left.

      My guess, as described elsewhere in this forum and as you say, is that when the scheduler puts two demanding threads on the same physical core, the output falters. Rescheduling the thread penalizes the cache and prediction algorithms.

      An intelligent scheduler should (a) protect thread-core match as long as possible (preserving the cache) and (b) only put two threads per core when there are at least 8 demanding threads running.

        • Waco
        • 3 years ago

        Exactly, yes. 🙂

        • ermo
        • 3 years ago

        The modern Microsoft under Satya Nadella probably won’t be averse to integrating a scheduler update for RyZen if this hypothesis turns out to be correct.

        • Mr Bill
        • 3 years ago

        Restating what you said for emphasis. This implies that the RyZen is actually doing very well in the hump to the right of the 7700K’s and falling back to slightly slower than the 7700K. Suggests to me that when SMT threading succeeds its actually a bit faster than the 7700K.

      • BiggieShady
      • 3 years ago

      Two humps (bimodal distribution) are simply because of two core complexes.

        • Waco
        • 3 years ago

        Perhaps.

Pin It on Pinterest

Share This