AMD bails on BAPCo, perhaps with Via and Nvidia

BAPCo, the non-profit consortium responsible for the SYSmark benchmark, appears to have lost a number of rather important members recently. SemiAccurate broke the story yesterday, reporting that AMD, Nvidia, and Via have all quit the group over what was included in SYSmark 2012.

AMD has since confirmed its departure, and head of marketing Nigel Dessau has written a blog post explaining the exit.

While it appears that AMD was successful in getting a number of real-world workloads included in the SYSmark suite, the weighting of the results seems to be a major point of contention. Of the 390 variables SYSmark measures across 18 applications, Dessau says the overall score is predominantly defined by less than 10% of the variables tested across just seven applications. He goes on to say that there’s too much emphasis on optical character recognition and file compression. As one might expect, Dessau also takes issue with the absence of workloads that tap the parallel processing power of modern GPUs. Everyday applications that make use of GPU resources may be relatively few and far between, but how often are you waiting on an OCR routine to make sense of a scanned document?

AMD and other BAPCo members that might want to make changes to the benchmark are supposed to be able to offer their proposed modifications for a vote. However, SemiAccurate asserts that Intel "owns the process, and overrides anyone’s views, thoughts and additions." The other BAPCo members listed on the consortium’s about page include Dell, HP, Hitachi, Lenovo, Microsoft, Samsung, Seagate, Sony, and Toshiba. None of them make CPUs—or GPUs, for that matter.

Interestingly, BAPCo’s about page doesn’t list AMD, Nvidia, or Via as members. Google’s cache reveals Via was on the member list as of last week, giving credence to the report that it has also dropped out of the group.

Comments closed
    • ronch
    • 8 years ago

    I never believed that a single number can ever represent CPU performance. Giving more or less weight to heterogeneous computing abilities isn’t the answer either, and since AMD has better graphics technology they obviously want BAPCo to give more weight to that, which I wouldn’t agree to either.

    I say, separately test x86 integer, x87 floating point, MMX, SSE, 3DNow (optional), memory controller, FSB (Hypertransport or QPI), and if possible, CPU integrated graphics. That gives a clearer picture of what a CPU is faster or slower at compared to other CPUs. A single number just won’t do unless you want a distorted figure..

      • Deanjo
      • 8 years ago

      IMHO no benchmark should be given any weighting as it is highly unlikely that a system is going to experience a load where all item benchmarks become a factor in real world use. Give the raw numbers and let the person determine if those benches will effect their planned use for a machine.

    • evilpaul
    • 8 years ago

    AMD only joined BAPCo back in what…2001 because they were annoyed with how Sysmark was favoring Pentium 4s over Athlons, and revealed that it was because the benchmark heavily weighed memory bandwidth intensive (the P4’s only strength, practically) Excel list sorting. As the child of two accountants I asked both my parents at the time how much they use the Excel sorting functions. My Mom said never. My Dad said it was something he might spend 2% of his time in Excel utilizing. When Sysmark 2003 came out it seemed a lot more even handed. I’m working from memory here, so correct me if I’m wrong.

    Now this new controversy… I’ve never personally used OCR software, so I’m wondering if there’s a good one that supports GPU acceleration, but BAPCo is sticking with an older CPU only one? I guess the AMD and Nvidia approach of GPU offloading stuff really is paying off.

      • yuhong
      • 8 years ago

      FYI, I provided some links on this in another comment.

        • evilpaul
        • 8 years ago

        That link with slides from AMD was the one I was remembering. Personally, I considered the overall Sysmark score worthless, but seeing that BAPCo is still fundamentally dishonest I’ll completely disregard the “benchmark” results.

      • NeelyCam
      • 8 years ago

      I use Excel on a daily basis, sorting and analyzing data.

        • OneArmedScissor
        • 8 years ago

        Awesome. I click the mouse, type in passwords, connect to a VPN, print, scan, copy files, adjust the volume, plug in USB devices, swap CDs, open browsers, check my email, use the calculator, make backups, and read PDFs on a daily basis, sorting and analyzing data. Sometimes, I go completely insane and I actually open My Documents. Let’s include all of those things, too, and then we can finally tell if we should get the 60 MHz or 66 MHz Pentium MMX.

    • yuhong
    • 8 years ago

    Some history, FYI:
    [url<]http://www.vanshardware.com/articles/2001/august/010814_Intel_SysMark/010814_Intel_SysMark.htm[/url<] [url<]http://www.vanshardware.com/reviews/2002/08/020821_AthlonXP2600/020821_AthlonXP2600.htm[/url<] [url<]http://www.vanshardware.com/reviews/2002/08/020822_AthlonXP2600/020822_AthlonXP2600.htm[/url<] [url<]http://www.vanshardware.com/reviews/2002/08/020822_AthlonXP2600/SYSmark%202002%20Analysis%20Presentation%20FINAL.pdf[/url<]

      • Palek
      • 8 years ago

      Van’s Hardware eh? What a blast from the past. +1 for good memory!

    • UberGerbil
    • 8 years ago

    I haven’t given any thought to OCR until this moment (and so a moment is all I’ve given it), but OCR sounds like something that would profit significantly from GPU acceleration. (Of course, if the bias arguments are accurate, had AMD or nVidia submitted a GPU-accelerated OCR benchmark, the weighting would’ve changed to deemphasize it)

      • Forge
      • 8 years ago

      Getting all weepy and sore because something should be GPU accelerated, but isn’t? Arbitrarily forcing something onto a CPU-only software path, to end-user loss? This reminds me of something else… What was it called? Physics? Fizz-X?

      Shoe; other foot.

      You play or you are played, you win or you die.

        • theonespork
        • 8 years ago

        Ummm. no. Oh, and WTF, “you win or you die”?

        This is not some important, substantial, or meaningful accomplishment that proves superiority; this is a flawed benchmark that ANY neutral observer would come to the conclusion is structured to play favorites.

        Really? “you win or you die”? Really?

        I have read your comments for years on this site, hell since the days of “there is no “the” in techreport, dagnabbit” and this has to be, by far, one of your least reasoned and most flawed posts.

        Is there anybody, anywhere, with real computer knowledge that relies on BAPCO or SYSMARK for anything relevant. Apparently you do not, because you are here every day reading real benchmarks for relevant items and reacting to them. As for the “average” consumer, I know a lot of them, and I know none of them who could tell you what SYSMARK is or why it could possibly be relevant.

        “Shoe; other foot”? No. Just a realization that that a shoe designed and manufactured for a specific pair of feet for a specific athlete guarantees nothing to any other athlete that bothers to put it on.

        BTW, I am not trying to attack you personally. Really. Like I said, been a reader for years, and it has taken me years to really have any reason to take issue with you.

          • Forge
          • 8 years ago

          Fair enough, a well-written and well-worded reply.

          The play-or-played, win-or-die was pointed solely at Nvidia. The article indicates (or at least strongly implies) that those two are bitter at Sysmark’s current CPU-only OCR focus. Nvidia in this situation is particularly ironic and funny, for the reasons indicated. Nvidia making PhysX CPU-only on mixed GPU systems was fair and expected, but Bapco forgoing GPU acceleration is unfair.

          AMD, Via, and Nvidia withdrawing from Sysmark in no way validates or invalidates Sysmark’s relevance, at least in the opinions I’ve solicited. It went from irrelevant to irrelevant.

          Thanks, by the way, for the (slightly backhanded) compliment. If this is the worst I’ve dished out, some of my other posts I loathed and regretted must not have been very memorable.

          I’ve got to go pirate some Sysmark and validate my rig’s performance now. 😉

          • UberGerbil
          • 8 years ago

          I’m glad I wasn’t the only one scratching my head over this response. It was a reply to my comment, so I assumed it was directed at me, but it seemed incongruously hostile and oddly incoherent. There are certain posters on here you would expect that from, but Forge isn’t one.

    • Sam125
    • 8 years ago

    I thought it was common knowledge that Intel and to a lesser extent the system box builders pretty much dictated what’s included and what isn’t. A perfect example of an 800lb gorilla throwing its weight around if you ask me.

      • ronch
      • 8 years ago

      Intel CPUs used to have iCOMP index markings on them, which were derived using BAPCo. Funny how Intel is not part of BAPCo. Is there something happening behind the scenes here?

    • Prion
    • 8 years ago

    > Samsung

    ARM SoCs?

    > Sony, and Toshiba

    Even though Cell is over now, doesn’t Toshiba have their fingers in HPC?

    I guess HP is completely out of the Sun and Itanium games?

      • designerfx
      • 8 years ago

      there was a lawsuit over this. the answer about hp is yes.

        • destroy.all.monsters
        • 8 years ago

        If the answer about HP is yes then why are they suing Oracle over Oracle’s dropping of an Itanium line?

    • flip-mode
    • 8 years ago

    I don’t have time for any thoughtful comments until I get through with all this OCR work I have to do.

      • Farting Bob
      • 8 years ago

      Dont worry, you’ll have plenty of time while your CPU crunches the numbers for hours at a time to work out if that I in a 1 or an l, or even an |.

      Im always waiting on my OCR for everything, its easily the most important benchmark.

        • xeridea
        • 8 years ago

        I haven’t used an OCR since My P3 800Mhz. I don’t remember it taking much time then, so why would it now on a CPU 20x faster?

    • Flatland_Spider
    • 8 years ago

    Maybe they should support a test suite that is FOSS, cross-platform, and GPLed. I don’t see the value in supporting a closed source benchmark which is tied to Windows.

    [url<]http://www.phoronix-test-suite.com/[/url<] [url<]http://openbenchmarking.org/[/url<]

      • Bauxite
      • 8 years ago

      Agree, the entire chain of compilers and code should be visible. Of course then nobody would be able to turn a profit on BS benchmarketing tricks…

      • jpostel
      • 8 years ago

      Great point. I did not realize that the phoronix suite was open.

        • Palek
        • 8 years ago

        [quote<]I did not realize that the phoronix suite was open.[/quote<] Isn't that a bit like saying "I did not realize phones made phone calls," what with phoronix being linux people?

          • dpaus
          • 8 years ago

          Not all Linux code is even open, let alone FOSS.

      • Anonymous Coward
      • 8 years ago

      Well FOSS is an improvement, but real world applications are better.

        • Deanjo
        • 8 years ago

        Most of phoronix benchmark profiles do use realworld applications instead of synthetics.

          • evilpaul
          • 8 years ago

          I realize that Microsoft manages to make Word take longer to open with each iteration, but do people really do stuff with Word, Excel, etc that aren’t instant on any recent dual core CPU?

            • Deanjo
            • 8 years ago

            Oh heck yes, there are some complex spreadsheets that people have take a long time to recalculate and if you are trying to do some decent desktop publishing in Word some of those assets can really bog down the system.

      • cygnus1
      • 8 years ago

      [quote<] I don't see the value in supporting a closed source benchmark which is tied to Windows.[/quote<] That sentence should have stopped at benchmark.

    • TheEmrys
    • 8 years ago

    Unbiased often means unfunded. Its a shame it had to go this way.

      • derFunkenstein
      • 8 years ago

      I can’t decide if you’re saying that SYSmark was previously unbiased and now is going un-funded because of it, or if you’re saying it’s biased and it’s now getting more funds as a result (which doesn’t make any sense given the article)

        • khands
        • 8 years ago

        I think he’s trying to say it was unbiased but since Intel is providing the majority of the funds it has become biased (that last bit is an assumption on my part though it would make sense).

          • UberGerbil
          • 8 years ago

          Well, and as a general philosophical point: anyone willing to fund something that isn’t intrinsically profitable tends to have an agenda. Given that we don’t have a lot of philanthropic organizations devoted to advancing unbiased hardware metrics, and even academic studies have to get their money from somewhere, truly unbiased tests tend to be purely voluntary efforts.

            • dpaus
            • 8 years ago

            [quote<]"...truly unbiased tests tend to be purely voluntary efforts"[/quote<] And characterized as 'amatuer' by those whom the results do not favour.

            • UberGerbil
            • 8 years ago

            Well, certainly you should let the work speak for itself: there’s no reason to summarily dismiss the work of amateurs, but there’s no reason to be predisposed to favoring it either . There are many good reasons to be a “non-professional” at something, but sometimes it truly is because no one would pay you for that quality of work.

Pin It on Pinterest

Share This