Microsoft takes a closer look at touchscreen latency

There’s something very natural about the touchscreen interfaces on modern tablets. Seeing a user interface react fluidly under one’s fingertip is remarkably satisfying, and after some eight months with my Eee Pad Transformer, the novelty has yet to wear off. Indeed, the Asus tablet’s latest Android update has made the UI noticeably snappier, highlighting the importance of a responsive interface.

Microsoft seems to understand the importance of low-latency touch interfaces, too. Its Applied Sciences Group has posted a video illustrating the lag associated with current touchscreen technology, which is purportedly around 100 ms, and compares it to examples with 50, 10, and only 1 ms of latency.

The demo uses a projection-based display to simulate the lower latencies, and the difference is palpable at each step along the way. I’m even surprised at how much lag I can see in the 10-ms setup, although it doesn’t look like the delay would bother me while dragging around elements of a tablet UI. As the doodling example in the video shows, lower-latency interfaces are particularly important when one is drawing on the screen, be it with a stylus or a fingertip. Touchscreen latency seems particularly obvious because the gap between one’s input and the UI’s reaction can be plainly seen; keyboards, mice, and touchpads keep the input side of things well out of view.

Microsoft has set a touchscreen latency target of 1 ms for the next decade, and I hope it doesn’t take that long to achieve it. Thanks to The Verge for the tip.

Comments closed
    • Hattig
    • 8 years ago

    The latency for a touch screen device is down to the following:

    1) Touch Screen Controller – incorporates a CPU to process touch screen actions and turn them into data to be sent to the host. This requires complex analysis of 2D touch data to ascertain where fingers are touching properly, and tracking individual fingers. I can see that there could be latency here, but solvable with faster embedded CPU in the touchscreen controller.

    2) Sending data to host system – should be quick as the data packets should be small.

    3) Host system processing an interrupt when the data is received, and passing it onto the operating system – should be quick as it’s not an issue for other USB input devices like mice.

    4) The operating system manipulates the data, works out what application it is for, passes the data on to the application. Also quick.

    5) The application processes the data and does its work, updating the display. Can be slow, maybe the application can optimise feedback by rendering something quickly and immediately, then doing the real work behind the scenes.

    6) The operating system does the requested compositing of the display. Double buffered? That introduces latency.

    7) The GPU sends the current display frame to the display.

    8) The display shows it. Many monitors have a frame or two of input lag…

    There’s a lot of different systems to work on to achieve 1ms latencies – not least the 1000Hz monitor – but the obvious place to optimise is the operating system layers. Getting the touch data to the application rapidly from the interrupt rather than passing it through ten layers of library abstractions, etc.

    • Hattig
    • 8 years ago

    So I’ve just opened Sketchbook Mobile on my Samsung Galaxy SII – a reasonably complex sketching application – and the latency on the drawing of simple lines, etc, appears to be far faster than 100ms, more like the 10ms example videos. Trying with more complex brushes shows little slowdown until they are quite large and complex, and even then it is still not 100ms.

    However the 8x slowdown portions of the video reminds me of drawing on my old Palm IIIc!

    • ssidbroadcast
    • 8 years ago

    This is the same guy that came with the BRILLIANT pressure-sensitive keyboard.

    [url<]http://www.youtube.com/watch?v=80FWh_fQ_Zg[/url<] That Microsoft never released. So... yeah. Don't hold your breath that Microsoft will actually apply business to their applied sciences division.

      • TakinYourPoints
      • 8 years ago

      Yeah, if it makes it to market it probably won’t be from Microsoft. MS Research is very frustrating for me, all of the money and insane talent they have there is wasted on a company that is either too afraid or is incapable of doing anything but iterating on Windows/Office or being late to market with a me-too product.

      Things would be very different if Microsoft could convert their research into desirable products in the same way Apple does.

        • EtherealN
        • 8 years ago

        Well, this might be an illusion caused by Microsoft actually talking about their stuff at early prototype stages, and some of their stuff do end up as products – Kinect for example.

        Also, note that it doesn’t have to end up in a Microsoft-branded product for it to benefit both Microsoft and the consumer – in this instance they might for example lay the groundwork for the screen input technology that gets the user experience where they want it, and then they can license the patents to screen manufacturers in order to get this out to Windows 8 handheld devices. (Or even to all manufacturers, no matter the target product, and thus perhaps even start taking a cut from every future iPad sale 😛 )

        But yeah, if one company shows off early prototypes for all kinds of things, and another keeps everything secret until there’s a product out, of course one will look like it has a higher failure rate than the other.

          • ssidbroadcast
          • 8 years ago

          Alright, then where is my pressure sensitive keyboard?

            • EtherealN
            • 8 years ago

            Well done, you completely ignored the contents of the post you replied to! Awesome! 😀

            Your score is: OVER NINE THOUSAND!

    • axeman
    • 8 years ago

    Good on Microsoft. This exact problem is a big part of why I find the user experience less than satisfying. If all of your actions didn’t seem so “rubbery”, I’d probably be less of a tablet/smartphone naysayer.

    • DarkUltra
    • 8 years ago

    Wouldn’t 120Hz display refresh improve this further? Regular desktop use is dramatically better than 60Hz, but this seems to have eluded most developers so far. If you doubt about the improvement, read some reviews of these “3D Ready” monitors.

      • Meadows
      • 8 years ago

      Display latency is not the same as input latency, it’s only a minor portion of it.

        • Anonymous Hamster
        • 8 years ago

        They define latency as the time between the user’s touch and when the image actually changes, which is close to the definition of end-to-end latency: the time between user input action and user perception of the resulting output.

        I can’t quite understand how they will get down to 1 ms end-to-end latency when there could be as much as 16 ms from the display latency alone (assuming 60hz refresh). Since 1000hz refresh displays are fairly non-existent, then I assume he must be talking about predictive tracking in order to provide the appearance of reduced latency.

        (The only ~1000hz displays I’m aware of use micro-mirror display engines in black & white mode–rather than time-division-multiplexing them to achieve 24-bit color.)

          • Meadows
          • 8 years ago

          Yes. And did you see any colour on the special display in the video?

            • Anonymous Hamster
            • 8 years ago

            I understand that they could have made a ~1000hz refresh display for research purposes. But where do they plan to go with this product-wise? A display that constantly refreshes the whole screen at 1000hz is for the most part just wasting power. What’s needed is a display that is only active when updating the pixels that need to be changed. And if you can, for the most part, solve the problem in software (using predictive tracking), then what’s the point of reinventing the hardware?

    • ShadowTiger
    • 8 years ago

    I’m sold! Now I have an excuse not to buy a tablet until they have 1ms responses… hurray!

      • ImSpartacus
      • 8 years ago

      You’ll be waiting about a ten years. Watch the video.

    • Arclight
    • 8 years ago

    Can they also take a look at how they screwed up the desktop? For once mouse input has been accelerated since forever. Even if you uncheck the corresponding box acceleration is not disabled. For W7 just like for XP i had to download and install a .reg file to actually disable acceleration.

    For those wondering why would that bother me when using a mouse…..i’m too angry to even formulate a response.

      • Meadows
      • 8 years ago

      You probably have no idea [i<]how little[/i<] that has to do with input latency.

        • Glix
        • 8 years ago

        He’s not saying they are connected, he is saying Microsoft should get their house in order before they criticise others.

          • Meadows
          • 8 years ago

          Mouse acceleration hasn’t even been a problem ever since Vista. Uncheck the box and forget it, if you so wish.

            • Arclight
            • 8 years ago

            You just made my blood boil…Do you have proof for your wild statement? Just because they offer a box to check and uncheck doesn’t mean the result of unchecking it has the desired result.

            My proof? Here for W7:
            [url<]http://www.youtube.com/watch?v=YjCChK5-e8U[/url<] and another [url<]http://www.youtube.com/watch?v=uaAu4tF_X7I[/url<] ForXP there was CPL mouse fix (CPL stands for Cyberathlete professional league) Show me your proof that Windows 7 and XP disables mouse accel when you untick the "Enhance pointer precision" option.

            • Meadows
            • 8 years ago

            [i<]Like I care[/i<] about your low blood pressure. I would advise a pressure suit and some caffeine for that boiling. If you turn off "enhance pointer precision", [u<]then it's turned off[/u<] until applications speciflcally call for it otherwise. Most applications don't define what kind of input they want, so if you uncheck the box, all of your regular programs will be fine. If a particular [u<]game[/u<] turns it on for its own session however, that's the game's fault. Windows has nothing to do with it, besides not offering a "prevent applications from changing these settings" checkbox of course. Maybe you could request that for Windows 8. Any self-respecting game - for example Quake Live - will give you the necessary interface to bypass those problems altogether. Without downloading any "fix" from the internet. I'm not seeing the problem, and you're blaming the wrong party for no reason.

            • Arclight
            • 8 years ago

            [quote<]If you turn off "enhance pointer precision", then it's turned off until applications speciflcally call for it otherwise. Most applications don't define what kind of input they want, so if you uncheck the box, all of your regular programs will be fine. If a particular game turns it on for its own session however, that's the game's fault.[/quote<] Well specifically, let's talk about Counter-Strike:Source. I have unchecked the "Enhance...BS" and i have unchecked the ingame option for mouse acceleration and YET i STILL EFFING HAD IT. Now pls do spare me of your BS.

            • Meadows
            • 8 years ago

            A broken game. Moving on.

            • Arclight
            • 8 years ago

            That’s not a counter-argument.

            You said it would work if both Windows and the game (CSS is a more self respecting game than that browser game-Quake live) have the option to disable it. You have been proven wrong.

            • Meadows
            • 8 years ago

            Counter-Strike is not even an example of a good game, in [i<]any kind of way[/i<]. And QL is a browser-launched game, yes. So what?

            • Arclight
            • 8 years ago

            Oh i’m sorry, just cause you don’t think it’s a good game didn’t stop anyone from making effing leagues world wide with professional gamers. How many leagues and competitions did QL have? Meadows srsly, drop it.

            • Meadows
            • 8 years ago

            You still talk like a teenager. Why, in particular? It can’t be the obvious, can it?

            • Arclight
            • 8 years ago

            Well thanks for the ad hominem attack. I thought the “adult” would have more common sense. I also thought you could gracefully accept defeat…but oh well.

            • Meadows
            • 8 years ago

            Oh, I’ve been pretty graceful so far – in my opinion -, but no defeat took place.

          • WaltC
          • 8 years ago

          Who was Microsoft criticizing? There isn’t anything there except readily available general knowledge and common sense.

          Has little to do with this video, but the current crop of touchscreen devices are bizarre, imo. Do we learn to write cursively with our fingers or with pencils? Do most artists finger paint, or do they use brushes? In both cases, the human finger is the much-less precise instrument, which is why we do not use it in those capacities.

          If there are any professional typists on earth who prefer a touch-screen facsimile keyboard to a real keyboard for typing, I’ve yet to hear of a single case. It seems to me the reasons for that are both legion and obvious.

          Much PR gobbledegook about the New iPad’s “retina” display is currently making the rounds (“retina display” being not a scientific term, but a marketing term based on the-once-Holy-Grail of 300dpi in the decades-old desktop-printing genre–which has been thoroughly outclassed & exceeded by 600dpi and 1200dpi & greater printing capability today, available in consumer-grade printers. When inkjet printers, dot-matrix, and lasers were all brand new technologies and very expensive–it took a printing density of 300 dpi so that the individual “dots” in the characters could not be perceived by the human eye. That is precisely what Steve Jobs was thinking about when he made his marketing statements about pixel density–except that Jobs parlayed the description to [i<]pixel[/i<] display in his inimitable RDF style.) And, like is true for so many marketing terms, the "retina display" of the New iPad is not "actually" 300 dpi--but is considered "close enough" by Apple to be called one...;) Impressive as [url=http://www.techairlines.com/2012/03/07/apple-ipad-third/<]264 dpi [/url<] may sound, though--this [url=http://benchmarkreviews.com/index.php?option=com_content&task=view&id=107&Itemid=1&limit=1&limitstart=2<] mouse [/url<] has a top scanning dpi precision of up to 3200 dpi. Indeed, you can buy a $10 mouse at retail today which will boast a scanning precision of 800 dpi to 1000 dpi. What is the scanning precision of the human finger? I'm not really sure, except for the fact that it doesn't come close to that of an ordinary optical mouse. [Amazingly, as I was double checking the iPad's pixel density, I ran across some articles that had [i<]were very confused[/i<] about that--and claimed it was [url=http://www.msnbc.msn.com/id/46658144<]326 dpi[/url<]! Getting the New iPad confused with the iPhone--just because Apple marketing calls them both "retina displays"? Here's another such [url=http://www.latimes.com/business/technology/la-fi-tn-apple-introduces-new-ipad-20120307,0,4185055.story<]"mistake"...[/url<], and [url=http://www.newsfactor.com/story.xhtml?story_id=013001A0YW5J<]another[/url<]...indeed--there seem to be more sites that got it wrong than got it right. This one got it [url=http://arstechnica.com/apple/news/2012/03/apples-ipad-with-retina-display-quad-core-graphics-launches-today.ars?comments=1#comments-bar<]right.[/url<] Of course, our own TR got it right, too. What's remarkable is that here we are now several days after the iPad's debut and these sites still haven't corrected themselves. (I suppose that's about how much you can trust such information today.)] So, if you can't tell, I am unimpressed with touchscreen devices as I find them far too primitive to work with, and far too expensive out of the gate as I can put together a very nice desktop with significantly larger monitor, a keyboard, a mouse, a faster cpu and faster gpu (on the order of a few thousand percent at least), much better sound output, and lots of other things--for the price of the top-end max-tricked New iPad--*and* I can run a lot more software with it of a kind the New iPad does not have the horsepower to run even if you ported one of these apps/games to it. And I can upgrade it internally to my heart's content--as often, or as little, as I like, without having to sell it or give it away and buy a brand new one. My savings there are significant. Oh, did I forget how much cheaper and faster my wired Internet connection is compared to either 3G or *choke* the joke that is currently called "4G"? Yes, I am prejudiced--as if you could not tell. In 25 years of personal computing work in and out of the industry I have owned and built many desktops--but by choice have never owned a laptop. But honestly, compared to these "tablet" monstrosities, even laptops are looking pretty good to me these days...;)

        • Arclight
        • 8 years ago

        And you have sooooo little idea of what i was reffering to…….next time read until you get it.

      • Glix
      • 8 years ago

      I’ll do it for you.

      Dragging boxes round things quickly is impeded as you will overshoot due to acceleration.

      Turning quickly in fps games will effectively be the same effect as using a stick.

      Micro selecting a unit in an rts game will be harder.

      I’m sure I’ve missed something.

        • Meadows
        • 8 years ago

        Mouse acceleration [b<]does not[/b<] affect "micro-selecting", unless you throw the mouse around from one RTS unit to the next.

          • Arclight
          • 8 years ago

          It’s just sad that you don’t shut up when you don’t know what your talking about. It’s even more upseting when you down thumb people that do, like Glix.

            • Meadows
            • 8 years ago

            I’m not seeing a counter-argument.

            Acceleration doesn’t impede slight movements. In fact, in Vista and W7, acceleration – correctly – increases precision for slight movements, but multiplies quick movements. (I agree however, that acceleration was lame and imprecise in XP.) [i<]Try it for yourself[/i<], if you don't believe me. You're the one who doesn't know what he's talking about.

            • Arclight
            • 8 years ago

            [quote<]Acceleration doesn't impede slight movements.[/quote<] That's your opinion, myself, i mind acceleration very much. Now can you stfu? |Edit: No self respecting PC gamer with a reasonably good laser mouse wants his precision screwed up by accel. For the life of me i can't understand why people from Microsoft hasn't figured this out. Must be because of people like Meadows, who think it's OK for farmville.

            • Meadows
            • 8 years ago

            I’m a Quake Live player with a laser mouse, high accuracy, and reasonable precision.

            I hit upwards of 40% (accuracy) in most every match I play. I have the game’s internal classic “acceleration” option [u<]enabled[/u<] at all times to achieve it. (Windows is set to 6/11 sensitivity and no acceleration, but QL bypasses that API altogether anyway.) [i<]Everywhere else[/i<], I just play with what I get, and don't even try to compete for that reason. (Mostly because most games have terrible ways around the mouse, and they don't even have to be console ports to be that noticeably bad.)

    • Parallax
    • 8 years ago

    Good for you Microsoft! You’ve learned what display makers have never been able to.

    • dashbarron
    • 8 years ago

    Great post. This should be read by mobile device shoppers so they can get a grip on what makes a good/bad device. The Kindle Fire has a lot of problems with its touchscreen, one issue making it a hard sell to critics. I don’t think their problems are latency necessarily though, more like general responsiveness–I wonder how we quantify that.

    • ImSpartacus
    • 8 years ago

    Goddamn it, the spec sheet on my short-term dream laptop keeps getting longer!

    Now, I need a [b<][i<]10ms[/i<][/b<] 13.3" ~1080p touchscreen running in a [url=http://youtu.be/Yz2R9y9ZvkA<]Yoga[/url<]-hinged <1" chassis in front of a Haswell SoC that drives Windows 8 for at least 8hrs on a charge and holds at least 256GB of solid state storage. Haswell comes out in 2013. Windows 8 will be mature by then. Make it happen, OEMs!

      • indeego
      • 8 years ago

      You ask for so little for storage. Aim higher!

        • Usacomp2k3
        • 8 years ago

        He wants it affordable.

          • ImSpartacus
          • 8 years ago

          You’re right. While 2.5″ SSDs will be cheap by the time Haswell is out, I’d prefer a smaller mSATA drive. Those will have limited capacities, but 256GB mSATA drives already exist, so I expect they might get cheaper, but certainly not dirt cheap.

      • yogibbear
      • 8 years ago

      I would buy one of these in a HEART BEAT and pay probably $1500 for the privilege if it weighed less than 2 lb.

        • ImSpartacus
        • 8 years ago

        Yeah, it makes me so mad that we get these tiny chassis, but only 5-6 hours of battery life. As long as the machine is under an inch, it’s “thin enough”. Fill the rest of it with battery!

        This will be especially important as we move forward with high res, power hungry displays.

          • yogibbear
          • 8 years ago

          What laptop are you looking forward to (if any)?

          I always struggle to see anything as worthwhile and always glumly give up the search each year and stick with my PC, wind u100 for really hard times, iPhone 4S for imdb factoid confirmation during heated debates while watching a movie and work laptop for anything serious (Lenovo T400… unlikely to ever be upgraded till we shift to Windows 7 in 2018,,, god our IT dept sucks)

          I think the Ivybridge laptops should have pretty good battery life. I might finally not worry about them just using the HD4000 gpu as it seems to be good enough to run some old games in a tight spot lying around an airport for a few hrs. Hopefully either Lenovo, Samsung or Asus can come up with something good.

            • OneArmedScissor
            • 8 years ago

            I’m crossing my fingers that the still unannounced Thinkpad T and X series will have thin bezel screens and be a little thinner and lighter.

            And maybe, just maybe, the replacement for the T420s will have an IPS option, or at least the nicer 1080p option that the bigger models have, instead of being stuck with that weird, grainy 1600×900 panel.

            At least they should all have USB 3.0 by default. Even if they change nothing else, I could live with an “X230″ with the IPS panel, so long as I don’t have to pay $300 for an i7 dual-core to get the USB 3.0. That was just retarded.

            An Ivy Bridge successor to the Samsung Series 7 would also be great pretty much as-is, but especially if they offered quad-cores in the 14” model this time.

            • ImSpartacus
            • 8 years ago

            Tiny bezels would be nice for laptops, but there are caveats.

            But if you want a convertible, you almost [i<]need[/i<] a normally sized bezel because palm rejection probably wouldn't be perfect. And I'm ever more pissed that companies release thin bezel laptops without increasing pixel count. I can't find the laptop now, but I think it was LG that released a 13" size laptop with a 1366x768 14" screen. They expect me to get excited when they [i<]decrease[/i<] pixel density and increase power consumption. Before bezels shrink, I'd rather see companies master higher resolution and the increased power draw that they bring. Only [i<]then[/i<] will we be able to actually benefit from larger screens in smaller chassis.

            • OneArmedScissor
            • 8 years ago

            The Samsung Series 7 are thin bezel and 1600×900. Since so many Thinkpads have higher resolution options, hopefully they’re going the same route.

            There were quite a few reviews that complained about how the X220 and X1 are stuck at 1366×768, despite the IPS screens. Lenovo would be shooting themselves in the foot to ignore them.

            Some Ivy Bridge “ultrabooks” are supposed to have 2560×1440 screens, which the X220 and X1 will have to compete with.

            For once, things look pretty good for laptop screens.

            • ImSpartacus
            • 8 years ago

            I don’t think we’ll see 1440p Ultrabooks for at least a year.

            On a 13.3″ screen, that’s 220 ppi. If yields are terrible on a 9.7″ 264 ppi display, I can’t imagine that they are much better on a slightly larger (albeit slightly less dense) display.

            But you’re definitely right about the X220 and X1. The X220 was a good machine, but it was marketed to users that needed a 12.5″ 1600×900 TN display, but it had a 12.5″ 1366×768 IPS display. A painfully bad misjudgment on their part. The X1 was worse. Sure, it was skinny, but it was a skinny 15.6″ laptop with only a 13.3″ screen! And battery life wasn’t great either.

            • yogibbear
            • 8 years ago

            An Ivybridge based Zenbook with a slightly higher screen res and some improvements made to the ports would pretty much sell me.

            • ImSpartacus
            • 8 years ago

            It’s embarrassing, but I’m looking forward to the new MBPs. I love bootcamping Win7 on my MBP13’09.

            Like I said, it’s about an inch thick and it has a huge battery. It also has a full voltage processor, so I don’t need to spend extra cash upgrading the CPU to get passable performance.

            And that trackpad, oh boy. While the bootcamp drivers aren’t as good as the OSX drivers, it’s still been my best Windows trackpad experience.

            And the new ones are rumored to basically be thicker MBAs with high res screens and full voltage processors. Aside from touch functionality, that’s exactly what I want.

            As for touch-based machines, I really really like Lenovo’s Yoga laptop. While I would prefer a sturidier Thinkpad branded version (snobby, I know), that hinge design is perfect. It’s a bit thinner than I would prefer, so battery life might be mediocre. I also wouldn’t be surprised if Lenovo ruined it with a 1366×768 resolution.

            • SomeOtherGeek
            • 8 years ago

            Yea, that is just it. If Apple can do it, the others surely can, right? It just doesn’t seem so and it is frustrating the hell out of us!

      • NeelyCam
      • 8 years ago

      I would give you +10 if I could….

      This is [i<]exactly[/i<] what I'm waiting for. (Although, I could be happy with a 11.6" one as long as the resolution is the same)

    • DeadOfKnight
    • 8 years ago

    Do want

Pin It on Pinterest

Share This