Last week, Apple released the iPhone 5C and iPhone 5S. As soon as these devices hit the streets, they’ll invariably be benchmarked. Nowadays, we have benchmarks for mobile CPUs , GPUs and even repairability . However, when we talk about responsiveness, our reviews are much more qualitative and subjective.
As a technology platform focused on low-latency streaming of apps from the cloud to mobile devices, Agawi’s 15 person team has been thinking, breathing and dreaming response time (also known as latency) on mobile for the last 3 years. Since every few milliseconds of latency reduces the responsiveness of the app being streamed, we have focused on relentlessly identifying, measuring and eliminating latency to make streaming applications to mobile devices as responsive as possible.
Today, our latency experts are using their knowledge to introduce the first quantitative and objective benchmark of app response times: TouchMarks. By introducing TouchMarks to the market, we hope to bring more rigour to discussions around touchscreen response times, device lag, streaming latency and other topics related to how responsive an application feels on a mobile device. We’ll define some terms in the space so when we talk about response times, we’re all talking about the same thing. We’ll also try to bring to light all the sources of latency that many people and companies are unaware of so they can improve their products taking them into account.
The hardware and software behind TouchMarks will be open sourced so that others can replicate our results, improve TouchMarks and use these tools to improve the responsiveness of their products and services.
Agawi's Touchscope was built in-house to get multiple samples of touchscreen response times very quickly, the Touchscope measures App Response Time (ART) by capturing the time delta between activation of the Force Sensitive Resistor on the glove and the Light Sensitive Resistor positioned over the device. The Touchscope is based on the Arduino platform and uses easily available electronic components so it's easy for any electronics hobbyist to replicate. The specs and code will be released soon.
With that, let’s get into our first release: TouchMarks I.
For TouchMarks I, we decided to measure and reveal the minimum response times of flagship smartphones from top manufacturers. Are Apple’s touchscreens more responsive than Android devices as Apple lovers claim? Or does Samsung save its best displays for its own use?
Using a combination of high frame rate cameras capturing at 240fps and custom Agawi hardware (pictured above) , we can accurately measure the App Response Time (ART), which we define as the latency from the time the user feels that they’ve touched the device’s display to the time the user sees a response on the screen. For TouchMarks I, we wanted to measure the minimum response time an app developer could expect on various devices. We built simple, optimized apps to flash the full screen white* as quickly as possible in response to a touch. The apps contain minimal logic and use OpenGL/DirectX rendering to make sure the response is as quick as possible. Since these are barebones native apps doing nothing more than filling the screen in response to a touch, this benchmark defines the Minimum App Response Time (MART) a user could experience on a mobile app on each device.
Here are the results:
As you can see, the results are remarkable. At a MART of
55ms 72ms, The iPhone 5 is twice 1.5X as responsive as any Android or WP8 phone tested. All the Android devices’ MARTs fell in the same 110 – 120ms range, with the WP8-based Lumia 928 falling into that bucket as well. (Incidentally, the ranges all span about 16ms, which is expected given the 60 Hz refresh rate of these smartphones. 1/60s = 16.6ms)
There are several possible reasons for this. Since touchscreen hardware has significant latency itself (check out this video from Microsoft Research for a visual demonstration), our best guess at Agawi is that Apple’s touchscreen hardware is better optimized or more sensitively calibrated for capturing and processing touch. Another possibility is that while the Android and WP8 code are running on runtimes (Dalvik and CLR respectively), the iPhone code is written in closer-to-the-metal Objective-C, which may reduce some latency. In future TouchMarks, we’ll compare C/C++-based Android apps to Java based apps to determine if this is the case.
Regardless of the reasons, the conclusion is clear: the best written apps on iPhones will simply feel more responsive than similar apps on the current gen of Android devices. (We speculate this might be a major reason why the iPhone keyboard generally feels better than the Android keyboard to many people.)
Most developers are unaware of this latency inherent in touchscreens, leading them to often dramatically underestimate the end-to-end latency of their apps. For example, if the app is calling out to the network to respond to a touch event, achieving an end-to-end latency of under 80 milliseconds is unrealistic (not even counting app processing time), even on an iPhone. Our suggestion is to add the device’s MART score to the app’s internally calculated response time to approximate the app’s end-to-end latency.
In this TouchMarks report, we explored the Minimum App Response Time – essentially the best an app could possibly do. In future TouchMarks, we’ll look at how device latencies have changed over time (including benchmarking the new 5C and 5S) and the actual response times achieved by a few apps on these devices. Stay tuned…
*Theoretically we might be able to do better by only filling a small portion of the screen. However, we felt filling the full screen is more representative of a typical app use case, which might involve panning and shaders (in games) or screen transitions (in apps). We might explore the fill rate’s effect on latency in future reports.
Special thanks to Tim English at Stanford for consulting and checking our work on TouchMarks.
UPDATE: In preparing for our TouchMarks II release, we discovered an optimization in our iOS test app that was not present in our Android or Windows Phone test apps. To keep the benchmark consistent across all devices, we have removed the optimization from our iOS test app and updated the iPhone results and graph in this post to reflect the change. We’ll be exploring the effect of the optimization in a later post. I apologize for the error.