Cnet’s Smartwatch Step Count Accuracy Test Uses Flawed Methodology

When choosing a fitness tracker, or smartwatch, most people go for style of features other than the actual fitness tracking. Very few reviews seem to think that accuracy is important; I’ve found that many fitness trackers are inaccurate, and I was very pleased to see that Cnet did a test of fitness trackers and smartwatches, comparing them to the Apple Watch.

Unfortunately, the methodology used is flawed. The article says:

“With those caveats in mind, I developed a testing methodology to try and reduce variables as much as possible. I wore each activity tracker or smartwatch on my left wrist at a single time and walked on a treadmill for a mile (as measured by the treadmill’s built-in distance tracker). I then compared the mileage from the treadmill to the mileage recorded on the watch. This test was performed three times with each device I tested to ensure accuracy. The same treadmill was used for the test, and I walked at the same speed (3.5 mph, which came to about 17 minutes each time).”

Most fitness trackers do quite well on treadmills; it’s easy to count steps in a situation like that. It’s much harder to count steps in real-world situations.

When I reviewed the Fitbit Charge, I found that it was very accurate on a treadmill. Compared to the Fitbit One – probably the most accurate fitness tracker, because of the fact that you wear it on your belt – I found the two to be very close:

“Both devices recorded nearly the same number of steps on the treadmill (2,048 steps for the Charge, and 1,997 for the One)…”

But let’s look at the entire paragraph that quote is taken from:

“Here are some examples. One day, I tested the devices by wearing both of them. I went about my usual business, and I walked on my treadmill for 30 minutes. Both devices recorded nearly the same number of steps on the treadmill (2,048 steps for the Charge, and 1,997 for the One), but for the rest of the day, the numbers diverged greatly. Near the end of the day, when the Fitbit Charge was at 5,000 steps, the One was almost exactly at 4,000 steps. These convenient numbers make it very easy to calculate the discrepancy between the two devices. If you take away the 2,000 steps on the treadmill, where the devices nearly matched, the Charge recorded 3,000 steps, and the one 2,000. In other words, the Charge is recording 50% more steps than the One. (I’ve noticed that the Charge records some steps when I’m asleep; a half-dozen here, a dozen there, adding up, some nights, to 50 steps or so. And, no, I don’t sleepwalk.)”

So, comparing the two by only looking at a treadmill test, they look like they’re very close. But comparing the two throughout a full day’s activity, the Fitbit Charge recorded 50% more steps than the Fitbit One.

So it’s good that Cnet decided to test the accuracy of fitness trackers and smartwatches; it’s a shame that they didn’t come up with a valid test.