A minute-by-minute data comparison from 10 different sleep trackers

[jina963]

Hey all, I work at Brown University in a research group that spends a lot of time thinking about sleep tracking devices and apps. Last semester, I was curious to conduct a study on how some of the really popular sleep tracking devices out there compare with each other. You can see how the devices report different levels of sleep activity throughout the night, down to the minute-by-minute data.

Here's a screenshot of an example night --

I included a rundown of the devices used and the data I was looking at. There are also links to data extraction guides and source code if you’d like to make a similar chart yourself. It was fun seeing how the devices compare to each other and with the AMI MotionLogger, and the results have been useful for improving our Sleep Coacher project Here's the link, check it out: http://sleep.cs.brown.edu/comparison/

[Dan_Dascalescu]

Isn't it odd that the "gold standard", ~$800 AMI MotionLogger device never recorded any deep sleep on any of the 9 nights?

I wish the QS Emfit were included, which from what I've seen in the forum, is the consumer-grade gold standard these days.

[Elo]

fitbit just released a new output for some of it's devices - can you run it again? it actually has sleep stages. And it looks promisingly accurate.

[Elo]

also what about garmin?

well done! I don't mean to only add questions, this is great!

[Jacky]

Fantastic study!
Should the consistent inconsistency of results between the devices be of concern?
Do they highlight the need for users to be aware of their potential reaction to inaccurate results?

[Agaricus]

Just want to add my thanks to those already here - very useful.

[jina963]

Great observation. The AMI MotionLogger only tracks Sleep vs Awakeness. I actually asked them about this in an email, and they reported their software does not offer deep sleep tracking because they believe it cannot be monitored by actigraphy/motion alone.

I didn't make a note of what stages are actually tracked by each device (e.g. only Sleep as Android claims to track REM) -- will add that soon!

[jina963]

Awesome, I haven't heard of this new output. Do you have a link to any info on that release?

I was also just shown by a friend that Fitbit's algorithm offers two modes (Normal and Sensitive tracking) on their app. So for each night, you can see two different sleep phase timelines based on which algorithm you choose. I'm looking into adding this as well.

I didn't have a Garmin device and, in my research, it didn't seem quite popular enough to be of interest. I am all ears for model recommendations, however.

Lastly -- any and all your questions are appreciated, so no worries! Thank you for asking

[jina963]

Thank you, Jacky!

These inconsistencies are concerning to me because the companies often market promises like "a better night's sleep" and "improved lifestyle and health", etc. However, given that these are consumer level devices (minus the AMI ML), I think people do take it with a grain of salt.

Fitbit offers an FAQ page on their accuracy, but never mention sleep (only fitness activities like steps). Their page on sleep stages offers some more detail, but never an explicit precautionary notice about possible inaccuracies (except to consult a doctor if you are concerned about sleep disorders). A quick search on the other devices yielded similar results.

I agree it would be nice if they would give some sort of disclaimer or at least published more on the research behind their products (!!!), but it's probably a bad business strategy...

[Dan_Dascalescu]

However, given that these are consumer level devices (minus the AMI ML), I think people do take it with a grain of salt.

Even with the QS Emfit, for me it's been far from obvious how the previous night(s) sleep influenced my alertness, focus, and willpower during the day. After years of (admittedly not super OCD) sleep tracking, I can't say I've learned anything actionable that I couldn't have figured out without a sleep tracker (i.e. sleep 7.5 or more hours a night, use earplugs, make the room dark). Whatever effects other interventions have (exercising early in the morning or late at night), they seemed to have been overshadowed by randomness and factors outside my awareness or control.

[Tom]

Garmin forerunner only measures movement, not much use for seasoned bad sleepers who don't jump around all night, just lie there! In fact, I got up and used the computer for an hour and it recorded me as asleep for the whole time!