Sharing a tool I wrote to analyze my data

Quantified Self Apps & Tools
1

I have a bunch of data sitting in different csv files. I wanted to analyze something I consider rather basic and it turned out to be impossible in excel: to take data from 2 csv files (of Zeo Sleep Manager and Oregon Scientific Heart rate logger) and match them to tell me what is the average heart rate during different phases of sleep. I know it is geeky and probably unnecessary but hey, curiosity…

I also had some free time over 2 week-ends and decided to somehow create an application for this analysis myself. Long story short, after a lot of learning & reading I managed to write something that does what I needed in Python over Google App Engine and it is possible to add your own analysis and devices to the code if you know Python. Here I’m sharing it with you guys.

It turns out this required: uploading data, converting it to a database friendly format (when necessary), do some time adjustments to make sure time ranges of data matches exactly, and then finally do the analysis with cleaned data.

Once the setup was there I also added Withings and Omron heart pressure csv readers so almost all my data was in a database. It is rather easy to add other devices. And well, when everything was in a database it also was amazingly easy to chart it to see basic/general trends.

If this crazy thing sounds interesting to you and/or you have other devices you want to do analysis with, you can download and update the source code from here. It’s rather easy to upload a project to Google app engine but if you don’t want to deal with it you can use the online app from here, there is an option to delete all data, you can check the code. With the daily limits I guess the app can serve 10-15 people per day doing uploads, analysis and all.

Thinking it’s unnecessary I didn’t document much, just stuff I didn’t want to forget but if you’re interested please let me know here and I can try to explain what I did.

Please note that I’m not an engineer so the application is probably sub-optimal, but it just works :slight_smile:

Anyways I’m just sharing, if you like it or update/improve anything let me know.

PS: on the analysis itself, I was surprised to see that in my case heart rates are lowest during light sleep (I had expected deep sleep to be lowest but hadn’t really researched)
PPS: I know the name is cheesy, I had to choose a name to upload :slight_smile:

2

Ari,
Great project! I tried to access the charts and analysis for the devices you listed with the online app and did not see an output. Can you advise if there is something I need to do differently. Thanks!

JB

3

Glad you liked it.

The idea is that you upload your data using csv files that various devices generate and the app does some stuff (chart & analysis). Did you upload data and are not seeing the charts?

4

How is it that you couldn’t get it to work in Excel? Not that I’m touting Excel, I do all my QS analysis in Python myself. It just seems really odd that you couldn’t do an analysis like that in Excel.

5

That’s exactly what I had in mind when I first sat in front of my computer: “this should be easy to do in excel”. Bottom line is that it could be done in excel but it would be so much manual work that it’d take a very long time.

If you’re interested with the details, the main problem is the structure of data coming from different devices. For example, heart rate data was in 2 second increments but zeo sleep data was in 30 second ones. To bring them to same increments through taking average of 15 data points and to same row/column structure would probably require writing a nasty macro in excel. Also there is the added complication of data starting not exactly at the same time so the need for trimming some data points etc.

The bonus point of having done this is that now all my data sits in a proper database (and in the cloud). I can extract it anytime I want and possibly add more analysis if I have another curiosity attack :slight_smile:

6

Didn’t realize that there wasn’t a sample dataset available. I’lll have to find a dataset to test it with.

7

That is awesome of you to share your work with everybody. Thanks, Ari!