What is the best format for storing data?

Quantified Self Data Formats, Export and APIs
#1

I’m looking for a preferably pre existing format that I can easily automatically add new data to.

From my current understanding the best filetypes would be JSON, possibly a database such as SQLite or a spreadsheet.

What are your experiences with this?

My main motivation for looking for a standard is so that I can easily gather most of my data trough API’s and automation, ultimately requiring very little work on my part.
The end goal is to have it all analysed trough python, and since python is the tool i’m most comfortable with, I prefer to use python to write the code to gather the data.

Apologies if this has been asked before, I could not find something similar, I’ll happily be redirected.

#2

What kind of data do you want to store ? What APIs do you want to use ?

#3

Currently I am looking at storing mood data, weather data and data from a fitness tracker.
Once that is all setup I intend to expand to incluide data from my time tracker, screen time, and just anything I could reasonably get my hands on.

Sofar I’m using an API to acces the weather data from visualcrossing, and my mood data from Jotform.

Its currently on a day by day basis, but once its all setup properly I would like to see if I can get a higher resolution, but I have not yet looked into that thoroughly.

#4

The best format for storing data is generally JSON.

#5

ok great - where are you pulling the data to ? Did you set up your own server/cloud, or is it on localhost?

#6

Currently its just stored locally, I’ve set it to update every day my laptop is turned on taking into account it might have been turned off several days.

I might setup a server at one point but for now it feels like added complexity and security risk.

Thank you for your help by the way.

#7

ok that’s awesome.

So two things:

a) it depends if you want to -at some point- add the data online or not.
b) you’d need some data model to map.

for a), I would say that if you store on your PC, a CSV format is fine, because it’s easily translatable to a database. However, if you intend at one day to add it online, then probably you would want to store in a SQL form. There are some good SQL tools that translate an excel file to them, if you are not familiar.

for b) each fitness tracker has it’s own data models, so if you intend to add only one device, you can use their docs. However, if you intend to add more than one, then you’d need a standard. (we created one here )

Does this help ?

#8

It does thank you!

I now gotta get back to tinkering, thank you for you help!

1 Like