Hello Lars,
I’m in the same process as you are 
building a QS dashboard using python. It’s something I have been thinking of for a while but haven’t started until a couple of weeks ago. Here are some of my thoughts on my experience so far as well as some of the questions you asked:
-
START WITH A VERY BASIC MINIMUM VALUABLE PRODUCT (MVP): that’s really the most important thing I believe: draw what the most basic interface would look like for you, and get into coding it right away instead of trying to build the complete / complicated final version you have in mind. Here is my MVP for example: a simple dashboard connected to my OURA ring data, and displaying the time I went to bed over the last 7 days. Looks like this:
It’s ugly (no CSS effort), its incomplete. But it is still something I DO watch everyday when I boot my PC up. 2 reasons to focus on an MVP: 1/ it’s something tangible so it motivates you to continue, 2/ you’ll already have to learn and do a lot to have even a basic thing working (chose a library for building the interface, connect data to the API, learn how to embed a graph, …)
-
For the interface I’m using dash. Advantages: easy integration of interactive plotly graphs (it’s done for that). Drawbacks: less flexibility than other solutions like React.js (which I don’t know yet, so learning dash is easier)
-
For the database: you can start with a very basic SQLite database, which is a SQL database but stored in a file (no need to install any additional DB software), then upgrade to an MySQL or an equivalent. But at the very beginning (for the mvp) just stick to csv files, then make it more robust.
-
for importing your data, you could add an “upload” block direclty in your interface (drag and drop your csv) so you don’t need to go through an intermediary email. (just an idea)
-
I personally am not interested in correlations for now, as I find through my experience that its a lot of noise. I’m focusing on building something that gives me a good and flexible view of the metrics I care about so that I can myself spot correlations just by watching the trends. Automated correlations are definitely not on my priority list.
Start small, then progressively move towards your ‘100 variables through hopefully more than 50 years’ vision.