Correlation analysis on all of my data

Tags: #<Tag:0x00007f53cd29fd10> #<Tag:0x00007f53cd29fbf8>

I’ve been keeping a record of daily metrics (77 of them) for almost 4 years. I decided to write about my analysis and my findings here. The page includes some of the interactive data visualizations I created.

Also, I made the analysis available to visitors of my data blog site here. If you’ve got a CSV of QS data you can visualize it using the online tool (your data won’t get uploaded to any servers).

Let me know if you have any thoughts or feedback, looking forward to discussing it with you!


This looks fascinating. I would LOVE to use it to explore my data but I have hit a complete dead end - the table of imported metrics sometimes pulled up my list of headers and told me it couldn’t understand the dates at all (they were in AU format) so I converted them to USA format (surprisingly difficult in Numbers for MacOS) before exporting and now it’s just saying

uploadedtable = RuntimeError: undefined is not an object (evaluating 'n.filter((function(e){return e.value}))[0].date')

which is beyond my debug ability and also it’s bedtime. This is the toy version of my data I’ve been trying to get it to understand:

date,Fatigue (0-6),Insomnia (0-3),Restless Legs (0-3),Abdo pain (0-3),Bladder pelvic pain (0-3),Gallbladder pain (0-3),Pain (0-3),Hay Fever (0-3),Facial Flushing (0-3),Overheating (0-3),Skin itch (0-3),Gastro upset Nausea (0-3),Diarrhoea (0-3),Commode (0-3),Stemetil,Forgot meds 8/28/2020,2,2,0,1,3,0,2,3,1,2,3,1,0,0,0,FALSE 8/29/2020,2,2,0,1,3,0,2,3,1,2,3,3,0,0,0,FALSE 8/30/2020,2,2,0,1,3,0,2,3,1,2,3,2,0,0,0,FALSE 8/31/2020,2,2,0,1,3,0,2,3,1,2,3,2,0,0,0,FALSE

Would love any suggestions. It looks just like yours in format to me! Tried different line endings, everything.

Also - totally beside the point - but your formatting rules say integers only but your example data has decimals in it. I only noticed because my actual data has a few half values too and I wondered if it mattered :slight_smile:

Hope to get this working eventually!

Hey Ricky, thanks for giving it a go, and letting me know of the issue you faced. Really helpful that you uploaded an example here. I’ve made an update to fix the issue and confirmed that your example data works now. Why don’t you give it a shot again?

1 Like

Thanks! I got that tiny snippet to work after you updated whatever-it-was, but then I tried a slightly longer bit (13 rows) and ran into a similar glitch. I don’t want to keep pasting data into here and spamming everybody with unreadable stuff so I filled out your feedback form so you’ll have my email address instead.

One thing that would be really useful is if your parser allows to give a bit more error feedback - I noticed the page doesn’t seem to know when something has gone wrong. When it’s mis-parsing the data file the counts in the “uploaded metrics” table are all wrong, but the parser doesn’t seem to have noticed anything has gone wrong and continues to try to draw graphs and make insights and spit out error messages.

If it could tell me what line in the data it thinks is invalid it would be MUCH easier for me to know if either it’s actually invalid (in which case I can fix the data) or it’s not invalid (in which case I can tell you and you can fix something). At the moment when there’s an error I have to try to make sure every character and line in the entire file is valid which works OK if the file has 3 lines but my actual live data file has 2335 rows and 30 columns, so there’s no way known I would be able to check the whole thing for validity by eye if the parser craps out!

You’re right, the notebook isn’t very resilient to data inconsistencies. It could be much more user friendly to point out the reason why it fails to process.

That’s why I’m excited to have others give it a go, and try other CSV formats besides what’s in the example files. It’ll get better! Thanks Ricky, I’ll email you to work through the issue and get your file uploaded.

What kind of notebook is it?

@Agaricus it’s an ObservableHQ notebook, embedded into my blog.

I’ve been having a lot of fun with Observable recently. The fact that it’s javascript-based makes it much easier to make interactive data visualizations than python notebooks for example.

Has anyone else played around with Observable?

That’s what I thought! I’ve been playing around with Observable a bit. Are you able to come to the virtual show&tell on the 14th? We’re going to be talking about dashboards.