Need your insights on a “Self Data Hub” ideation

Hi! I’m working on something like this. It’s not exactly a ‘platform’, but I do treat is as a personal ‘hub’ for my data:


The basic idea is

  • data exports run separately and result in raw data on your disk (e.g. json/csv/sqlite/whatever). Ideally you’ll reuse existing projects for retrieving data, GDPR exports, etc.
  • HPI handles data processing and normalizing, again ideally relying on existing repositories if they exist
  • as a result you get a nice Python API for your data
  • it’s very extensible and flexible, you can leverage all the existing software to process/visualize/transform the data

Responding to the topic starter:

  • end user owns the data: yes it’s all on your computer
  • end user determines which app has what access: possible to restrict with Docker
  • can focus on building software only by consuming data through the API: yes, that’s the goal!
  • device/source makers can focus on building the best way to collect data: sadly not something I personally can influence, but my approach helps in the sense that the device developers don’t have to write apps to visualize data etc., as long as they provide some way to export the data, I’m happy.

A couple of my projects that are already using this HPI thing as an API:

The ‘downside’ of having this as a Python library is that it requires somewhat nontrivial (at least for non-programmers) setup, e.g. running your own data export scripts, messing with Python code at times etc. But given enough time and effort, it’s totally possible to extend this to make more accessible to more people.

2 Likes