Input Desired - I'm considering creating a free open-source local-network entire QS data integrator

1

Like many here, I’m using various and sundry QS data collection devices, including sleep trackers, nutrition/diet/exercise trackers, GPS walk/hike/bike trackers, etc. All that data, so many different places, and so many vendors of these trackers want me to pay for their “cloud” aggregation of my data, and of course use my data behind my back to make money for the vendors.

So I’m considering creating a “one ring” (in the Tolkien-sense) QS-System and QS-App that would:

  1. Allow me (where needed) to manually enter QS data via the QS-APP
  2. Allow me to pull QS data from other tracker Apps via published APIs from those Apps via adapters
  3. Allow me to pull QS data from cloud-based tracker datastores from their APIs via adapters
  4. Allow me to use multiple devices (Android, iOS, Windows, Mac, Linux) running the same QS-App to allow entry and pull of QS data
  5. Use an always-on low-power Raspberry Pi (QS-System) with Wifi and Wired to act as a “local-cloud” datastore where the QS-Apps on the various devices can find a common database to sync to, and the QS-System can be setup itself for QS data pulls from IOT QS devices and cloud-based tracker datastores
  6. Allow software on PC, Mac, or Linux to pull data from the common database on the QS-System for analysis in Excel, or better pay-ware statistical tools.
  7. Allow the QS-App on the devices to perform useful but limited summary reports and statistics via a Dashboard using a plug-in framework.
  8. Allow for “back-entry” import of already recorded data from text or Word files, Excel files, CSV files, etc.
  9. Of course, management features for customization, backup/restore, import/export, etc.

Has there been any attempt to create such a framework open-source, free, non-internet-cloud that is focused primarily on phone/tablet/tracker Apps, Cloud, and IOT for data collection? Can you prove web links to their efforts or available Apps?

Also, how many people would be interested in such a system? I’m likely going to make this for myself anyways. But if I knew there was a large number of interested users, it would affect my design decisions in creating a more flexible system rather than one, far more quickly developed, that is custom only to my individual needs.

Finally, if anyone has suggestions on common ontologies for QS data entities and attributes, suggestions for free APIs available for trackers, suggestions on long-term data management, etc I’m certainly open. I’m a 30-year software/system architect and software engineer/programmer, so I do have quite alot of experience on the design and development of this. But everyone including myself always can use suggestions, recommendations, feedback, critical review, etc.

Thanks all!

3 Likes
2

I would be very interested.
Be sure to check out open mhealth open source software. It does some of the aggregation part with some of the popular vendors (It’s in Java though, wouldn’t be my first choice).

3

Thanks for the reference to Open mHealth. I had done some looking at Apple’s HealthKit for background; Open mHealth has that covered as well. The fact that it is in Java is not really a major disadvantage; Android is Java-based so quick reuse; Java can run safely as a backend (non-browser) in Windows, Mac, Linux/Unix; probably the only issue is for use in iOS Apps.

I’m not sure if the Open mHealth folks have thought about the small scale … certainly they are thinking about the large scale pay-ware App environment. There would be a tremendous jumpstart to my concept to get their datastore running on a Raspberry Pi for use by at most just a family.

My own planned concept for the QS-App’s health data model is very much in-line with theirs, with another group’s (i2b2.org), and with HealthKit. For internal convenience and data-storage-space savings I wanted to use a numeric-based ontology, but may have to stick with a textual one that can comply with these others. The Advantage of Open mHealth is that they’ve got a mapping concept already in place (at least between their ontology and HealthKit’s) and likely could adapt to others as needed.

Well … some more research to do. I need to visit Open mHealth’s forums.

4

Another open-source project is Fluxtream. All of their code is posted on Github.

2 Likes
5

This is an itch i’ve been meaning to scratch for a while. Thanks for reminding us of Fluxstream.

6

Fluxtream looks interesting, I will have to check it out.

I am also happy that Mike is considering i2b2.org in case he develops his own. I am big fan of i2b2, but the software itself is for deployment at hospitals and large medical schools and managing of patients, what I think applies at the personal level is the i2b2 dimensional model (specifically the observation table) and the ontology.
Something else to consider is to store the data in such a way that it will be easy to import CCDA records, see also CCDA and i2b2 interoperability, or the newest alternative, FHIR

7

Thanks for all the suggestions and references. As I understand it, Fluxstream is a data aggregator, but NOT a data normalizer. So if you say have sleep data from a fitband and from say a Zeo, they are not related together … they are just separate data and you have to figure out how they relate.

i2b2 and Open mHealth establish a normalized set of data attributes, and in particular in Open mHealth’s case, data extracted from measuring devices and data input manually get normalized into a common set of attribute/value pairs … so said example of sleep data from two different sources would get stored in the database as identical data “side-by-side”. i2b2 has an ontology for its data, while Open mHealth is pretty “flat”.

Then there are all the developments that Apple’s HealthKit is opening up on the iOS side of things, with its own rather flat ontology.

But those draw also from references to several national patient medical ontologies. Plus the ones that patbuen just mentioned. So really, there are TOO MANY ontologies to choose from for normalizing measurements and observations of the human condition.

So, I’ve decided to go simpler. My root data structure will be a completely flat set of normalized attributes and values without any requirement for an ontology. I can draw data from any ontology (aka from any of Open mHealth, i2b2, Fluxstream, Healthkit, Fitbit, Zeo, Jawbone, electronic scales, manual data entry … whatever), and it all gets stored “atomically”. So I can import from anything, and it gets normalized so say “body-weight” from any source (manual or measuring device) is always stored as “body-weight”.

Information that was imported together under some notion of an ontology (for example a “body-weight” and a “body-fat-%” measurement from an advanced scale, or for example systolic and diastolic blood pressure measurements including meta data about which device was used and what posture the person had) is marked as related for only those specific measurements. So my dataset does not demand any pre-established ontology of related or hierarchical or categorized attributes within a measurement. Its just accepts what is given, stores it atomically, and links together only attributes and meta-data that were gathered together. So each measurement can come from a different ontology; that original ontology can be re-created, or a different ontology can be composed from the “atomic data”.

Exporting into a framework like Open mHealth or i2b2 or Fluxstream or HealthKit just means the atomic data will be “bundled” into whatever ontology that framework chooses to impose. And similarly, exporting into a local analytic tool like Excel or SPSS, the end-user can create their own ontology to organize the “atomic data” any way they wish.

In effect, my solution is pretty close to Open mHealth’s approach, as they are pretty “atomic” as well. But they are missing ALOT of attributes since I need nutritional attributes (for food/drink eaten), exercise/activity attributes that are pretty specific (for say run/hike/bike app imports and for exercise tracker app imports), sleep attributes (something I know alot about since I wrote an App for the Zeo Mobile headbands), demographic attributes. Open mHealth is mostly right now focused only on Vitals attributes.

So I’ve created a master root of attributes, and I’m drawing them from Open mHealth, i2b2, bioportal.bioontology.org, HealthKit. Thus I’m ensuring as much as possible that I can pull data into an “atomic normalized” set of attributes, and push data back in whatever ontologies are required.

Open mHealth’s schema’s are in one sense a bit “chubby” (not well suited for an App-sized database) but on the other hand quite narrow in completeness and lack of internationalization support. I’m starting from the outset for the App and all its root definitions to be internationally localized in terms of language, so I’m not using their schema’s directly. But I could create literally 100’s of schema’s for them in an automated manner for the attributes they are missing.

I’ve learned alot in a very short while about the state of the national patient databases and their pretty uniform ontology standards versus the App-level health data ontology standards that are fractured. Thanks for all the citations (and those citations in many cases point to other national-level citations).

Back to writing code.

8

The standards may be uniform, but their application, not so much :slight_smile:

9

I would be very interested! I will tell you I would be interested in Fitbit, iHealth and Apple Health (I have at least one of each :wink: )
I also have a Synology NAS that you could create a package for and run it on that!

10

LOL … exactly my point and frustration … there is no STANDARD … just lots of duplicative standards

11

Well … this may be too daunting for me to handle alone and goes way beyond my personal needs. What is needed to be created are:

  1. A personal-cloud server:
    a. That can run in particular on a low-power small-form computer (for example Raspberry Pi) as well as traditional computers (Windows, Mac, Linux/Unix)
    b. Can use attached hard drive or NAS or both
    c. Provides an over-the-local-network data transfer API to the below end-user’s personal devices and computers.
    d. Provides an over-the internet secure/authenticated data transfer API to the below end-user’s personal devices and computers when they are not-at-home.
    e. Provides an over-the-internet secure exporter tap into cloud-databases (such as say Fitbit’s) where the end-user has an account and the cloud-database has an export API.

  2. An application that runs on Windows, Mac, and Linux/Unix that easily taps the personal-cloud server to export all the stored data into tools like Excel or SPSS or whatever, and likely that acts as a foreground “health dashboard” that emulates something like Apple Health but instead runs on a PC.

  3. An iOS background-App that exports/imports HealthKit data, and exports data from other iOS Apps that have an API but are not choosing themselves to export into HealthKit (and get Apple to approve that App)

  4. An Android background-App that exports from Android Apps that have an API, including exports/imports of Google Fit before data gets sent to the Google-cloud.

  5. Possible iOS foreground-App that acts as a superset surrogate of Apple Health allowing the iOS-based user to see all their data beyond what Apple chose to store in the HealthKit database and to do data entry (and get Apple to approve that App that somewhat competes with Apple Health App).

  6. Possible Android foreground App allowing the Android-based user to see all their data beyond what Google choses to store in the Fit database, and to do data entry.

My personal needs are for #1 (just a Pi without NAS), #2 (just Windows and Excel), #4 (not a wearable), and #6. Although I own several of both iOS and Android devices, I have to pay Apple money annually just to run my own Apps on my own devices for personal use. On Android, I can develop and run my Apps on my devices for personal use for free forever.

Given how cheap some Android Tablets are (example Amazon Fire), I can purchase multiple Fires, and keep ones dedicated in bathroom, kitchen, bedroom, and still access all the data from my PC, from my Android Phone, and from the Fires, all kept in auto-sync via this proposed framework.

12

I stand corrected on this one point. I did not realize that Apple changed their policy last year to allow anyone that writes (or obtains) iOS source code to now compile it and run it on their own personal devices for free instead of paying $99 every year to allow it to keep running. That makes creating iOS apps more viable.

13

It’s been interesting to me to read this thread, especially since I stopped following the data flows topic in detail more than a year ago. (Though I still try to pay general attention.) Even if the “one ring” solution belongs in the realm of fantasy, partial solutions still have value, not only to individuals whose custom needs they support, but also as pieces of kit for others. Our QS Access app, which is just “get your Health Kit data in a table” has a steadily growing number of users. I suspect something like your “personal case” - #1 (just a Pi without NAS), #2 (just Windows and Excel), #4 (not a wearable), and #6 - would also be useful to others whose situation is close to your own.

14

Thanks Gary for your inputs as one of the co-founders of QS. I think my “open-ontology” concept gets me out of the discussions and disagreements about data models and data flows for health data/QS data. And since I’m thinking this is a personal framework, not an aggregate-everyones-data and make money type of framework, each individual person’s or family’s instance will conform automatically to the data models of the Apps they use and the measuring devices they use. I won’t have to force-fit the data models of the Apps and products each person uses into yet another model, and the data recorded by each instance of my framework will automatically have a data model that end-user or family already understands from the Apps and products they use.

Because I live in Califormia and electricity costs TWO-arms and a leg, I do want a very low power personal cloud server. Fortunately, there has been alot of work done on freeware open-source personal cloud servers, and I think I’ve found the right combination that will run on something as small as a Pi to as large as a full-scale high-end server. I’m going to buy a Raspberry Pi today and get it setup as a personal cloud, then start configuring it as a data-sync server.

The data sync App’s themselves are not really that hard to create. I’ve already got starter iOS and Android Apps for code-reuse purposes (having just finished the free open source Zeo Mobile Companion Android App). On the #2 PC side, I’m likely to write that in java since it will run therefore on any of Mac, Windows, or Linux. I’ve already got a portion of my #4 and #6 Android Apps written and storing data.

Of course I can only test interfacing with other Apps and products with ones I own, so the first iteration of this likely will interface to the following:

  • Fitbit Charge (for me just steps & heart rate though I’ll write the interface to pull all the possible web data)
  • Tanita Scale - manual input via Android (though maybe I’ll go buy a wireless one)
  • An iOS-based GPS-based Walk/Run/Bike App that has an API
  • An iOS-based Gym Workout App that has an API
  • An Android-based Diet/Nutrition App that has an API
  • Zeo Mobile Android App (sleep data)
  • ZeoCompanion Android App (sleep data)
  • Meteobridge (weather data and indoor environment data)
  • Windows Excel (hopefully via both CSV and ODBC interfaces)

The actual daunting part is not the code or getting it to work (based upon my ZeoCompanion experience, I think I’ll have my own system for personal use fully completed within 4 to 6 months). The daunting part would be to make it an open-source on-going supported offering, since then everyone would want new interfaces to new devices and Apps. I’m not a company that wants to charge and make money, so I cannot go out and purchase one of everything to figure out how to interface it.

Now I have to figure out a name for this that does not cross over anyone’s trademarks. App names like myHealth, myOwnHealth, myPersonalHealth … all taken LOL.

  • Mike
1 Like
15

That is the hard part. The rest should be trivial :slight_smile:

1 Like
16

LOL … I did find a relevant name to use. So now onto the trivial part. I have seen alot of success in a very short time. I’ve already got a Raspberry PI Cloud Server up and running. I’ve already got an Android App up and running that collects and stores selected QS data. I’m right now working on the sync-to-personal-cloud-server part … nearly done. Then my next step is to write a HealthKit iOS sync-to-personal-cloud-server part.

Then I’ll have a working initial prototype implementation that is merging my weight data, nutrition data, and walk/run/bike data from both Android and iOS devices onto the Pi, and being pushed on demand from the Pi to Excel. Not a finished product, but a working proof of concept.

1 Like
17

Mike, I have begun recording this data as well. The system I currently have setup uses:

iOS app with Links to HealthKit, and records data into a PouchDB Database
The PouchDB automatically syncs to my web server, that is running a CouchDB

You can see an example of the at https://health.schappet.com - if you want to log in I can give you a sample user.

18

ScreenShot
https://www.schappet.com/wp-content/uploads/2017/05/Screen-Shot-2017-05-11-at-10.53.20-AM.png

19

Interesting thread! I’ve considered the same, not finding any platform that can aggregate and let me analyze my data as I want. Fluxtream is the closest one I’ve seen, but I wish it was easier to create new visualizations.

I ended up writing importers for my data (from fitbit, garmin, google fit, etc…) into Elasticsearch. So far, I’m just making standard visualizations in Kibana, but would like to do more datamining of it (comparing days, finding patterns over time for example between steps and head ache…).

I have been thinking of either trying to get into the fluxtream project and adding what I need, or just continuing making my own, or perhaps even making a kickstarter of it (if that’s even plausible since I’d like to keep it free and open sourced). But I know we are more wanting the same thing, but with slight variations…

Regarding aggregation, I know that Google Fit has a lot of aggregated streams of different types, and an open api, but you can’t really install it at home =)

1 Like
20

What name did you choose? Did you make a repository on gitlab?