Any recommendation for "Centralized Cloud Service and Mobile App (iOS)" for QS

I am guessing that this question is probably a repeat… I could not find a FAQ and the “topic is similar to” did not help as well… so I am posting…

I am looking for a place in the cloud to store and a mobile iOS app… I use a fitbit and have an iPhone. And any “Genomic” relations to QS?

Thanks and sorry for the duplicate question…but I am hoping to stand on the shoulders of the giants in QS :slight_smile:

Hi Krishnan,

Will you describe a little bit more what you are trying to do?

Gary

Gary, thanks for your response. Its an experiment… I am trying to basically have one central place cloud where I can store ALL of my history. With the hope that I can create a machine learning model to predict when I fall sick :slight_smile: or when I am happy/sad and see if there are any patterns using some prebuilt machine learning models…

  1. Milestone Dates
    a. Personal Milestones - Graduation, Wedding, Jobs, Kids etc.
    b. Health Milestones - Accidents, Hospital admission (Dates) and Events etc.
    c.
  2. Health Data
    a. Weight
    b. BP
    c, Height
    d. Pulse
    e. Sleep
    f. Steps
    g. Exercises
  3. Nutrition / Food Data
    a. Instagram pictures of food eaten
  4. And others?
    a. Photos?
    b. videos?

Just trying to restitch what happened and see if there is a pattern in there…

This is the FINAL goal, I am guessing it will take a few steps to get there! So I want to find out what others have done to “centralize” their data store… I can buy some cloud store to keep it… and keep it encrypted!

Regards

Krishnan

Dear Krishnan, this is also one of my final goals. I have my (two years worth of) data stored in one dataset but aggregated to a daily level. I have Milestone data (stored in text) Healthdata, some food data, and mood data in my dataset. I store my dataset offline. But, I am not able (yet) to apply machine learning on my data, because I lack the ability to program this. However, this is something I really want and I try to find people who are interested in this.

Looks like nobody has any ideas or suggestions for a noobie! :cry:

Hi Krishnan,

Your question is difficult to answer; I’d be glad to give some guidance but it’s a bit hard. If you just wanted a cloud storage solution for files, well obviously that’s trivial. But if you want a database in the cloud that will accept data from an iOS app that allows you enter all your personal data: that’s something that does not exist in anything like the abstract perfection suggested by your query. It’s an easy request to make, and a rational one on its surface, but if you start to look a little more closely at the nature of the data you are asking to store you will recognize that it is vastly heterogenous: some of it is about a number, some of it is about an interval, some of it is about a score across an interval, some of it is scalar, etc etc. The most intensive effort to provide something like this is Apple’s HealthKit, and if I were you I would start there. By using it for a while and thinking about what is missing, you’ll get further along the road you are defining. If you want access to all your HealthKit data you can export it as an .xml file. If you want to look at it in a table, you can use the QS Access app.

Does this help?

Gary

Hi,
MyIndicators is a cloud platform for QS,
Where you start out from premade indicators and personalise data collections for your need including scorings lists etc.
See more…

http://myindicators.net/

/Patrik

@Gary - yes it does help. I do appreciate the complexity of the data variation problem that this creates. We live in the times of open source, cloud providers and powerful mobile devices. The QS movement has been around for a long time. This lead to my assumption of having something like this. Data correlation across the various datasets can give us answers to questions we do not even know to ask!

I like that you suggested HealthKit which is backed by a serious company with resources to follow through on any long term technology aspirations. I have used the QS Access App - thanks to whoever wrote it! It was beneficial in getting my multi year data recovered from my phone before I reset it. As you already know the flip side of a large company is the challenge of them holding on to the data for use thats not compatible with our need to store that data. Thats something we cant do much about these days - so I just accept it!

The other problem i am running into with HealthKit is the fact that two publicly traded companies (FitBit) and (Apple) have competitive considerations that squashes a Quantified-Selfer (is that a word :slight_smile:).

Do you have a “quick and dirty” map of what HealthKit webservices/apps that are out there I could quick-start to?

Thanks and appreciate the thoughtful answer.

Hi Patrik - thanks for your response. I checked it out - and seems like a good “general purpose app” - I even installed it but promptly I removed it. It did not build upon the data set I already have (or did that easily). But I appreciated knowing about this. Its a very good “customizable” application to capture information on your mobile phones. For sure! All the best.

Hi @Krishnan Krishnan (and @justintimmer as this applies to your post as well)

How about combining Microsoft’s ML Studio, Azure SQL and Excel w/ Get & Transform plugin - let me explain.

I don’t think you have to be overly technical to create a workable solution using these tools either. The first step is whether you are comfortable storing the data you referred to in a cloud-based Microsoft SQL Server database. Azure SQL supports the necessary controls to help you secure your data in this regard. If yes, then I recommend you open a MIcrosoft Azure cloud subscription and start a Azure SQL Server subscription. It is not that expensive (I pay about $3.00 per month to store 20MB of data using this service.)

Once your data is in Azure SQL, i recommend you create a subscription to Microsoft’s cloud-based Machine Learning Studio (i.e. ML Studio). ML Studio is a visual Machine Learning algorithm designer. You don’t have to be an expert mathematician, statistician o programmer to use it either. Also, with your data already in Azure SQL, you can easily access this data from ML Studio as part of your designing a new Machine Learning Algorithm. Once you have created your algorithm, ML Studio automatically creates a Web Service API for your ML algorithm and creates an Excel file that you can download in order to put this algorithm to use. For example, once you create your machine learning algorithm (I assume you want some linear regression algorithm that assigns a probability score on whether you will be sick or not), you can enter a new row of data from the Excel file and press a “Predict” button to get a probability score. Regarding costs, I host a few ML algorithms in ML Studio using the basic plan and the cost is free.

Google and Amazon offer similar capabilities. As you can expect, tools from each provider are tightly integrated with services from each provider. This means that hosting your database in the Amazon cloud but using MIcrosoft’s ML studio to get to this data is harder then keeping data + algorithm with Microsoft, or vice versa.

Microsoft’s strategy has always been to empower the ordinary user - so, moreso than cloud-services from amazon and Google with all the Azure cloud services, you don’t have to be a programming whiz to use them.

I hope this helps!
Sergio

4 Likes

Sergio, this is a very interesting answer. Thank you for writing it. I would love to read about an example from your own projects, if you have time to extend this.

1 Like

Hi Gary,

The diagram below illustrates some of the tech i use in my QS projects, including examples of how it is being used .

I would preface by saying that I face a very similar problem to that mentioned by @Krishnan and @justintimmer - multiple years worth of tracking data (and growing!) and the need for an efficient way to store and manage it, as well as innovative ways to tap into the value of these growing datasets.

Those with backgrounds in traditional Business Intelligence functions will recognize the data platform depicted here as a very classic distributed data pipeline feeding a data warehouse. Data comes in from various sources on the left hand side, is cleansed and transformed before being loaded into a traditional data warehouse. On the right hand side, you see the visualization and analytics reporting options.

Here are a few examples of how I utilize my data:

Visualizing Growth - We have three kids so I can easily generate time-series of their physical growth directly from my iPhone using Microsoft’s PowerBI app. These are useful during pediatric visits and help build on the conversation between parent and pediatrician. Most of the measures i’m tracking are recorded manually except for weight which is loaded automatically from a Withings WIFI scale.

Reporting on Vaccinations - I track all medicines and their dosages taken by my family, including vaccines. From my phone I can also quickly pull up immunization records for the whole family compared to the vaccination schedule we are following, or any allergic reaction we’ve had to a medication over time.

Anticipating Symptoms - I also track all the health symptoms we experience over the course of the year. Symptoms are normalized to the WHO’s ICD 10 classification. It was only after collecting a few years of symptoms data that I could begin to detect patterns in how our body responds to changes in weather, season, and a host of other factors. This gives us clarity in planning for vacations, for example. In general it gives us a pulse to understand how factors may be impacting any symptoms we feel.

Tracking Debits/Credits - Here is an example where a Machine Learning algorithm is being used successfully. I track daily credit/debit transactions. Before importing them to the data warehouse you see in the diagram, they are categorized according to a list I defined (i.e. Food, Groceries, Shopping). I wanted a simple way to look at the transaction’s description and assign a category. I also wanted a way to learn from previously categorized transactions, so that the process become more accurate over time. For this I designed a Text Classification algorithm using Microsoft’s ML Studio.

My background is IT, so i apologize if this comes across as very tech-oriented perspective. The reality is that I am much more interested in making self-tracking an effective tool for the whole family, regardless of the technology behind it.

Having said that, today more than ever, the technology options in front of us are really amazing. The cloud services alone, including data and machine learning services, such as the one I mentioned earlier, are helping us all “cash in” on the data we keep. In the diagram you see, I pay a total of about USD 18 per month to Microsoft for the following:

  • database in the cloud (Azure SQL Server)
  • machine learning in the cloud (Azure ML Studio)
  • data factory in the cloud (Azure Data Factory)
  • analytics and visualization in the cloud (Microsoft PowerBI)

The items in dashed lines don’t exist yet, but I would like to add them. These items include the sci-fi sounding chatbots or natural voice processing tasks. There is a practical reason for them however and it relates to Google, Amazon and Microsoft beginning to introduce “home” products like Alexa. These products listen and talk to you about your music, the weather and your groceries. How great would it be if these products could also answer in a context that is based on your self-tracking data!

Sergio

2 Likes

@Sergio I am super impressed at what you have been able to accomplish! How long did it take for you to build this out? I have all the moving parts - but they are in disparate systems in disparate formats. Are you able to share (maybe privately) how you arrived at something like this and more importantly how long did it take for you to do this.

I am curious why you used a traditional datastore instead of a keyvalue pair datastores like Cassandra or Hadoop since I am thinking the data that we start to store now will start exploding.

I am looking to create the kind of notifications that will proactively tell us about potential land mines and the chatbots that we could ask questions to.

I had a herniated back in 95 and since then every one or two years I get a lower back muscle spasm that prevents me from moving period. I cant even get out of bed without excessive pain. And I’d love to be able to predict what is causing this problem - weather, or a recent trip to the ski resort or blah blah… even if I get a 50% accuracy in prediction its better than what I have now.

I am from a technology product background as well, so happy to collaborate. If that’s of interest let me know. Though presently I do not have the beautiful model you have already created! I’d have to get there first :).

Super impressed!

Krishnan

Thanks @Krishnan. Regarding your questions, I can follow up with you privately but just quickly - this work is constantly evolving. Consider it started as a simple spreadsheet three years ago, whereas now it looks more like what you see in that diagram. Thanks to the cloud services such as those from Amazon or Microsoft, much of the heavy lifting is already done, and so you can arrive at something like this pretty quickly.

Disparate systems and disparate formats - you also mention the need for a model and that is the first thing to address to make this work. You can structure the model ahead of time and then spend time cleansing and transforming your data to fit he model. This is basically the approach I followed, it works but it is also the old way of doing things. Alternatively you can use the Big Data / Hadoop tools you mentioned, store all your data, regardless of how it is structured (or not), across a series of machines and then use the breadth of tools in that area to query the data as if it conformed to a logical model. I didn’t follow this approach because i am not yet too familiar with those tools and I also thing that the amount of data i’m managing, albeit growing, is peanuts compared to what those tools are really built for. Those are Big Data tools, yet i’m mostly dealing with small data.

Sergio

1 Like

Sergio,

Thanks for sharing this info. Very helpful, as I’m a newbie trying to figure out how to collect, store, analyze data.

I particularly like the diagram, and am thinking that it would be super cool to build a more elaborate picture of possible inputs (on the left), transform/storage (the middle), and possible outputs or outcomes (on the right).

Better yet, I wish I could find the analytics first, then work backwards to collect data in a way that feeds the analytics. This is because I want to get to the value without having to hand-build the mechanism (I have limited technical skills, but can learn if I know I’m building towards the right analysis).

Again, thanks for sharing - great inspiration!