FYI … I wrote a LUA dissector for Wireshark for the Zeo Headband Bluetooth protocol. So now its quite easy to record and replay the headband and Android device interactions.
Just use the Android Developer Mode Bluetooth HCI snooping feature on the device, then pull the log file off the device and into a computer, then open the file with Wireshark. Voila, all the packet flows are dissected. One does have to force a “Decode as” for the RFCOMM.
I can’t post the LUA file here yet since I’m too new.
Hey Mike,
I’m always interested in furthering my GitHub skills… there is so much I don’t know about collaboration tools.
I’ve a long DOS, Windows background… and a really long Unix, Linux background… Apps are still kind of new to me (they all look like Java with pretty window dressing to me). So I have this awful tendency to prefer simple cmdline apps, or services that enable things in other scripts or languages.
Doing a [whole] App top to bottom just either seems so “one off” or just so fragile, they never seem to meet everyone’s needs.
To your point about an original Zeo app that is truly free, based only on the ‘discovered’ APIs… it is interesting. There should be an alternative for Android and iOS… I am just not sure what that should look like.
I would like to look at your Lua dissector and get it up and running.
Two projects on my plate right now are laying a trail of breadcrumbs for other people to follow… I tend to loose interest all to often… and think its the ‘right thing to do’ to make it easier for people who want to follow up later.
So what I want to do are these things
document how to at least compile an rfcomm-server program using Visual C++ Express to receive the Zeo chatter - this wasn’t too hard because I found a demo app that interfaced with the native Microsoft bluetooth stack in the back of a book. So really it was just understanding and grabbing the unique UUID the MyZeo app writers advertised … and that was easy because Linux has a suite of cmdline tools to “search” rfcomm for services available within range.
it really bothered me that people didn’t see how easy it was to look at the code of an Android app that doesn’t even use the obfuscation natively available… I was thinking of decompiling the native ZeoCSV app as an example… it might get people to use Eclipse or Android Studio more often both for learning and for compiling their own apps
Tasker plugin… this is as close to a cmdline tool as I can see on Android, its widely available and scriptable for most people… a simple rfcomm-server that performed taskers bidding would be really cool and extensible
Your driving ahead really a lot faster than I am… so please take the lead. I’ll plod along at my slower pace and help if I can.
I was granted rights to post files now. I’ve attached the latest Zeo Headband Bluetooth Protocol document, and the Wireshark LUA dissector for the protocol. The protocol document has been augmented with a sample interaction flow between the headband and the android device for an example sleep session.
To use the dissector, you need to put the LUA script (on Windows) into “C:\Users<user>\AppData\Roaming\Wireshark\plugins”. After opening a btsnoop_hci.log file in Wireshark, one has to force a “decode as” L2CAP CID->BT RFCOMM. Then the Zeo dissecting should auto-active within the RFCOMM layer.
The dissector mostly works, but is a bit buggy … first one I ever wrote. And it has to perform its own packet re-assembly which I’ve kludged in a knowingly poor manner as I’ve not full worked out how to get the dissector to access information from prior layers (in order to know whether its a message-sent or a message-received since they are interlaced). I’ll fix that once I’ve looked over some more examples.
Also, to get all the expanded field elements in a multi-packet message (such as the SLEEP_MESSAGE), you have to click on just each packet of the overall message in-order top-down until the final one, then all the sub-fields will show in that final one. I’m not sure why Wireshark does not “remember” that since its doing it on its initial pass as seen in the “info” column. Another bug.
Zeo Headband Bluetooth Raw Data Format.txt (17.8 KB)
zeohbbt.lua (11.3 KB)
Awesome Mike
I’ll try this out tomorrow night. Its most excellent to have a reason to better understand how Wireshark works.
Lau makes me recall the Phillips TiVo box I once hacked… so cool
… Hey it might be an awful timezone conflict… but I’m going to attempt a Google Hangout on Saturday… just posted the offer over on grc. I invited the World… but seriously… how many could possibly truly be interested? Lol
It is easy to unpack an APK and recreate the original resources and source code. I’ve already done that, although the resultant code is quite incorrect and needs interpretation, which I’m good at. However, that code’s copyright is still owned by a legal entity, so the code cannot be used as a basis for creating some other publicly released program.
One could create a new base Zeo app for both Android and iOS, but it has to be created from scratch without utilizing any existing code. For a population of only ~2600 users, its a lot of work, especially when everyone that is using a ZeoMobile by-definition must already have an Android device and is already using the base Android Zeo app.
Its more efficient and less work to create a companion app that augments the Zeo App, rather than do all the work first to re-create the base app then all the work to augment it.
I don’t disagree.
I think the work you have done already goes a long ways to making that possible. But I don’t see Bluetooth as limited to the Android platform. Once the protocol is worked out (which I think you have done) it can be used between a Windows, Linux, OSX or Android and iOS device.
I don’t pretend to think I can write all of those programs all at once… but I find each has its charms… and its certainly going to be a lot of fun. If other people code up something faster… great… plenty of other platforms to look at and circle back to learn from them too.
It would be really cool if that company which bought the Intellectual Property appeared and granted license to legally look at the old app… and I take it for granted its not that interesting from what you say. But if they did I would still look at it. My guess is they have no interest in making the effort to do that…
Working on the Zeo Mobile seems like an Evergreen project… so many people have come before and had a go at it. I really liked the story about Brian who got them into opensource after hacking the bedside model.
The Terms do explicitly say reversing the apps is forbidden… so that’s the end of that. But the Bluetooth protocol is still open for interpretation.
US Patent regarding Sleep Monitoring
Systems and methods for sleep monitoring
Describes processing of the raw EEG data into sleep states
US Patent for the Zeo bedside
Data-driven sleep coaching system
Contains the equation for determining the ZQ value from acquired data.
US Patent for the Zeo Mobile
Multi-modal sleep system
A Zeo Developer describing Signal processing - Stephen Fabregas
Behind the Headband (1): How Zeo Works
Last Archive.org of the approved Developer API’s (including Android) and Terms
Zeo Developer Interfaces Welcome
An interview from 2013
Ben Rubin former CEO what came after Zeo
Libraries and Binaries
QS links to Zeo bedside final archives
Ha… I finally got my head around what you and Steve Gibson had been talking about.
Its so opaque when you don’t get what people “who know what they are talking about” are ‘talking about’… its also hard to remember what its like “not recalling how it was when you didn’t know something” when your explaining it… a communications mismatch.
[On the Android phone, including the latest Android 6.0 ‘Marshmallow’]
[Settings… gear icon ] [at the very bottom of the Settings list… is…]
[ {} Developer options ] <= tap to go inside this menu [third item down… is…]
[Enable Bluetooth HCI snoop log ] <= toggle this on by tapping the icon
It says “Capture all bluetooth HCI packets in a file”
This webpage Wireshark: Bluetooth says HCI_H4 is an encapsulation format that Wireshark understands. But that the early version of the pcap format that Wireshark reads did not support direction information (sent : received) Direction info missing for Bluetooth H4 captures in pcap-format the bug was filed in 2007 and fixed by 2008.
Since Android 2.2 Froyo was released in 2010 that bug predates current versions of Android by quite a bit. It would suggest that perhaps later versions of Android did include the fix, and that the information is there… maybe your saying the current dissector “direction” bug is in the current implementation of the overall dissector in Wireshark, or the recent Zeo protocol wireshark plugin you just created. “I am not quite sure which you meant… or if you have enough information to know where the bug is for sure at this point.”
I took a more pedantic direction and put together a simple Rfcomm service and screen dump… skipping all the protocol layers up to the raw data. “But” that obviously means I did not have a tool to cleanly and neatly dissect and interpret the messages and produce nice neat back and forth streams.
This approach is indeed “Faster”.
There is another generic Windows binary stream parsing tool I was thinking of using called “Hexinator”… but not as well known… and there are others… but I like this one better… Wireshark is more universally available on many platforms.
From this site 32 things you can do in Android Developer Options I see the Android device Bluetooth log can be found in /sdcard/btsnoop_hci.log
That now matches up with what you said,
"Just use the Android Developer Mode Bluetooth HCI snoop feature on the device, then pull the log file off the device… "
Not sure about
" … force a “Decode as” for the RFCOMM…
But I’ll figure it out
" … put the LUA script (on Windows) into “C:\Users\jwillis\AppData\Roaming\Wireshark\plugins”
Seemed to be correct for me, interpreting the path for my username
I got an error message loading the LUA however while starting
It could be the exact version of Wireshark I am running… currently
[Version 1.8.6 SVN Rev 48142 from trunk-1.8] needs to be updated.
I’m away from my test environment… so while I have my Android device with me… I don’t have the Zeo with me. I should have a lot more time to capture Bluetooth HCI packets this evening.
I’ve been using a lot of these tools for years… but never deep dived into Bluetooth or used Android debug or Wireshark in this particular manner… so my ‘learning curve’ could be entirely slow and frustrating for those who do this a lot more. It would be entirely understandable.
But nothing tried, nothing gained… and this is really interesting
Okay… got some data… but… Now I’m, somewhat confused
The [Sent : Received] packets are actually being interpreted correctly.
I have not updated my Wireshark… so I am still getting the initial LUA error on loading the dissector plugin, but perhaps I am using a newer version of Android for the bluetooth packet capture… and it has the HCI “fix” ? And the problem was never Wireshark… it was waiting for a pcap that had the direction info?
I’m wondering what version of Android your pcap captures were made with… that could be valuable information for setting a lower bounds on what works or doesn’t.
I can try Android 4.4.4 and perhaps 2.2 later tonight to see if I can reproduce the bug.
[Also]
I was looking at the API documentation for the opensource Raw Protocol that Zeo, Inc. released, I see some possible commonalities with what your specification indicates. This maybe over Bluetooth… but the data and its format could be handled similarly… if I recall the Bedside alarm clock/dock used the [same] instrument pod for the headband. So it would make sense that the communications protocols might be similar.
The Android app may be untouchable… but if they did document the protocol and granted developers permission to use that as part of the opensource documentation already out there. …that would be good.
The Bedside unit then would only differ in that it had a serial port interface… but the Bluetooth communications [might] contain the Raw data… even though we were not looking for it… Could be the only reason we don’t see the Raw data is we assumed what the Android app yields was all that there would be. possibly an incorrect assumption.
It would be nice if we had a Bedside units Bluetooth capture to look through. I don’t have one and I hear they are kind of expensive on eBay. And we don’t have Android debug on a Bedside unit… just spinning ideas here.
Do you use IRC or Slack or some type of communication tool other than forum posts?
I was thinking of bringing this up in the Hangout tomorrow… dev stuff is probably not that interesting to the general public and a lot can get done/overwritten a lot faster. If you prefer not, and rather use GitHub thats understandable… some people like to work with less distractions… I kind of like to talk things out.
You need to update Wireshark, Your old version didn’t even auto-dissect the L2CAP layer, and that is why you are getting the “no dissector table” error. I use Wireshark in “legacy” mode personally as I find it less confusing.
I’ve updated the dissector fixing numerous bugs, and repairing the kludgy packet reassembly with better examples from other well-experienced Wireshark dissector developers.
zeohbbt.lua (15.4 KB)
My captures are from an Android 4.4 device.
The ZeoMobile headband is DIFFERENT than the ZeoAlarm headband. The ZeoMobile headband does NOT send out the waveform information or other useful information unless we can figure out how to put the ZeoMobile headband into Test mode. I’ve run a capture for 24 hours looking at all the comms between the headband and the Android device. The ZeoMobile headband does all the signal post-processing inside the headband itself.
The Android Zeo App is just a “shell” that pulls the already post-processed data over Bluetooth and stores it into the database. The App has no post-processing logic, which I’ve confirmed by decompiling the APK, and looked at the App’s code.
Hey, thanks for the Tips!
I did update my Wireshark and finally got the previous version of your LUA dissector to run… but your warnings are right, while miraculous… it was a bear to get to work right.
I will be most happy to switch to the new version.
I did go through your script… but even with a folding editor it was dense for me… both familar and not so familar… LUA looks a lot like C.
I’ll have to find this Legacy mode your speaking of… nice tip.
Good and bad to hear about the Mobile vs Bedside… Good since I can’t verify it either way… Bad since we don’t know yet how to run those test modes. Not so bad for people who just want the post processed information.
–
I found the legacy mode… but it made [Decode As…] really unusable for me… then it crashed… so I just went back to using the new LUA dissector with the new Wireshark interface… its harder to read… but at least I know how to make it [Decode As…] and there is still much I don’t know about that feature… for instance I have to set the BT RFCOMM value to 4 or it won’t process the packets into Headband protocol messages… the default is “0” which never works.
The RFCOMM DCLI=4 thing was based upon my traffic … the 4 means DLCI channel 4, and obviously on your captures that channel is 0. The Wireshark LUA interface requires that I set some value; I didn’t know if that value changed or not (and now I know it does).
I find I can set multiple values for that within the dissector definition. So I’ve updated the LUA. Give it a try.
zeohbbt.lua (15.4 KB)
That is really much improved Mike!
I didn’t have to [Decode As…] anymore at all… it just parsed the packets right away.
I think I’ll spend some more time learning Wireshark, I saw someone with a lot more experience parse a bunch of packets into “streams” and “follow” conversations.
I seem to recall a version of Wireshark that could even capture and replay packets back… if that could be made to work with an Rfcomm session… it would be nice.
Quick note… [I think] discovered how to activate the [debug logging] mode for the Zeo… if it wasn’t already doing that already. It could have been I just didn’t know how to download the logs.
Regardless…
Its been logging now and the entries have ‘tags’ of [V…D…I…W…E] ect… with timestamps.
Could be useful or useless… I don’t know, but it happened while trying to do something entirely different.
Posted the details to the blog
Combining the debug data with the protocol observations might help understand some things.
You didn’t mention in your blog post HOW you activated the Zeo App’s Verbose-level logging mode … mine appears to only be in Info-level mode. You can also view these logs in real-time from Eclipse or Android Studio rather than emailing them to yourself…
FYI you can generate those logs right from the Developer Options menu … its right at the top. Using the power button is convenient for a “popup” in the middle of an application when you don’t want to send the app you are working on into the background.
My logs for some reason are so large, the stupid Email client won’t send the attachments.
Updated:
Oh … looking at the Zeo App’s code and your blog post, I can guess you re-signed the Zeo App after likely altering some of its permissions. The Zeo App auto-turns off its “verbose” log messages if its signature matches a hardcoded numeric string. Altering the signature will have the side effect of putting the Zeo App into “verbose” log message mode.
Yes… I deleted the META-INF in the original APK without extracting or touching the other compiled components. The signature is by its nature exposed. Then I made a new keystore and used that with the java jarsigner to re-sign it. I didn’t even use eclipse.
Interesting what you said about Info vs Verbose modes… I didn’t know it would do that… I was just bumbling around in the dark.
I did get a slight clue though since in the back and forth on that Android developer forum in 2012 someone mentioned developer builds versus release builds and toggling logging.
I just did not know what the trigger was. When I got the first Bug report log dumps… I knew something was happening that I didn’t expect.
I see the Top of the Developer menu [now] that you mention it… I didn’t think it was click-able I thought it was just a menu Title bar… but your right it does work.
I Really like the idea of viewing them ‘real time’ as opposed to emailing them… I was hoping there would be a way to do that. I assume it has something to do with the ADB… which I know very little about.
Long term figuring out the Bluetooth protocol will probably be the best thing… since it opens up so many possibilities. Accessing the database was just a [thought experiment] that has gone a bit too far… the debug logging was a really nice side benefit (though now I worry about shutting it off ) 
I think I will have to take a break tomorrow… its a work day… and there is a problem with Outlook and autodiscover misbehaving… someone set all of the users primary smtp addresses away from the auth domain… and they look untethered from autodiscover… oy… I got hopes I can use a .prf file to reconnect them… or some wicked scripting to adjust the profile configs in their registry… I already have a headache just thinking about “that”
Although the Zeo App’s debug messages have some interest, everything from the App will still be limited to the post-processed data that the Headband sends out.
I got my ZeoCompanion app for Android up and running in an “alpha” release mode. I’m able to directly interact with the headband, receiving data and sending commands. I had to incorporate the same packet re-assembly logic as was needed in the Wireshark dissector.
The next step I’m going to take is to see if I can discover how to put the Headband into Test mode, and therefore gain access to the pre-processed data.
Hi Mike,
Sorry about the slow reply.
While, I’m interested, I’m not sure that I have the chops or time to contribute meaningfully right now.
This is why:
All that said, I’m more than happy to help test, an AM interested in learning some Java / mobile app development and would be interesting in tinkering once I figure out how life works. It would just be a while before I was more of a help than a hindrance.
-Alex
That was really fast… congratulations!
I wish you all the luck with the test modes. I haven’t seen anything about them anywhere, but I will be keenly interested to hear if you can activate them.
The two quotes I read about the bedside versus the mobile model seemed to indicate the beside model ran all the time broadcasting its raw measurements and ate up a lot of power.
Nothing concrete to offer… but I would [guess] they would need raw sampling data to fine tune the software code computing the FFT.
I have seen on a different project board an FFT function approximated by a neural net using some sort of fine tuned mathematical matrix transform equation in code. That is they used a standard maths library for the matrix calculations… but they had to provide the coefficients using Matlab or something… if they had the raw data, and could keep working it back and forth they could increase the accuracy… of filtering to filter out unimportant things to deduce the sleep states. I don’t think that helps much.
But it would be a good reason for the raw data to be made available in a test mode.
The big win for transmitting only processed results would be savings in processing power… and not having to write reduction code for every platform.
Just for reference: Zeo IP was bought by Resmed, and I’ve pinged my one contact there (again) - but my last requests were over a year ago and never got a response.
I’m not sure if I’m going to write a full app or not … depends on whether or not there is more to pull from the Zeo than just its present dataset. But if I do, of course I’ll be looking for some beta testers at least.
