Tracking Focus and Task Engagement on a Computer with Purely HCI Metrics

Hello! I’m new here, not a researcher or academic by any means, just a programmer who falls in this niche (which is relevant to this post actually).

This idea was inspired by google reCatchpa; which seeks to detect if a user is a human/bot not through a catchpa string but by tracking all of the ways a user interacts with the page. For instance if the catchpa is clicked 0.2 seconds after page load, or the mouse moves in a perfectly straight line with 0 jitters to click the box, it is probably a bot doing it.

This got me thinking about how the digital metrics when interacting with a computer (mouse position / movement, opening/closing tabs, switching windows/applications) are different when I am focused on a task properly versus when I’m getting distracted, procrastinating, or engaging in compulsive behavior’s like tab hoarding or opening social media.

When I started paying attention the qualitative/quantitative differences between (focused) mindful and mindless computer usage are quite stark; such that I think you could take all this data in the form of logs mostly and use it to make a digital ‘mindfulness assistant’ app. (And I don’t think it’d be that hard actually).

A simple example for how this could work is say you select in the app that you are doing ‘a writing task
about abc topic in xyz program’

Right off the bat the app now knows, and knows to watch for:

  1. keystrokes / typing
  2. when xyz program is open or closed, and how it is interacted with (like if typing is being done with xyz selected or not)
  3. that you are engaging with topic abc, using AI the app can check whether a tab, resource, website etc is relevant to topic abc when opened, and if not restrict it or remind the user

Then using audio/visual indicators; the app can call the user back to focus on the task at hand if they get sidetracked into something else.

What’s cool to me about this (and makes me think it could have legs) is that it’s a pure software / pure vanilla PC solution that doesn’t need so much as a webcam (but a mic and webcam would certainly help). Combined WITH eye tracking it would be even more potent.

With user feedback to let the app know when they are/were focused or distracted etc you could even fine tune some simple machine learning models on that users personal data, which should get you to ‘seemingly reading your mind’ levels of effectiveness.

I’d love to get some input on this and hear your thoughts, and if anyone is interested I’d be happy to talk about collaborating!

1 Like

You could go further and see if it reflects general health, future disease such as a migraine or IBS attack, dementia, diabetes, high blood pressure, stress. You have to model the person and have a big dataset of different people. The ML possibilities are endless,

1 Like

Great ideas. I am working with a BCI startup called Blueberry. You can use the device to track focus / eye tracking, for your particular use case. Would love to collab.

If interested in learning more about the device check it out here. [https://blueberryx.com/]

Good idea- I’d probably be best to have the app run locally and be open-source. People (me!) get nervous about a program logging all inputs!

Make sure to underestimate how well the app would know if searches are off track—take writers for example. Almost anything is searched and accidental findings are the joy in story. More useful for a writer is if the writer wants to opt in to a words on the page session or a research session or both.

For health it would be unique to track productivity for task fatigue, computer fatigue. More than a reminder to get up to stretch or move, but it could do that as well. I think of individuals with brain injuries or COVID brain changes who hit cognitive or task or fatigue walls. Help them catch their degrading productivity so they can rest.

Very good points all around.

On the task fatigue / break reminder front (let’s call that high level focus or a use of executive function), I’d be afraid of making an overcomplicated pomodoro timer.

I’m really far more interested in the very low level atomic moments of focus and distraction, I myself have quite severe ADD and some lingering neuro symptoms from an auto-immune disorder that really limit my working memory and attention span for deep work without a lot of structure and effort on my end to grease the wheels. A system to alert me when it detects I’ve drifted off into space in the middle of a task (mid sentence even) is the sort of thing I had in mind.

But if it can accomplish that robustly then a wider loop around work sessions should be easy in comparison.

Ideally such a system could also have an ‘observe’ mode in addition to working with human feedback, once it has a lock on your habits and quirks it should be able to discern all sorts of patterns and such.

Yo that looks awesome! I’ll hit you up later today or this week to talk.

1 Like

Judging from I’ve seen the TikTok recommendation algorithm do (literally diagnosing people with rare diseases before their doctors, or even before they were properly sick, and many other interesting examples) that’s a very good point I hadn’t considered.

Yes :sweat_smile: security and privacy are the big thing. It would pretty much be a high tech keylogger on steroids. Probably by Spring or Summer of next year though I see current ML/AI/LLM models being miniaturized enough to support fully local (self hosted would also be a good option). Analyzing content for semantics will require an LLM which for now is mostly an offsite thing, everything else should be able to stay local. I wonder if some kind of VM / private silo instance could work.
Firmly agreed on open source!

wonderful

1 Like

Yo! I don’t think the QS forum has a private messaging feature implemented so just replying here. My email is grantpswal@gmail.com and my GitHub profile is /uly-s , shoot me a message so we can talk some more!