In this breakout session, we will be discussing how to use computer vision to extract metadata from lifelogging photos, enrich a photo timeline with other personal data, and draw insights from massive longitudinal photo collections. We hope you can join us for what we hope will be a fascinating discussion.
Niclas Johansson (Narrative)
Rami Albatal (Dublin City University)
Cathal Gurrin (Dublin City University & University of Tsukuba )
Very much looking forward to this. Hope it doesn’t overlap with my break out session.
Thank you to those that came to the breakout session! It was a lively and excellent session with plenty of audience interaction. There were about fifteen participants who had an interest in photo lifelogging.
The session started with a presentation by the session chairs. The Narrative representatives discussed the Narrative clip and their plans for supporting photo lifelogging. This was followed by the DCU team giving an overview of what is possible with photo lifelogging, covering the technical possibilities of what is reasonable to achieve today.
What came across from these presentations is that photo lifelogging is not difficult, but the computer analytics to mine and extract meaning and knowledge is certainly challenging and even state-of-the-art computer vision analytics techniques can often fail to identify the valuable content of photos.
There were a number of core points of discussion, and these were:
Food and Diet. It seemed to the panel and the audience that food and diet monitoring was a key requirement for photo lifelogging and should be the key challenge to be addressed. It was accepted that this is challenging to do, but it was pointed out that recent academic findings suggest that indeed this is possible to achieve in some circumstances. It was pointed out that the most promising technologies to achieve this required a significant investment in time to label food eating photos and there were a number of willing volunteers to help with this activity. If it is possible to release a dataset of food eating photos, then the QS community will be able to help to label the data and build a large amount of training material for machine learning and AI techniques to utilise to build better food and diet monitoring tools. The organisers have taken this point on-board and will return at the next global meet-up with a plan.
Behaviour / Lifestyle. Analysing the behaviour of the individual was discussed in terms of data correlations over time and visual day logs. Visual day logs, being the easiest to achieve today is available from the current generation of lifelogging tools, so this is available to anyone to begin to manually explore today. The extraction automatically of temporal patterns of behaviour was suggested as a valuable tool to begin this analysis.
Media Consumption Analytics. It was suggested that analysing the media that a lifelogger consumed could be very valuable both for organisations and as a context source for better quality search. Once again, the discussion came to the conclusion that this was also difficult to achieve, but that it is a worthy goal for the research teams.
Other discussion points included support for and appropriateness of sharing in real-time. Past experiences were shared of when this can work and when it can go wrong. It was also suggested that a ‘loved-one’ reminder tool could be developed as a form of ‘remembering future intentions’, which was pointed out in the Lifelogging talk earlier that day as one of the five use-cases for photo lifelogging.
The session ended with the organisers thanking the attendees and the post-session discussions began and continued for thirty minutes, with some continuing to this day. In summary we found out that both food / diet and behaviour / lifestyle were the most important QS-based automatic monitoring tools that should be refined and made available to the QS community.