I made some research beforehand. All datasets I found are indeed facial expression datasets. To classify a facial expression is very specific task and not the thing I need.
I've was thinking of using speech data, but this looks harder than with photos.
I want to track more subtle things than facial expression. I want to know if the person is tired / having a bad mood. I believe that if you are not technically showing "sad" face (even if you are forcing a smile), it's still possible to understand that you have bad mood by looking at person. But this is only a hypothesis.
It is an interesting idea, for some people it's a easier to manually grade your state than to take a picture (or record a voice).