The researchers used image-analysis AI on these photos so that MoodCapture's predictive model could learn to correlate self-reports of feeling depressed with specific facial expressions—such as gaze, eye movement, positioning of the head, and muscle rigidity—and environmental features such as dominant colors, lighting, photo locations, and the number of people in the image.
The concept is that every time a user unlocks their phone, MoodCapture analyzes a sequence of images in real time. The AI model draws connections between expressions and background details found to be important in predicting the severity of depression, such as eye gaze, changes in facial expression, and a person's surroundings.
Over time, MoodCapture identifies image features specific to the user. For example, if someone consistently appears with a flat expression in a dimly lit room for an extended period, the AI model might infer that the person is experiencing the onset of depression.