Modelling Visual Attention

Updated November 13, 2024 14:33

it is helpful to distinguish deep saliency models from image saliency models. Image saliency models are computed from the scene image alone by combining local contrasts in low-level, pre-semantic image features like color, luminance, and orientation across multiple spatial scales. For example, a bright red flower surrounded by green grass would be a region that would be predicted by an image saliency model to capture attention. In comparison, deep saliency models use a data-driven approach that combines deep convolutional neural networks trained on large object recognition datasets with additional network layers that are subsequently trained on human fixation data.

Deep saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data (such as SALICON) for training. Indeed, according to the current leaderboard of the MIT300 MIT/Tübingen saliency benchmark, deep saliency models dominate the top ranks and achieve AUCs of up to ~0.88 compared to human benchmark data.

Deep saliency models are state-of-the-art in modelling human attention. This is why brainsuite leverages deep saliency models in all attention-related KPIs.

Example of predictive eye tracking with a deep learning model.

While image saliency models only focus on low-level image features, deep saliency models – like the human brain – also leverage high-level, semantic image aspects such as objects, faces and emotional cues in general (e.g. Hayes & Henderson, 2021). This fact is the main reason deep saliency models perform much better in predicting human attention since human attention is known to strongly leverage higher-level, semantic and emotional aspects (e.g. Fan et al., 2023; Stoll et al. 2015). This is evident when looking at the results shown in the figure below.

In all three images, humans focus on the semantically important aspect and largely ignore the irrelevant surroundings (stone walls, floor, rocks). Importantly, aimpower’s deep saliency model closely mirrors this pattern (3rd column in the figure). On the other hand, image saliency models get distracted by the myriads of edges and intensities in the irrelevant image regions and fail to predict human attention.