Affective Image Set Builder

INTRODUCTION

Generating emotional states in the laboratory can be challenging. Most available methods require researchers to choose between maximizing ecological validity or experimental rigor. Affective images seem to offer a desirable balance between these values. Researchers are therefore fond of published databases of images accompanied with statistics illustrating the core affect each image is likely to elicit.

However, careless reliance of picture sets constructed using normative affective ratings can be risky. Ideally, independent variables constructed from such databases (e.g., a neutral and a negative condition) should be independent of major non-affective sources of variance in dependent variables (e.g., subjective, bodily, or brain responses). Unfortunately, this ideal is not guaranteed when stimulus sets designed to vary in core affect. Provide evdince how sets built on valence and arousal tend to be biased on other features which can affect findings.]

The present work aims to provide a toolkit for reserachers interested in creating balanced affective stimulus sets. In particular, we have worked in four inter-related areas.

1. Created a meta-database of all published affective image sets;
   
2. Extracted perceptual features of the images in the meta-database;
   
3. Extracted semantic features of the images in the meta-database;
   
4. Created an algorithmic pipeline for compiling balanced image sets.

META-DATABASE

Here are the databases that are included in the meta-database. Below is a more detailed list with each database name, citation, web link, # of pictures, picture categories, normative ratings compiled by Ola Abdelhadi.

One thing to note on this figure are the different normative rating distributions between different databases. While some of this variance is caused by genuine differences between pictures, visual inspection of the images suggests that there may be systematic biases rating scale performance (even after we have converted all ratings to a common scale).

International Affective Picture System (IAPS)

• Lang, P.J., Bradley, M.M., & Cuthbert, B.N. (2008). International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8. University of Florida, Gainesville, FL.
• Pictures: 1196
• Categories: Multiple, from furniture to complete scene, wide range of semantic categories.
• Normative ratings: Valence, arousal, and dominance

Geneva affective picture database (GAPED)

• Dan-Glauser, E. S., & Scherer, K. R. (2011). The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance. Behavior Research Methods, 43(2), 468-477. doi: 10.3758/s13428-011-0064-1
• Pictures: 730
• Categories: spiders, snakes, and scenes that induce emotions related to the violation of moral and legal norms (human rights violation or animal mistreatment). Positive and neutral pictures were also included.
• Normative ratings: Valence, Arousal, external norm and internal norm.

Nencki Affective Picture System (NAPS)

• Marchewka, A., Zurawski, L., Jednoróg, K., & Grabowska, A. (2014). The Nencki Affective Picture System (NAPS): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database. Behavior research methods, 46(2), 596-610.
• Pictures: 1356
• Categories: people, faces, animals, objects, and landscapes
• Normative ratings: Valence and arousal

Erotic subset of Nencki Affective Picture System (NAPS-ERO)

• Wierzba, M., Riegel, M., Pucz, A., Lesniewska, Z., Dragan, W. L., Gola, M., … & Marchewka, A. (2015). Erotic subset for the Nencki Affective Picture System (NAPS ERO): cross-sexual comparison study. Frontiers in psychology, 6, 1336.
• Pictures: 200

• Categories: opposite-sex couple (50), male couple (50), female couple (50), male (25) and female (25)

• Normative ratings: Valence and arousal

Set of Fear Inducing Pictures (NAPS-SFIP)

• Michalowski, J. M., Drozdziel, D., Matuszewski, J., Koziejowski, W., Jednoróg, K., & Marchewka, A. (2016). The Set of Fear Inducing Pictures (SFIP): Development and validation in fearful and nonfearful individuals. Behavior Research Methods, 1-13.
• Pictures: 886
• Categories: Spiders, bugs, blood injection, fearful social exposure, angry faces
• Normative ratings: Fear, arousal and valence

Open Affective Standardized Image Set (OASIS)

• Kurdi, B., Lozano, S., & Banaji, M. R. (2016). Introducing the Open Affective Standardized Image Set (OASIS). Behavior research methods, 1-14.
• Pictures: 900
• Categories: Animal, Object, Person, Scenery
• Normative ratings: Valence and arousal

Emotional Picture System (EmoPicS)

• Wessa, M., Kanske, P., Neumeister, P., Bode, K., Heissler, J., & Schönfelder, S. (2010). EmoPics: Subjective and psychophysiological evaluations of new imaging material for clinical-bio-psychological research. Journal of Clinical Psychology and Psychotherapy, Supplement, 1/11, 77.
• Pictures: 378
• Categories: Diverse social situations animals, and plants
• Normative ratings: Valence, excitement and complexity

Besançon Affective Picture Set-Adolescents (The BAPS-Ado)= The Attachment Picture Database (APD)

• Szymanska, M., Monnin, J., Noiret, N., Tio, G., Galdon, L., Laurent, E., … & Vulliez-Coady, L. (2015). The Besançon Affective Picture Set-Adolescents (the BAPS-Ado): Development and validation. Psychiatry research, 228(3), 576-584.
• Pictures: 93
• Categories: Distress, comfort, joy-complicity and neutral
• Normative ratings: Valence, Arousal and dominance

EmoMadrid emotional pictures database

• EmoMadrid affective picture database (http://www.uam.es/CEACO/EmoMadrid.htm)
• Pictures: 700
• Categories:
• Normative ratings: Valence and arousal

Military Affective Picture System (MAPS)

• Dretsch, M., Thiel, K., Born, S., Goodman, A. M., & Katz, J.S. (2013). Military Affective Picture System (MAPS) [Picture Repository]. Unpublished instrument. Auburn University.
• Pictures: 240
  
• Categories: Military
• Normative ratings: Valence, dominance, and arousal

Image Stimuli for Emotion Elicitation (ISEE)

• Hanjoo Kim, Xin Lu, Michael Costa, Baris Kandemir, Reginald B. Adams, Jr., Jia Li, James Z. Wang and Michelle G. Newman, ``Development and Validation of the Image Stimuli for Emotion Elicitation (ISEE),’’ Association for Psychological Science Annual Convention, poster, New York City, May 2015.
• Pictures: 10 696
• Categories: Pictures with emotional label and categorization from Flickr
• Normative ratings: valence, arousal, dominance, and likeness

Open Library of Affective Food (OLAF)

• Miccoli, L., Delgado, R., Guerra, P., Versace, F., Rodríguez, S., & Fernández-Santaella, M.C. (2016). “Affective Pictures and the Open Library of Affective Foods (OLAF): Tools to Investigate Emotions toward Food in Adults.”. PLOS ONE, DOI 10.1371/journal.pone.0158991
• Pictures: 95
• Categories: Food
• Normative ratings: pleasure, arousal, dominance, and food craving

The DIsgust-RelaTed-Images (DIRTI)

• Haberkamp, A., Glombiewski, J. A., Schmidt, F., & Barke, A. (2017). The DIsgust-RelaTed-Images (DIRTI) database: Validation of a novel standardized set of disgust pictures. Behaviour Research and Therapy, 89, 86-94.
• Pictures: 240
• Categories: food, animals, body products, injuries/infections, death, and hygiene
• Normative ratings: disgust, fear, valence, and arousal.

PERCEPTUAL FEATURES

We provide a Matlab script for extracting key image statistics that can be used to assess the perceptual homogeniety of experimental stimulus sets.

We selected a small set of preceptual features based on advances in relating image statistics to core principles of primate perception. Specifically, we computed aggregate statistics of visual information on two levels:

• non-spatial distribution of luminance and color
• spatial organization of luminance.

Non-spatial distribution of luminance and color

We first converted each picturs to the CIE Lab color space to separate luminance from color information. The CIE color space characterizes each pixel of an image with three features: its location on a luminance, a blue-yellow, and a red-green dimension.

Separating luminance information from color corresponds to functional spcialization in the brain where many perceptual computations are mainly sensitive to luminance. The characterization of color using blue-yellow and red-green dimensions corresponds to both the principal components that explain large amounts of naturally occuring color variance, as well as to the senistivity curves of bipolar color-sensitive cells in the primate visual system.

To efficiently capture the non-spatial distribution of pixel values along these dimensions, we computed the first four statistical moments of these distributions: mean, standard deviation, skewness, and kurtosis. Note that mean luminance corresponds to brightness and standard deviation of luminance corresponds to contrast.

Spatial organization of luminance

Spatial organization of luminance information is a cruical input to perception of basic qualia such as orientation, texture, movement, and shape. A useful method for numerically representing this information is spatial frequency analysis. We therefore used the 2D Fast Fourier Transform of the CIE Lab luminance channel to extract the 2D spectogram of each image.

To efficiently aggregate the 2D spectogram, we first averaged it rotationally and then computed the intercept and slope of a linear model fitted to the average spectrum on log-transformed frequency and power axis. The steeper this negative slope, the more energy can be found in lower spatial frequency relative to higher spatial frequences. Visually, such images have fewer vivid details and more large-scale edges.

The 2D spectogram has also been shown to capture spatial organization of scenes relevant for broad distinctions such as between urban and natural environments. To extract orientation information from the 2D spectogram, we fitted regression lines to the spectograms along four orientations - 45, 90, 135 and 180 degrees. We then found the orientation with higher energy, yielding a dominant orientation, and computed the standard deviation of all four slopes, yielding a measure of heterogeniety of orientations in the given image.

SEMANTIC FEATURES

We generated a semantic labelling scheme and had reserach assistants provide lavels for the images.

Below are pairwise inter-rater agreements for a subset of 600 images on two different levels of detail.

SET COMPILATION ALGORITHM

The algrithm compiles affective image sets with desired number of parallel versions and member images. In short, it seeks to minimize within-set and maximize between-set variance along affective features as well as maximize within-set and minimize between-set variance along perceptual and semantic featurues.

The algorithm has the following basic steps: 1. Define input (use full or specific image pool, set inclusion and exclusion criteria) 2. Define output (affective seeds with desired number parallel versions and members) 3. Cluster pictures separately in three different spaces: affective, semantic, and percpetual 4. Make a joint clustering by re-allocating pictures as needed 5. Distribute images from the joint clusters to requested sets

1. Define input

The algorithm can work on either the full meta-database or a user-provided source set. The input set can be further pruned by explicit inclusion and/or exclusion criteria.

Below are illustrations of the exclusion critria we used to select a subset of the meta-database for initial semantic labelling. Specifically, we excluded images that; * had a different aspect ratio from 3:4
* had fewer than 600 pixels on the shorter edge
* likely contained a solid background
* had normative valence ratings higher than 0.55 on a 0 to 1 scale

No id variables; using all as measure variables

2. Define output

For the current implementation of the algorithm, we defined:

• seeds for desired affective categories (4 corresponding to neutral, low, medium, and high negative intensity)

• desired semantic categories (4 corresponding to animate nonhuman, inanimate nonhuman, individual human, social human, and bruxism-related content)

• number of images within each set (9, driven simply by the goal to have 180 image in total)

• number of parallel sets (3)

3. Independent clustering

As its first stage, the algorithm, generates three independent clustering solutions based on affective, semantic, and perceptual features. The goal of this stage is to obtain summary features (cluster labels) that can be used to build balanced image sets. Specifically, we assume that if two image sets comprise an equal number of images from a set of clusters, then these sets will be both balanced and variable along the given clustering dimension.

Based on the desired output, we generated:

• 4 affective clusters

• 5 semantic clusters

• 9 perceptual clusters

3.1. Affective clusters

We used the K-means algorithm over normative valence data to distribute all input images into one of four clusters based on manually set seeds at 0.5 (neutral), 0.35 (low), 0.25 (medium), 0.1 (high intensity negative).

We used only valence, rather than valence and arousal to bypass apparent systematic biases in normative ratings for different source sets in our meta-database. We noticed that even after converting all ratings to a uniform 0 to 1 scale, different sets seemed to exhibit systematic biases. We therefore rely on valence ratings alone, since within negative images valence and arousal tend to be highly correlated.

3.2. Semantic clusters

The default is manual specification of labels and their desired proportions.

Options include:

• using some selection of labels to generate data-driven semantic clusters
• using Multiple Correspondence Analysis to define the feature vectors used for data-driven clustering

3.3. Perceptual clusters

The default feature set includes.

First four statistical momemts for the distributions of CieLab dimensions:

• mean (brightness), standard deviation (contrast), skewness, and kurtosis for luminance
• M, SD, s, and k for the green-red dimension
• M, SD, s, and k for the blue-yellow dimension

Representations of 2D image spectogram computed from the luminance channel:

• slope of the rotationally averaged spectrum
• mean rotationally avergaed spectrum in 7 bins
• dominant direction of the 2D spectrum out of 0, 45, 90, and 135 degrees (largest slope)
• standard deviation of slopes of all these directions

The default is hierarchical clustering with two approaches for finding a suitabel cut:

• top-down method - start from the largest number of clusters possible given the size of input and output. Lower the number until a solution with a sufficient number of sufficiently large and emotionally diverse clusters emerges. The idea is that the higher the cut-off, the more closely-knit the clusters will be allowing to miniize perceptual variance once pictures from the same perceptual cluster will be distribited across the eventual sets.

• dynamic tree cut - an R funtion that directly produces the desired number of clusters

 ..cutHeight not given, setting it to 19.7  ===>  99% of the (truncated) height range in dendro.
 ..done.

4. Joint clustering

A custom-built algorithm with the following steps:

• Create a 3D matrix from the identified affective, semantic, and perceptual clusters.
• If a cell has one or members, keep the one that has the best silhouette statistic relative to perceptual clustering, free up others
• Loop through cells without any members in randomized ordred
• Rank all free images from the affected semantic and emotion cluster and any perceptual cluster based on a) their distance to the centroid of the preceptual cluster build from only the currently selected images and b) emotional cluster
• If the above step does not yield any images, consider pictures from other emotional categories closest to the current seed until at least one is found

5. Set compilation

We now have n clusters of k pictures of p contents. Use latin square to distribute semantic contents randomly across parallel versions.

REFERENCES