Estimating Color-Concept Associations from Images

Quantifying color-concept associations is a central part of our research, but obtaining those judgments from human participants is costly in time and effort. Building on prior work using large-scale image databases, we developed a new way to automatically estimate color-concept associations from images. We developed a hybrid approach that uses image statistics and human judgments. That is, we trained and tested models using human ratings on a specific set of colors, and once the models were trained, they could be used to estimate color-concept associations for new concepts and new colors without humans in the loop. The most effective model used features that were relevant to human perception and cognition, aligning with perceptual dimensions of color space and extrapolating within color categories (Rathore, Leggon, Lessard, & Schloss, 2020).

The figure below shows a process account of of how people may form color-concept associations motivated by this study (Rathore et al., 2020). When people experience co-occurrences between colors and concepts, they extract color distributions and use them to update color-concept associations. Here, color-concept association strength is indicated using bar width (e.g., blueberry is highly associated with certain shades of blue). However, in this work, we found that it was insufficient to only extract colors (or nearby colors) from images to estimate human color-concept associations. It was necessary to extrapolate to other colors that shared the same category as colors within the image to produce more accurate estimates. Based on these results, we propose there is another part of the process—category extrapolation—illustrated by the path with dashed arrows in the figure below. While extracting the color distribution from color input, people categorize colors using basic color terms (e.g., “blue”). This categorization process extrapolates to colors that are not in the visual input, but are within the same color category (e.g., extrapolating to all colors categorized as “blue” upon seeing a blueberry, even though only a subset of blues are found in the image). We believe that extrapolated colors augment the color distribution extracted from color input, which in turn further updates color-concept associations. Given prior work suggesting that categorization may influence color perception and memory, category extrapolation may also feedback to influence color experiences. Ongoing studies in our lab are directly testing the category extrapolation hypothesis for how people learn new associations between colors and concepts.

These methods have potential to advance the field’s understanding of visual reasoning for visual communication in a few ways. First, they will help make visual reasoning research more efficient by circumventing the need to collect human ratings each time we to quantify color-concept associations to use for experiment designs on interpreting information visualizations. Second, these models provide insights into how humans form color-concept associations. Third, they will help achieve a long-term goal of automatically producing optimal color palettes for semantically interpretable visualizations.