Visual language models (VLMs) can follow complex text instructions, yet they struggle to reason with only visuals. In particular, current models fail to extract shared concepts from exemplar image sets and apply them to new input. We present Visual Concept Inference from Sets (VICIS), a task that tests this ability. Given a small context set of images that share a concept with a query image, the model must generate new images that preserve the concept defined by the context while remaining relevant to the query. We show that state-of-the-art VLMs perform poorly in this task, often ignoring visual context or switching to biased generations. To address this gap, we propose a training framework and architecture that learns to extract concepts from image sets and extract specific conceptual embeddings from queries. Experiments on artificial data and large ImageNet/WordNet data show that our model produces more accurate and unique outputs and integrates abstract concepts and methods such as diagrams.
