Multimodal AI Models Explained: Text, Image, Audio, and Video

Introduction

This guide covers multimodal AI models explained: text, image, audio, and video, from the encoders that read pixels to the training recipes that align them with language. A multimodal model is one system that accepts several kinds of input, such as photographs, speech, and video clips, and reasons over them together. The multimodal AI market was valued at about USD 2.6 billion in 2025, according to PS Market Research, which projects USD 23.3 billion by 2032. Our earlier overview of multimodal AI covered the market story and the headline products. This article goes one level deeper and explains how the machinery actually works. You will see how each modality becomes tokens, how those tokens are fused, how training and evaluation differ from text-only models, and what deployment costs in latency and money. Every technical claim links to a primary paper or vendor page, so you can check the numbers yourself.

Quick Answers on Multimodal AI Models

What is a multimodal AI model?

Multimodal AI models explained: text, image, audio, and video all pass through modality encoders that turn raw signals into tokens, which a shared transformer then processes together to produce text or other outputs.

How do multimodal models read images and audio?

Each modality gets its own encoder. A vision transformer slices an image into patches, an audio encoder converts waveform frames into embeddings, and a projector maps both into the language model’s embedding space.

Are multimodal AI models more accurate than text-only models?

Not automatically. They answer visual and spoken questions that text-only models cannot, but they still hallucinate objects and misread charts, so benchmarks such as MMMU and POPE remain essential checks.

Key Takeaways

• Every modality is converted into a sequence of tokens or embeddings, and the language backbone reasons over that mixed sequence.
• Fusion design (early, late, or cross-attention) decides the trade-off between quality, training cost, and flexibility.
• Image, audio, and video inputs consume far more tokens than text, so token budgets drive latency and cost.
• Benchmarks and hallucination tests are required because fluent answers about a picture can still be wrong.

What Is a Multimodal AI Model?

In the context of multimodal AI models explained: text, image, audio, and video, a multimodal model is a single neural network that encodes several data types into a shared token space and reasons across them jointly.

How Multimodal Models Grew Out of Separate Pipelines

Building on that definition, it helps to remember that early “multimodal” products were really chains of separate specialists. A speech recognizer produced a transcript, an optical character reader extracted text from pictures, and a captioning network described scenes, after which a language model stitched the outputs together. Each handoff discarded information, because a transcript loses tone and a caption loses spatial layout. Errors also compounded, since a mistake in the first model became an unquestioned fact for the next one. Engineers accepted those costs because no single network could handle every signal type well.

The turning point came when researchers showed that one architecture could ingest any signal once it was cut into pieces. The Vision Transformer treated an image as a sequence of patches, the same way a language model treats words, and removed the need for convolutional layers. In 2021, OpenAI’s CLIP trained on 400 million image and text pairs from the web and learned to match captions to pictures without task-specific labels. That contrastive recipe gave the field a shared embedding space where words and pixels could be compared directly. Once images and text lived in one space, every later multimodal system became an exercise in connecting that space to a language model. Courses on computer vision fundamentals still teach the convolutional era, but the patch-and-attend approach now dominates.

From there, the field moved quickly through a series of bridges between frozen components. Flamingo showed that interleaved images and text could support few-shot prompting, and BLIP-2 connected a frozen image encoder to a frozen language model through a small trainable module. LLaVA then proved that a simple projection layer plus synthetic instruction data could produce a capable visual assistant on a modest budget. Commercial launches such as GPT-4o and Gemini pushed further by training one model across text, vision, and audio instead of chaining specialists. The result is a family of systems that differ mainly in where they place the seams between modalities.

The Anatomy of a Multimodal Model: Encoders, Connectors, and a Language Backbone

Turning to the architecture itself, most modern multimodal models share the same three-part layout. A modality encoder converts raw pixels, waveforms, or video frames into a sequence of feature vectors. A connector, often called a projector or adapter, reshapes those features so they match the width of the language model’s token embeddings. The language backbone, usually a decoder-only transformer, then reads the combined sequence and generates text one token at a time. This layout lets teams reuse strong pretrained parts instead of training every component from scratch.

The encoder is typically a Vision Transformer trained with a contrastive objective, because that training already aligns images with language. For audio, the encoder may be the front half of a speech recognition model or a neural codec that quantizes sound into discrete codes. The connector can be as simple as a single linear layer, as in the first LLaVA release, or a small multilayer perceptron in later versions. More elaborate connectors compress the token count, which matters because every extra visual token costs compute at inference time. The language backbone stays recognizable: it is the same attention stack that powers text-only chat models, now fed a longer and more varied input.

A well-known compression design is the querying transformer introduced with BLIP-2. It uses a fixed set of learnable query vectors that attend to the image features and distill them into a short sequence for the language model. Because the image encoder and the language model both stay frozen, only the small bridge module needs training. The same idea appears in the Perceiver Resampler used by Flamingo, which squeezes variable-length visual features into a constant number of tokens. These bridges show a general principle: the connector decides how much visual detail survives the trip into language space. Developers who want a refresher on how vectors represent meaning can review how word embeddings work, since the connector’s job is to land image features in that same space.

A fourth component appears in models that also generate non-text outputs. A decoder head, such as an image tokenizer decoder or a speech vocoder, converts the model’s output tokens back into pixels or sound. Understanding-only models such as many visual question answering systems skip this stage and emit text exclusively. Any-to-any systems keep it, which is why they can talk, draw, and describe within one conversation. Knowing which of the four components a product includes tells you most of what it can and cannot do.

How Images Become Tokens Inside a Vision Encoder

Looking closer at the image path, the first step is to cut the picture into a grid of equal squares. A common configuration uses 16 by 16 pixel patches, which is the origin of the paper title “An Image is Worth 16×16 Words” from the Vision Transformer authors. Each patch is flattened, multiplied by a learned matrix, and combined with a position embedding so the model knows where it came from. The resulting sequence feeds into ordinary transformer layers that let every patch attend to every other patch. A 224 by 224 image cut into 16 pixel squares yields 14 by 14, or 196, patch tokens, and larger images or smaller patches raise that count quickly.

Token count is the central cost lever for any vision-enabled application. Many assistants use an encoder with 14 pixel patches at 336 pixel resolution, which produces 576 tokens for a single image. Doubling the resolution multiplies the patch count by four, so a high-detail screenshot can consume thousands of tokens before the user types a word. Newer designs such as Qwen2-VL let the number of visual tokens vary with image size instead of forcing every picture to a fixed square. That flexibility preserves detail in wide charts and tall documents but makes cost less predictable. The mechanics of splitting inputs into discrete units are the same idea described in our guide to tokenization in large language models, applied to pixels instead of characters.

How Audio Becomes Tokens: Spectrograms, Codecs, and Speech Encoders

Moving on from pixels to sound, audio enters a model through one of two main routes. The first route converts the waveform into a log-Mel spectrogram, a picture of how energy is distributed across frequencies over time, and feeds it to a transformer encoder. Whisper takes this path, processing audio in 30-second windows and producing roughly 1,500 encoder frames for each window. The second route uses a neural codec that compresses the waveform into a stream of discrete codes, which a language model can predict like words. The first route favors understanding and transcription, while the second route favors generation, because codes can be decoded back into sound.

Meta’s EnCodec is a widely used codec of the second kind. It encodes 24 kHz audio at about 75 frames per second and quantizes each frame with several stacked codebooks, a technique called residual vector quantization. With eight codebooks of 1,024 entries each, every frame carries 80 bits, which works out to a 6 kilobit per second stream by simple arithmetic. Language models then treat those codes as a vocabulary, a method explored in Google’s AudioLM and MusicLM systems for generating speech and music. Choosing more codebooks improves fidelity but multiplies the number of tokens the model must predict for each second of sound.

Once audio is tokenized, the language model treats it like any other sequence, though the rate matters. Speech arrives at roughly 50 to 75 frames per second, so a minute of audio becomes thousands of tokens before codebooks are counted. Understanding models therefore pool or downsample encoder frames before handing them to the backbone. Real-time assistants face an extra constraint, since they must start generating before the speaker has finished. Audio is cheap to hear but expensive to speak, because generation must predict many codes for each second of output. This asymmetry explains why voice assistants usually stream their responses in small chunks.

How Video Adds Time: Frames, Spacetime Patches, and Token Budgets

Shifting focus to moving images, video is the hardest modality because it multiplies the image problem by the number of frames. The simplest method samples a few frames per second, encodes each as an image, and concatenates the tokens in order. At one frame per second and 256 tokens per frame, an hour of footage produces 921,600 visual tokens, which exceeds most context windows. Models therefore reduce the load by sampling sparsely, pooling tokens across neighboring frames, or compressing the clip before it reaches the language model. Each shortcut trades temporal detail for affordability, so fast actions and brief events are the first things lost.

Generation models handle the time dimension in a different way. The Sora technical report describes compressing video into a latent representation and then cutting it into spacetime patches that act as tokens for a diffusion transformer. Because an image is treated as a one-frame video, the same scheme covers stills and clips of different lengths and aspect ratios. Understanding is measured on separate benchmarks such as Video-MME, which contains 900 videos totaling 254 hours with 2,700 question and answer pairs. Its clips range from 11 seconds to one hour, which shows how wide the span of temporal reasoning really is. Video is where token arithmetic stops being an implementation detail and becomes the main design constraint.

Fusion Strategies: Early, Late, and Cross-Attention Compared

Stepping back from individual modalities, fusion describes where and how the separate streams meet. In late fusion, each modality is processed by its own network and only the final outputs or embeddings are combined. This approach is cheap and modular, but it limits the interaction between modalities, because a text query cannot influence how the image is encoded. Early fusion does the opposite by converting everything into one token stream from the start, so a single transformer sees interleaved text, image, and audio tokens. The middle path uses cross-attention layers that let the language model query visual features at selected depths.

Flamingo is the classic cross-attention design, which inserts new layers into a frozen language model so it can attend to visual features while keeping its text skills intact. Meta’s Chameleon represents the early-fusion extreme, a family of token-based mixed-modal models that treat images and text as one discrete vocabulary. The projector style used by LLaVA sits between them: visual tokens are placed in the input sequence, but the encoder was trained separately. Early fusion offers the most flexible generation, because the model can emit an image token wherever it likes. It also costs the most to train and demands careful normalization so that one modality does not destabilize the others.

Choosing among these designs depends on the constraint that binds first. Teams with limited compute usually start with a projector or cross-attention bridge because strong frozen parts already exist. Teams building a general assistant that must interleave modalities in its output gravitate toward early fusion. Late fusion remains common in search and recommendation systems, where the goal is to compare embeddings rather than to generate language. In every case the design should be tested against the task, since a clever architecture can lose to a simple one with better data.

The interactions inside the model also matter a great deal for safety. When fusion is deep, an instruction hidden in an image can influence the text the model writes, which creates attack surface that late-fusion systems avoid. Cross-attention models can in principle be audited layer by layer to see which visual regions drove a given word. Early-fusion models make that kind of attribution harder because every token mixes with every other from the first layer. The deeper the fusion, the more capability you gain and the more testing you owe.

Training Multimodal Models: Contrastive Pretraining and Instruction Tuning

Moving on from architecture to training, most multimodal models are built in stages rather than in a single run. The first stage teaches an encoder to align pictures or sounds with language, usually through a contrastive objective on hundreds of millions of pairs. CLIP’s recipe pulls the embedding of a matching image and caption together while pushing mismatched pairs apart across a large batch. The second stage connects that encoder to a language model and trains only the small bridge on caption data, leaving the big parts frozen. The third stage unfreezes more of the system and tunes it on instructions, so the model learns to follow requests instead of merely describing scenes. Speech systems follow a parallel path, and Whisper was trained on 680,000 hours of multilingual audio paired with transcripts collected from the internet.

Instruction data is the scarcest ingredient, because people rarely write long conversations about pictures. The LLaVA team solved this by prompting a text-only GPT-4 with image captions and bounding boxes, asking it to invent questions, dialogues, and detailed descriptions. They trained on that synthetic set and reported an 85.1 percent relative score against GPT-4 on a synthetic instruction-following benchmark. Synthetic instruction data from a stronger model is the cheapest way to turn a captioner into an assistant. The approach has a known weakness, since the generated conversations inherit the biases and blind spots of the model that wrote them. Readers can find the details in the LLaVA paper, which also reports 92.53 percent accuracy on Science QA when combined with GPT-4.

Data quality decides the final result more than model size does. Web alt text is noisy, duplicated, and often unrelated to the picture, so teams filter by similarity score and remove near duplicates. They also balance concepts to avoid a long tail of rare objects. Mixing modalities also requires careful sampling, because a batch dominated by text can erode visual skills and a batch dominated by images can erode reasoning. Preference tuning adds a final layer, where human or model judges rank answers about images and the system learns to avoid confident mistakes. Licensing and consent questions follow every one of these steps, which is why documentation of data sources has become part of responsible release practice.

Evaluating Multimodal Models: Benchmarks and Hallucination Checks

Turning to measurement, evaluation is harder than for text because answers depend on what the model actually saw. The MMMU benchmark gathers about 11.5 thousand college-level questions across six disciplines, 30 subjects, and 183 subfields, with charts, diagrams, and tables embedded in the questions. When it was published, Gemini Ultra scored 59 percent and GPT-4V scored 56 percent, which showed how much room remained. Video benchmarks add temporal reasoning, while audio benchmarks test transcription accuracy, speaker handling, and spoken question answering. A single leaderboard number hides these differences, so teams should read subscores for the skills their product needs.

Hallucination deserves its own test, because fluent captions can mention objects that are not present. Researchers behind POPE found that several representative vision-language models showed severe object hallucination, especially for objects that appear often in training instructions. They proposed a polling method that asks yes or no questions about specific objects, which gives a more stable measurement than judging free-form captions. A model that scores well on knowledge benchmarks can still invent a dog in an empty room. Production teams should therefore build a private evaluation set from their own images, audio, and documents, and rerun it whenever the model or prompt changes. Automated judges can scale that work, but they need spot checks by humans who can see the original input.

Putting Multimodal AI Models to Work: Latency, Cost, and Deployment

Shifting to implementation, the first deployment decision is a budget for tokens and time. Every image, audio second, and video frame converts into tokens that are billed and processed alongside text. A single high-resolution image can cost as much as several pages of prose, and an hour of sampled video can exceed a context window outright. Engineers therefore resize images to the smallest resolution that still answers the question, crop to the region of interest, and sample video sparsely. Our guide on reducing LLM inference costs covers batching, quantization, and routing tactics that apply directly to multimodal workloads.

Latency follows its own demanding rules for voice and video applications. A conversational assistant has to start speaking within a few hundred milliseconds or the exchange feels broken. Audio is therefore streamed in small chunks instead of returned as a finished file. End-to-end models avoid the delay of chaining transcription, reasoning, and synthesis, because a single network hears and speaks directly. Pipelines still make sense when you need to log a transcript, swap a voice, or apply a text-only safety filter between stages. Measure time to first token separately from total time, since users judge responsiveness by the first sound or word they receive.

Cost modeling should start from tokens per request and requests per day, not from the vendor’s headline price. The cost per token trends we track show prices falling while usage grows, which means budgets are set by volume more than by rate. Caching helps when the same image or document is asked about repeatedly, because the encoded prefix can be reused. Routing also saves money, since a small model can handle easy requests and escalate hard ones to a larger system. Monitor the share of requests that escalate, because a rising rate often signals a data shift before users complain.

Reliability work comes last but decides whether a pilot survives. Validate file types and sizes before they reach the model, strip metadata that may leak locations, and set timeouts for slow video jobs. Keep a fallback path for outages, such as a text-only mode that asks users to describe what they see. Treat every image, audio clip, and document as untrusted input, because instructions can hide inside any of them. Log inputs and outputs under your privacy policy so that failures can be reproduced, and review a sample of them weekly.

Where Multimodal AI Models Fall Short: Risks and Failure Modes

Despite the progress, the weaknesses of these systems are specific and predictable. Models misread dense charts, count objects incorrectly, confuse left with right, and lose track of order in long video. Optical character recognition is strong on clean documents but degrades on handwriting, glare, and low-resolution scans. Audio models can mishear names, accents, and overlapping speakers, and they may fill silence with plausible words that nobody said. The lowest hallucination rates among top AI models are still above zero, so no deployment should assume perfect fidelity.

Generative video shows another distinct class of failure that deserves attention. The Sora report admits that the model does not accurately simulate many basic physical interactions, such as glass shattering, and that long clips can drift into incoherence. Reviewers have catalogued flaws found in Sora videos that match those admitted weaknesses. Security researchers add a further risk, because text embedded in an image can act as a prompt injection that redirects an assistant. The practical rule is to keep a human in the loop wherever an error would be expensive or irreversible. Clear user messaging about confidence and limits reduces over-reliance, which is a risk even when the model is accurate.

Ethics, Privacy, and Consent in Multimodal AI

Turning to the human side, multimodal systems touch data that is far more personal than text. Faces identify people, voices carry biometric signatures, and background details in a photo can reveal home addresses or medical conditions. Training sets scraped from the web often include such material without the knowledge of the people shown, which raises consent and copyright questions that courts and regulators are still resolving. Teams should document where data came from, honor opt-out requests where they exist, and avoid collecting biometric data they do not need. Retention limits matter too, because a stored voice clip is a liability long after the feature that captured it ships.

Synthetic media sharpens these privacy and consent concerns in a considerable way. Open text-to-speech systems such as the Dia open-source speech model can generate realistic dialogue quickly, and the same capability enables voice cloning for fraud. Provenance tools such as invisible watermarks and signed metadata help, though none is robust against determined removal. Regulators increasingly expect disclosure when audio or video is machine generated, so products should label outputs and keep records of how they were made. Consent should be explicit for any feature that clones a person’s voice or likeness.

Fairness problems also appear in subtle and easily overlooked forms. Image-text training data over-represents certain regions, skin tones, and languages, so captions and recognition accuracy can vary by group. Accessibility features such as scene description for blind users show the upside, yet the same cameras can enable intrusive surveillance. Impact assessments before launch, diverse test sets, and a clear channel for reporting harm are the minimum practices. Documenting limits publicly allows downstream users to make informed choices instead of discovering the gaps in production.

Open Weights Versus Closed APIs for Multimodal Work

Choosing among the build options, the market splits between hosted APIs and models whose weights you can download. Hosted systems such as GPT-4o and Gemini offer the strongest general performance, managed scaling, and frequent upgrades, but your data leaves your environment. Open-weight families give control over deployment, fine-tuning, and data residency, at the price of running infrastructure yourself. The authors of Qwen2-VL report that their 72B model achieves results comparable to leading closed models on multimodal benchmarks. That claim should be verified on your own data, but it shows the gap has narrowed.

The right choice often depends on volume and sensitivity rather than raw quality. Low-volume prototypes favor APIs because there is nothing to host, while steady high-volume workloads can justify dedicated GPUs. Regulated data, such as medical images or call recordings, may force an on-premises model regardless of benchmark rank. Whichever route you take, build a repeatable test harness, and consider using an LLM as a judge to compare candidates at scale. Keep the model behind an interface so you can swap it when a better one arrives.

The Future of Multimodal AI Models

Looking ahead, the clearest trend is native multimodality, where one model is trained on all modalities from the start instead of being assembled from parts. Google describes the Gemini family as models that understand image, audio, video, and text, available in Ultra, Pro, and Nano sizes for different devices. The smallest sizes point to on-device assistants that process camera and microphone input without sending data to a server. Any-to-any generation is the second trend, in which a single model produces text, speech, and images in one response. Both trends raise the importance of token efficiency, because the cost of reading the world continuously will decide what is practical.

Agents and robots form the third major frontier for this technology. Systems that watch a screen or a camera feed, decide what to do, and act need low latency and reliable grounding, which current models only partly provide. Projects such as Google’s Project Astra demonstrate a universal assistant that sees and hears its surroundings in real time. Longer context windows and better video compression should extend understanding from minutes to hours of footage. Expect evaluation to move from static questions toward interactive tasks, since a model that can talk about a kitchen is not yet one that can cook in it.

For practitioners, the lesson of multimodal AI models explained: text, image, audio, and video is that the field is converging on a simple pattern with endless variations. Encoders turn signals into tokens, a connector aligns them, and a language backbone reasons over the result. Each modality brings its own tokenization rules and its own cost profile. Progress will come less from new architectures than from better data, cheaper tokens, and honest evaluation. Teams that learn to budget tokens, test on their own data, and respect consent will be best placed to benefit. Revisit those choices every few months, because both the models and their prices keep changing quickly.

Key Insights

• The multimodal AI market is forecast to grow from USD 2.6 billion in 2025 to USD 23.3 billion by 2032, a 36.8 percent growth rate in PS Market Research projections.
• Text still holds about 40 percent of multimodal AI revenue, yet the same market analysis names speech and voice as the fastest-growing data modality.
• CLIP learned from 400 million web image and text pairs without task-specific labels, and the CLIP paper reports that it matched a ResNet-50 on ImageNet zero-shot.
• Whisper was trained on 680,000 hours of multilingual audio with transcripts, and the Whisper paper shows that this scale of weak supervision yields robust zero-shot speech recognition.
• The MMMU benchmark holds about 11.5 thousand college-level multimodal questions, and its authors reported that Gemini Ultra reached only 59 percent at publication.
• Video-MME spans 900 videos totaling 254 hours with 2,700 question pairs, and the benchmark paper reports that Gemini 1.5 Pro performed best among the commercial models tested.
• Google’s Gemini Ultra advanced the state of the art on 30 of 32 benchmarks and improved on all 20 multimodal benchmarks, according to the Gemini technical report abstract.

The numbers above point in one direction: demand for multimodal capability is rising faster than the tools for measuring it. Training scale has moved from millions of labeled examples to hundreds of millions of paired web samples and hundreds of thousands of hours of audio. Benchmarks such as MMMU and Video-MME reveal that even frontier systems miss a large share of expert-level questions. Text remains the largest revenue slice, but speech and voice are growing quickly because they unlock hands-free interfaces. For readers following multimodal AI models explained: text, image, audio, and video, the practical message is to budget for evaluation as seriously as for training.

Dimension	Late fusion	Cross-attention bridge	Early fusion
Where modalities meet	After separate encoders, at the output or embedding level	Inside selected language model layers through attention	At the input, as one interleaved token stream
Typical examples	CLIP-style retrieval and recommendation systems	Flamingo and similar frozen-model bridges	Chameleon and natively multimodal assistants
Training cost	Low, since each branch trains independently	Moderate, because only new layers need training	High, because the whole model trains jointly
Reuse of pretrained parts	Excellent, any encoder can be swapped in	Good, with a frozen language model kept intact	Limited, since components are trained together
Non-text outputs	Not supported without a separate generator	Rarely supported, text output is the norm	Supported through image or audio tokens
Token efficiency	Highest, no extra tokens enter the language model	High, visual features stay outside the text context	Lowest, every modality adds tokens to one context
Cross-modal reasoning	Shallow, limited to comparing final embeddings	Strong on questions that point at visual detail	Strongest, with interaction from the first layer
Interpretability	Easy to inspect each branch separately	Attention maps show which regions were used	Hard, because every token mixes with every other
Latency profile	Fast, branches can run in parallel	Moderate, with extra attention at each layer	Depends on sequence length, often the slowest
Best fit	Search, matching, and classification	Visual question answering on a tight budget	General assistants with interleaved output

Multimodal AI in Practice: Three Deployments With Real Numbers

OpenAI’s Single-Model Voice Assistant

In practice, OpenAI trained GPT-4o as a single network across text, vision, and audio, and began rolling out its text and image abilities to ChatGPT users in May 2024. Before that, Voice Mode chained three separate models, which produced average latencies of 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. According to OpenAI’s launch announcement, GPT-4o answers audio in as little as 232 milliseconds and 320 milliseconds on average. By simple arithmetic, that is a latency reduction of roughly 94 percent against the slowest earlier pipeline. The single model can also hear tone, multiple speakers, and background noise, which a transcript-based chain discards. A limitation is that audio output was still restricted to preset voices at launch, and the company rated the model medium risk under its Preparedness Framework.

Salesforce’s BLIP-2 Bridge Between Frozen Models

Salesforce researchers built BLIP-2 by connecting a frozen image encoder to a frozen large language model through a small Querying Transformer. They trained the bridge in two stages, first for vision-language representation learning and then for vision-to-language generation. The authors report in the BLIP-2 paper that the system beat Flamingo80B by 8.7 percent on zero-shot VQAv2 while using 54 times fewer trainable parameters. That efficiency means a research team can adapt a strong vision-language model without paying for a full training run. The design is still limited by its frozen parts, since the language model’s weaknesses and the encoder’s blind spots pass straight through the bridge. Such a trade-off suits prototypes well, though products that need fine visual detail may require unfreezing more layers.

Google’s SigLIP Sigmoid Loss

Google researchers replaced the softmax contrastive loss in image-text pretraining with a pairwise sigmoid loss and trained a model called SigLiT with locked image towers. The new loss scores each image and text pair on its own, so it does not need a global view of the batch for normalization. Using only four TPUv4 chips, the team reached 84.5 percent ImageNet zero-shot accuracy in two days, according to the SigLIP paper. The result shows that a cheaper loss function can put strong multimodal pretraining within reach of small labs. Gains faded quickly at scale, because growing the batch to one million examples showed diminishing returns and about 32 thousand was enough. This limitation means that bigger batches are not a substitute for better data.

Lessons From Three Multimodal AI Case Studies

Case Study: Med-PaLM M and the Generalist Medical Model

Beyond single-task demos, clinicians faced a problem that specialist models could not solve: medicine mixes notes, scans, and genomic data, yet most AI tools handled only one of them. Google researchers built Med-PaLM M as a single set of model weights that handles clinical text, medical images, and genomic data. They also assembled MultiMedBench, which spans 14 tasks across 12 datasets and 7 modalities with more than 1 million samples. According to the Med-PaLM M paper, the model was competitive with or better than specialist systems on every benchmark task. The team trained versions with 12 billion, 84 billion, and 562 billion parameters to study how scale affected each task. Chest X-ray report generation became the headline test because it demands both image reading and fluent clinical writing.

In a side-by-side review of 246 retrospective chest X-rays, four thoracic radiologists preferred the model’s reports over the human-written reference reports in up to 40.50 percent of cases. The 84 billion parameter version reached that figure, while the 12 billion version scored 34.05 percent and the 562 billion version scored 32.00 percent. Those numbers show that a bigger model was not automatically a better one for clinicians. The authors note that scaling the language model helped little for mammography and dermatology, where the vision encoder appears to be the bottleneck. They also list limitations: the benchmark datasets are small, large multimodal biomedical datasets are scarce, and real-world validation and safety work remain open. The authors also state that the language models themselves are not being open-sourced.

Case Study: RT-2 Brings Web Knowledge to Robot Control

Next, robotics teams faced a data bottleneck, because robots collect demonstrations slowly while the web holds vast numbers of labeled images and captions. Google DeepMind researchers developed RT-2 by fine-tuning large vision-language models, including PaLI-X 55B and PaLM-E 12B, to output robot actions as text tokens. The same network therefore answers questions about a scene and also emits the motor commands to act on it. According to the RT-2 paper, the team ran about 6,000 evaluation trials across seen and unseen conditions. On unseen objects, backgrounds, and environments both RT-2 variants averaged 62 percent success, against 32 percent for RT-1 and 35 percent for MOO. On emergent tests such as symbol understanding, reasoning, and person recognition, the PaLI-X version scored 60 percent against 17 percent for RT-1.

Performance on familiar tasks stayed close to the earlier model, since RT-2 reached 91 to 93 percent against 92 percent for RT-1. The gain therefore appeared only when the robot met something new, which is exactly where web knowledge should help. The paper is candid about limits: RT-2 does not learn new physical skills and only redeploys motions that appear in its robot data. Performance was poor on grasping by specific parts, novel motions such as wiping, dexterous tasks such as folding a towel, and multi-step indirect reasoning. Inference speed is another concern, since the 55B model runs at roughly 1 to 3 Hz and the 5B model at about 5 Hz. Teams should treat such models as planners that still need fast controllers beneath them.

Case Study: Waymo’s EMMA Reads Cameras Like a Language Model

Turning to autonomous driving, engineers long struggled with modular stacks in which separate networks handled detection, tracking, mapping, and planning, passing brittle handoffs between them. Waymo researchers introduced EMMA, an end-to-end model built on a multimodal large language model that maps raw camera data to driving outputs. Navigation instructions and vehicle status enter as plain text, and each task is selected by a prompt. According to the EMMA paper, the model reached state-of-the-art motion planning on nuScenes and competitive results on the Waymo open datasets. Its average planning error on nuScenes was 0.32 meters when initialized from Gemini, and 0.29 meters after pretraining on internal data.

Co-training is the key finding, because a single model trained on planning, detection, and road graph tasks improved all three. The generalist outperformed single-task models by up to 5.5 percent, which supports the idea that shared representations transfer between tasks. The authors acknowledge limits all the same, including an inability to fuse camera inputs with LiDAR or radar, which restricts three-dimensional spatial reasoning. They also cite higher computational requirements relative to conventional models, along with the expense of realistic sensor simulation for closed-loop evaluation. Open-loop benchmark numbers therefore cannot prove road safety on their own. The takeaway is that language-model reasoning can unify perception and planning, but compute and sensor coverage remain open problems.

Common Questions About Multimodal AI Models