A vision model that can only recognize the 80 categories it was trained on is a poor fit for an agent that has to operate in an open world. If a browser-use agent needs to click “the button labeled Export, not the one labeled Save,” or a warehouse robot needs to pick up “the blue tote on the second shelf, not the red one,” a fixed-taxonomy detector cannot help — the target category was never in its training label set to begin with.

Open-vocabulary detection and segmentation models solve a narrower problem than that framing suggests, and the gap between what they actually do and what agent builders assume they do is where a growing share of production failures originate. This piece works through how these models function, how they get composed into pipelines like Grounded SAM, why agent architectures increasingly depend on them as a grounding layer, and — more importantly — the specific ways that layer fails silently once it is embedded in a decision loop rather than evaluated as a standalone benchmark task.

From Fixed Taxonomies to Text-Prompted Grounding

Classical object detectors — the Faster R-CNN and YOLO lineage that dominated computer vision through the late 2010s — are trained against a closed label set. A model trained on COCO’s 80 categories can detect “dog” and “chair” reliably and has no mechanism for detecting “the cracked ceramic mug” at all. Extending the label set means retraining or fine-tuning.

Open-vocabulary models break this constraint by aligning visual features with a language embedding space rather than a fixed classification head. The foundational move was CLIP-style contrastive pretraining, which learns a shared space where an image region and a text description of that region land close together. Once that alignment exists, “detection” becomes a matching problem between arbitrary text and image regions rather than a fixed multi-class classification problem.

Grounding DINO operationalized this for detection specifically: it fuses a language-conditioned encoder with a DETR-style detection transformer — the transformer-based detection architecture that emerged from the same shift toward attention-based vision backbones covered elsewhere on this site — so a free-text phrase, “the person wearing the red helmet,” not just “person,” can condition which regions the model proposes as boxes. Grounding DINO 1.5, trained on roughly 20 million grounding-annotated images, reports 54.3 AP on COCO and 55.7 AP on LVIS-minival in a zero-shot setting — meaning without any task-specific fine-tuning on those benchmarks (Ren et al., 2024, arXiv:2405.10300).

Segment Anything (SAM) solved the adjacent problem of promptable segmentation: given a point, box, or rough mask as a prompt, produce a precise pixel-level mask for the corresponding object, without needing to know the object’s category at all (Kirillov et al., 2023, arXiv:2304.02643). SAM 2 extended this to video, adding object tracking across frames from a single-frame prompt (Ravi et al., 2024, arXiv:2408.00714).

Neither model does what an agent actually needs on its own. Grounding DINO locates a region matching a text description but does not produce a precise mask. SAM produces a precise mask but needs a prompt — a point or box — that something else has to supply. The combination of the two, not either model individually, is what makes open-vocabulary grounding usable as infrastructure.

The Grounded-SAM Pattern

Grounded SAM (and its video-capable successor, Grounded SAM 2) is not a single model — it is a pipeline composition. A grounding model (Grounding DINO, or comparable models such as Florence-2 or DINO-X) proposes candidate regions from a text prompt; those regions are passed as box prompts into SAM, which produces the pixel-accurate mask; for video, a tracker propagates that mask across subsequent frames rather than re-running detection on every frame.

This composition pattern — text in, precise region out — is what makes open-vocabulary vision usable as a component inside a larger system rather than an end-user-facing tool. It shows up across a range of applied use cases: robotic manipulation (identify and segment “the object nearest the gripper” from a natural-language instruction), automated annotation pipelines (bootstrap labeled masks from text prompts instead of manual polygon drawing), video editing and rotoscoping, and — increasingly — as the perception layer inside multi-step AI agents that need to act on a specific region of a scene rather than merely classify it.

That last category is the one worth examining closely, because it changes what “good enough” performance means.

Why Agent Loops Need a Grounding Component

An agent that only needs to answer “is there a dog in this image” can tolerate a fair amount of imprecision — a wrong confidence score or a slightly off bounding box rarely changes the final answer. An agent that needs to click a specific UI element, direct a robotic gripper to a specific object, or verify that a physical precondition holds before taking the next step in a plan has no such tolerance. The vision output is not a label anymore; it is the input to an action.

This is the structural difference between vision-as-classification and vision-as-grounding-for-action: the model’s output has to be resolved to a specific coordinate, region, or object identity that a downstream action consumes directly. GUI agents are a clean example — locating “the button labeled Export” on a screenshot and clicking it requires region-level grounding that generalizes across arbitrary, previously unseen interface layouts, which is precisely the open-vocabulary problem rather than a fixed-taxonomy one. Recent work on region-aware grounding for GUI interaction treats this explicitly as its own sub-problem, distinct from general open-vocabulary detection in natural scenes, because interface elements have different visual statistics than the natural images most detection models were trained on (arXiv:2507.05673).

The same pattern holds for embodied and planning-oriented agents. A vision-language model asked to verify whether an action’s precondition holds — is the drawer open, is the object graspable from this angle — is being asked a grounding question, not a captioning question. Benchmarks built specifically to test this, such as ViPlan, evaluate vision-language models on visual planning tasks that require grounding symbolic predicates (open, on-top-of, reachable) to specific regions of an image, and the results are informative precisely because they separate “can the model describe the scene” from “can the model ground a precondition correctly enough for a planner to act on it” (Ubbiali et al., 2025, arXiv:2505.13180).

The Failure Modes Detection Benchmarks Don’t Capture

Standard open-vocabulary detection benchmarks report mean average precision against a fixed evaluation set with known ground truth. That number is a reasonable proxy for “does this model generally find the right kind of object,” and it is a poor proxy for “will this model’s output be safe to act on inside an agent loop.” Several failure modes fall into that gap.

Bounding-box imprecision that survives a passing score. Research on GUI grounding specifically has found low-IoU patterns where a model’s predicted box overlaps the target broadly but its center point — the coordinate most action interfaces actually click — falls outside the ground-truth region entirely. A model can be “roughly right” in a way that scores acceptably on overlap-based metrics while being wrong in the specific way that matters for an agent that clicks at a box’s centroid.

Hallucinated objects, attributes, and spatial relations. Vision-language grounding components can report objects that are not present in the scene, or correctly identify an object but misattribute a spatial relationship to it — “the cup is to the left of the plate” when it is not. In a captioning context this is an accuracy problem. In an agent loop that plans its next action based on that spatial claim, it is a correctness-of-action problem, and it does not announce itself as an error; the downstream step simply executes against a false premise.

Incorrect grounding of action preconditions. This is the specific failure ViPlan-style evaluation targets: a model can describe a scene fluently while grounding a precondition predicate — “is this drawer open” — to the wrong region or the wrong object, producing a plan that is internally consistent but built on a false perceptual claim. Analyses of vision-language planning failures identify this as a leading failure category, ahead of pure language-side reasoning errors, and distinct from the general captioning-hallucination problem because the error is specifically about resolving a symbolic condition to a region of pixels.

Tool-limitation failures that look like model failures. An agent frequently asks its vision component for a judgment the underlying model cannot reliably support at the resolution, distance, or occlusion level present in the scene — fine-grained text reading in a cluttered image, or object identity through partial occlusion. The failure surfaces downstream as a bad action, but the root cause is a mismatch between what the agent’s plan assumes the perception tool can do and what it can actually do reliably.

What makes all four of these load-bearing for agent design is that none of them require the grounding model to be badly wrong on average. A model with strong aggregate benchmark numbers can still produce exactly the wrong answer at exactly the moment an agent commits to an irreversible action, and the aggregate score gives no visibility into that tail.

Instrumenting Grounding as a Monitored Stage, Not a Black Box

The practical response is the same one that applies to orchestration failures in multi-agent systems more broadly: treat the vision-grounding step as its own instrumented pipeline stage with its own evaluation surface, rather than folding it into an opaque “perception” black box and only measuring whether the agent’s final task succeeded.

Log grounding confidence and region provenance separately from the action taken. If the agent clicks a coordinate, log which model produced the candidate region, its confidence score, and the box geometry that decision was based on — not just the click itself. This is what makes a silent grounding failure diagnosable after the fact instead of indistinguishable from an unrelated planning error.

Evaluate grounding against decision outcomes, not just IoU. A box with 0.6 IoU against ground truth that still contains the correct click target is operationally fine; a box with 0.8 IoU whose centroid lands outside the clickable region is operationally broken. Standard detection metrics do not distinguish these cases; an agent-specific evaluation harness has to check the metric that actually matches how the region gets consumed.

Use targeted grounding benchmarks, not general detection leaderboards, to validate the specific capability you depend on. ViPlan-style precondition-grounding evaluation and GUI-specific region-grounding evaluation each isolate a narrower capability than general open-vocabulary detection APs. Validate against the benchmark that matches your actual consumption pattern — clicking, gripping, precondition-checking — rather than a general-purpose leaderboard number that was never designed to predict it.

Fine-grained feedback reduces grounding-specific hallucination. Research on de-hallucination for grounded vision-language systems has found that dense, region-level feedback — rather than a single scalar reward for the whole response — combined with rejection sampling meaningfully reduces the rate of hallucinated objects and misattributed spatial relations, compared to coarser reward signals that only judge the final answer.

Treat perception-tool limitations as a distinct failure class in your taxonomy. When an agent’s action is wrong because the vision tool could not reliably resolve the scene at hand — heavy occlusion, poor resolution, an object genuinely outside the training distribution — that is a different failure to fix than a hallucination or a mis-grounded precondition. Conflating them under a single “vision was wrong” bucket makes the failure harder to route to the right fix.

None of this requires abandoning open-vocabulary grounding models as infrastructure — the composition pattern behind Grounded SAM and its successors remains the most practical way to give an agent a general-purpose “find this described thing” capability without retraining a detector for every new object category. It requires not trusting that capability past the point its evaluation actually covers.

Frequently Asked Questions

What is an open-vocabulary vision model?

An open-vocabulary vision model detects or segments objects described by arbitrary text at inference time, rather than being limited to a fixed set of categories it was trained to classify. It works by aligning visual regions and text descriptions in a shared embedding space, so a previously unseen category description can still be matched against image content without retraining.

What is Grounded SAM, and is it a single model?

Grounded SAM is a pipeline, not a single model. It combines an open-vocabulary detector such as Grounding DINO — which proposes candidate regions from a text prompt — with the Segment Anything Model (SAM), which turns those regions into precise pixel-level masks. Grounded SAM 2 extends the same composition with SAM 2’s video tracking, propagating a mask across frames from a single prompted frame.

Why does an AI agent need vision grounding instead of just image classification?

Classification answers “what is in this image,” which is enough for many perception tasks but not for action. An agent that clicks a UI element, directs a robotic gripper, or checks whether a physical precondition holds needs its vision output resolved to a specific region or object identity that a downstream action can act on directly — that resolution step is grounding, and it is a materially harder and more failure-prone problem than classification.

Can a vision-grounding model hallucinate, the way a language model can?

Yes. Vision-language grounding components can report objects, attributes, or spatial relationships that are not actually present in the scene, and they can also correctly detect an object while misattributing a spatial relationship to it. In an agent loop, this does not surface as an obvious error — the downstream step simply acts on a false perceptual claim, which is why logging grounding confidence and region provenance separately from the action taken matters for diagnosing these failures after the fact.

Do high detection benchmark scores mean a grounding model is safe to use in an agent pipeline?

Not by themselves. Aggregate benchmark scores like mean average precision measure general detection quality across a broad evaluation set; they do not measure whether a specific box’s centroid lands inside the region an action will target, or whether the model reliably grounds symbolic preconditions rather than just describing a scene fluently. Validate against evaluation that matches your actual consumption pattern — click-target accuracy, precondition-grounding accuracy — rather than relying on a general leaderboard number.