Pro-Bench

Prompt-robust open-vocabulary visual grounding across real-world heterogeneous environments

Linus Nwankwo1,*, Muslim Alaran1, Christian Rauch1, Stanley Chukwuebuka Obilikpa2, Elmar Rueckert1

1Chair of Cyber-Physical Systems, Montanuniversität Leoben, Austria
2Department of Design Engineering and Mathematics, Middlesex University, London, UK
*Corresponding author

A dimly lit mining tunnel with a sandy floor leading to an exit door at the far end. Annotated target, subterranean scene

Abstract

Open-vocabulary visual grounding enables robots to localise task-relevant entities from natural-language queries without dependence on predefined perceptual taxonomies. However, existing benchmarks largely rely on short category labels and web-scraped imagery, leaving it unclear whether open-vocabulary models can robustly ground diverse queries and visual conditions under real deployments. We introduce Pro-Bench, a prompt-conditioned benchmark for open-vocabulary visual grounding in heterogeneous, real-world environments. Pro-Bench includes 13k+ RGB frames from independent robotic domains (subterranean, industrial, indoor, outdoor, urban), with 74.5k manual instance annotations and 515 target queries covering categorical, attributive, relational, affordance, state, part-whole, negative, and compositional semantics. We benchmarked 16 open-vocabulary model configurations in strict zero-shot inference, measuring localisation accuracy across IoU thresholds, end-to-end inference latency, prompt-induced performance variation, and target recovery consistency. Our results show that prompt-robustness is strongly architecture-dependent. Most model configurations (10/16) perform best with short category labels, whereas free-form queries yield the highest accuracy for only one. Moreover, similar aggregate mAP can conceal substantial differences in consistent target recovery across reformulations. Pro-Bench enables systematic evaluation of these gaps and supports prompt-robust visual grounding.

Pro-Bench at a glance

13k+
RGB frames from independent robotic domains (subterranean, industrial, indoor, outdoor, urban), no web or synthethic imagery
74.5k
human-annotated polygon-based instances, boxes and masks in multiple formats, e.g., COCO, suitable for benchmarking visual reasoning models
515
operator-style queries, each with three target-preserving formulations
8
semantic prompt types, from categorical to compositional
16
open-vocabulary model configurations, evaluated strictly zero-shot on an RTX 4090

What Pro-Bench found

An operator can refer to the same object in several ways. Most current open-vocabulary models performed exceptionally well with atomic or short categorical phrasing, and some break on harmless rewording.

Short categorical labels still win

10 of 16 configurations score highest with the categorical label, 5 with the template form, and only YOLOE-X with the operator's free-form query.

A template-based method can break model predictions

Prefixing "a photo of" drops SAM3 from 0.417 to 0.044 mAP.5, while G-DINO-T gains 0.069 from the same change.

Stable boxes are not reliable recovery

SAM3's boxes barely move between phrasings (PLC 0.984), yet it recovers only 2.4% of targets under all three (PCR 0.024).

Real time costs accuracy

Within a 30 Hz budget, OmDet-T is the most accurate at 0.431 mAP.5. SAM3-Lite leads overall at 0.696 but needs 683 ms per frame.

Same target, three phrasings

mAP.5 of each model under each formulation. The line spans the prompt-induced variation Δmax; the ringed dot is the model's best formulation.

OriginalCategoricalTemplate

Best formulation across the 16 model configurations:

The benchmark

Pro-Bench RGBs are curated from real robot sensors across diverse environments, enabling the models to face low light, occlusions, unusual viewpoints, and domain-specific objects common in real-world scenarios. Click on a scene below to view the target annotations.

Where the data come from:

1) Subterranean and industrial
Tunnels, quarries, mines and excavation sites from DARPA SubT, EnvoDat, Hilti-SLAM and GOOSE-Ex.
2) Outdoor and urban
Forest trails, unstructured terrain and street scenes from EnvoDat, GOOSE-Ex, KITTI, MCD and our own campus recordings.
3) Indoor and laboratory
Offices, labs and tabletops with clutter and look-alike objects, from TUM RGB-D, EnvoDat and in-house scenes.

Data Annotation:

Each RGB frame is annotated with a free-form query and at least one semantic type, from which the categorical and template-wrapped variants are derived. Seven annotators contributed fine polygon boundaries for common and uncommon objects, and a separate reviewer checked every annotation against its query. Object sizes: 51.3% large, 28.4% medium, 20.3% small; median mask area is 2.5% of the image.

Eight semantic prompt types:

TypeIdentifies the target byExample
Pro-Bench construction pipeline: multi-domain scenes, human annotation of target instances, prompt construction with eight semantic types and three formulations, and dataset statistics.
Pro-Bench construction pipeline and dataset statistics. Given the RGB frames curated from multidomain robotic datasets that depict realistic robot operating scenarios (1), we provide annotations for task-relevant targets with fine-grained polygon-based masks (2). Each target is associated with a human-defined free-form query and at least one semantic prompt category, from which we derive the categorical and template-wrapped variants (3). The final panel (4) summarises the resulting Pro-Bench dataset statistics

Prompt robustness quantification

Every target is queried three ways. Each formulation keeps the same referent, so a robust model should find the same object every time.

Original porig

an object suitable for sensor calibration

The operator's free-form query, as written by the annotator.

Categorical pcat

a checkerboard

A manually reviewed class-level noun phrase that drops attributes, relations, state and affordance.

Template ptmpl

a photo of an object suitable for sensor calibration

The original query wrapped in the CLIP-style prompt template.

Prompt-induced variation

Δmax=maxk∈Kmk−mink∈Kmk

The spread of mAP.5 across the three formulations. Larger means the model depends more on how the target is phrased.

Prompt-consistent recall

PCRτ=1Nc∑i=1Nc∏k∈Kicorefri(k)(τ)

The share of targets recovered at IoU ≥ 0.5 under every formulation. One miss under any phrasing counts as a failure.

Localisation consistency

PLC=1|Q*|∑i∈Q*2Ki(Ki−1)∑a<bIoU(B^i(a),B^i(b))

For targets recovered under all formulations, the mean pairwise IoU of the matched boxes: how much the box moves when the wording changes.

Results

All the 16 model configurations are evaluated in strict zero-shot with frozen public weights and each model's own prompt interface. Latency is end-to-end on RTX 4090, after three warm-up passes.

Accuracy against latency

The line traces the Pareto frontier; the shaded band is the 30 Hz real-time budget of a standard RGB-D camera.

TransformerReal-timeSegmentation

Recovery against stability

PCR.5 (found under all three phrasings) against PLC (box stability once found).

Localisation tightness

How much of each model's mAP.5 survives at stricter overlap (mAP.5:.95 ÷ mAP.5).

Full results

Select a column to sort. Shaded cells are the best and second-best values; latencies within the 30 Hz budget are highlighted.

Qualitative examples

A model can find the right region yet box it far too loosely, or miss a target that is plainly visible when the query describes it indirectly. No representative model returns a valid prediction for "an object suitable for sensor calibration", although the checkerboard fills the frame.

Grid of seven scenes. Rows show the original image, the target queries with ground-truth boxes, and top-confidence predictions from YOLO-World-X, YOLOE-X, SAM 3 and OWLv2-L.
Top-confidence prediction per query for four representative models: real-time detection (YOLO-W-X, YOLOE-X), open-vocabulary segmentation (SAM3) and a transformer detector (OWLv2-L).

Video summary

Project videoCovers the benchmark, the protocol, the results and the demo. Steps for running the demo locally are in Try it yourself.
Compare models
Compares the models' performance per query, per scene, and for every selected targets.
Prompt robustness
Runs the original, categorical and template forms per model. Mark the target and each model reports whether it found it under each phrasing.
Live camera
Webcam, OpenCV or RealSense input. Each model runs on the newest frame at its own speed, and you can edit the query while it streams.

Try it yourself

Pick a scene and a query, then watch what each model returns when the same target is queried in three different ways. The walkthrough and the setup steps to run the benchmark on your own data are shown in the video.

How to use this demo
  1. Load a scene. Click a thumbnail, or hit Random to jump to another environment. The dashed white box marks the annotated target.
  2. Pick a target query. Each scene carries the operator-style queries written for it, labelled with their semantic type: relational, affordance, negative and so on.
  3. Choose the phrasing. All three at once to compare, or one at a time to see every box that phrasing returned with its confidence.
  4. Choose the models. Start with the top six, switch to the real-time group or all of them, and raise the minimum score to drop weak detections.
  5. Read the result. Each card says whether that model found the target at IoU ≥ 0.5 under each phrasing, how many of the three it recovered, and how far the box moved between them.
Demo walkthroughA recording of the demo running all 16 configurations, from a single query to the live camera.

The walkthrough, recorded in real time on an RTX 4090. The project video is in the Video summary section.

Run it on your own machine

The demo loads the real model weights, so it runs locally on a CUDA GPU.

  1. Get the code and the weights. Clone the release and install the requirements.
    pip install -r requirements.txt
  2. Start the demo server against the pro-bench config.
    python demo_server.py \
      --config configs/benchmark.yaml \
      --keep-resident \
      --examples-dir demo_examples
  3. Open http://localhost:8000 and pick a mode: compare models, prompt robustness, or live camera.
  4. Give it a scene. Upload an image, point it at a folder, or capture a frame from a webcam or RealSense camera.

Needs a CUDA GPU. All 16 models resident takes about 24 GB of VRAM; drop --keep-resident to load them one at a time on less.

Cite Pro-Bench

To cite this work, please use the following BibTeX entry:

@article{nwankwo2026probench,
  title   = {Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environments},
  author  = {Nwankwo, Linus and Alaran, Muslim and Rauch, Christian and Obilikpa, Stanley Chukwuebuka and Rueckert, Elmar},
  journal = {arXiv preprint},
  year    = {2026}
}