Evaluating VLMs On Real Business And STEM Tasks

Many leading benchmarks today evaluate general perception tasks—object detection, basic visual question answering, or captioning. These are valuable for measuring baseline capabilities, but they fall short in two important ways:

• They don’t reflect decision-making contexts: Real workflows involve interpreting financial reports, scientific diagrams, or operational dashboards—not labeling everyday photos.
• They avoid ambiguity: Most tasks are multiple choice, which favors surface-level understanding and eliminates the need for deep reasoning.

When vision models are deployed in the real world, ambiguity is everywhere. Charts are messy. Questions are open-ended. Tasks require synthesis across text and image—not just matching one to the other.

Turing’s benchmark is designed for business and STEM domains, where accuracy, context, and reasoning precision directly affect outcomes.

“We wanted to stress-test where VLMs fail when tasks look more like engineering problems or scientific questions—not quizzes.”
— Mahesh Joshi, Head of Research at Turing

Each task involves:

• Multimodal prompts: Text-plus-image inputs with domain-specific structure (e.g., diagrams, charts, data tables)
• Open-ended questions: No multiple choice—only free-form generation
• Domain relevance: Tasks that mirror how professionals interact with information at work

Example task:

“I’m a structural engineer designing a steel support beam. What value of h results in the maximum shear flow through the welded surfaces?”

This isn’t visual trivia. It’s an expert-level prompt requiring the model to understand a diagram, apply structural mechanics, and return a usable answer—not a guess.

To evaluate generated responses, the team uses a LLM-based judging system:

• Each model produces five outputs per prompt
• An LLM scores each output for accuracy and completeness (0.0–1.0)
• Final accuracy = average score across runs

This allows the benchmark to detect:

• Inconsistency across multiple runs
• Partial correctness
• The presence (or absence) of reasoning steps

Multiple-choice formats often hide these issues. LLM-as-a-Judge reveals them—and captures more realistic performance signals.

Across the benchmark, four leading models were tested. Key takeaways:

• All models underperform on tasks requiring iterative or counterfactual reasoning
• Temporal reasoning and analogical inference are frequent failure modes
• Tasks with longer question prompts showed higher variance in model responses
• Multistep tasks compound performance degradation, especially when chart-reading and inference are required in tandem

These aren’t fringe edge cases. They reflect the everyday demands placed on VLMs in enterprise and research workflows.

This benchmark is not static. It’s versioned, private, and built for continuous refinement. Future plans include:

• Expanding into medical, legal, and lab-based document reasoning
• Allowing collaborators to submit failure cases or prompt formats to expand coverage

For researchers and enterprise teams, this provides a framework to test real capability—not just leaderboard potential.

Read: Inside the Turing Applied AGI Benchmark for VLM 1.0 →