MindTopo Exposes a Planning Gap in VLM Spatial Reasoning

Microsoft Research and collaborators from Northwestern and Stanford introduced MindTopo on August 12. The benchmark tests whether multimodal large language models can reason about topological properties: relationships such as continuity, separation, order, enclosure, and knots that remain meaningful as objects bend, move, or change shape.

MindTopo contains 11,016 procedurally generated instances across 13 task types. Eight reasoning tasks ask models to interpret static rendered scenes, while five planning environments require a sequence of legal actions that creates, preserves, or removes a topological relationship. The simulators provide exact ground truth and adjustable difficulty, helping separate a perception mistake from a failure to track structure over time.

The project leaderboard covers 11 proprietary and open-weight multimodal models. Its best listed scores are 54.1% overall, 60.0% on the reasoning subset, and 48.3% on planning; the human references are 97.4%, 95.8%, and 100%, respectively. The top score varies by slice, so the benchmark does not identify one model as uniformly strongest.

Microsoft's analysis says static errors often begin with perception, such as missing an opening or crossing. Planning failures appear after a scene has been understood: a model chooses a locally plausible move, loses track of a relation over multiple turns, or proposes an action that conflicts with the environment's dynamics. Image generation sometimes helps when the relevant relation fits in one frame, but generated video often changes topology or violates the task rules.

A robot, accessibility tool, or interactive assistant needs more than a verbal description of a scene. It must maintain facts such as whether a route remains connected, an object stays enclosed, or two strands remain linked while actions change the view. MindTopo gives researchers a controlled way to measure that persistence.

The release is still an early research artifact: the project page labels the paper, code, and dataset as forthcoming. Teams should therefore treat the current leaderboard as a diagnostic result to reproduce, not a production-readiness ranking. The practical question is whether explicit state tracking or topology-preserving world models can close the planning gap without hiding perception failures.