A GPT-6 Astra Agent (Zero-Shot, xhigh Reasoning Effort, Codex Runtime) Succeeds on 12/15, 10/21 and 0/12 Real-Robot Trials Across Three Contact Levels in a Small, Non-Peer-Reviewed Benchmark

Analysis based on information available as of 2026-11-03.

30-second summary

• What: The PhysLadder preprint (arXiv, 2026-10-25, 19 authors) groups 16 household manipulation tasks into three levels by how much contact they demand: L1 contact selection and alignment, L2 force regulation during sustained contact, L3 contact reconfiguration plus force regulation. It tests one general-purpose frontier LLM, GPT-6 Astra, run zero-shot at reasoning effort xhigh inside a Codex agent runtime, on a bimanual ARX R5 robot and in a matched Isaac Sim environment. Real-robot successes were 12/15 (L1), 10/21 (L2) and 0/12 (L3); simulation gave 10/15, 10/21 and 0/12.
• So what: This is a small exploratory evaluation: one model plus agent harness, one robot platform, three trials per task per environment, no peer review and no independent reproduction. All figures are the authors’ claims. The four L3 tasks were 0/3 in both environments. The L1-to-L2 difference is not distinguishable from chance at this sample size (Fisher exact p of about 0.08, our calculation, assuming independent trials). The robot has no force sensor, and no robot-specific or alternative frontier model was tested, so the cause of failure (model reasoning versus missing sensing) is not separated.
• Now what: Read it as a benchmark design proposal and a descriptive data point for one model-plus-harness system, not as a measurement of what general models can or cannot do. Code and simulation assets are public on GitHub (Apache-2.0), so the simulation side can be checked by others. Watch for re-evaluation with other models, larger trial counts, blinded judging and force-sensing hardware. TRL 2 (measurement tool, not a product).

5-minute read

A ladder of contact demands

Most robot-manipulation benchmarks mix tasks of very different physical difficulty into one success rate. PhysLadder instead sorts 16 household tasks by the kind of contact reasoning they require. L1 (5 tasks) asks the system to choose where to touch and to align geometry. L2 (7 tasks) asks for sustained contact while regulating force, such as wiping a whiteboard. L3 (4 tasks) combines changing the contact configuration with force regulation, for example unscrewing a bottle cap or a nut, pushing and rotating a stack of books, or sweeping a block. The levels are cumulative and defined by task design, not by how well any model performed.

The evaluated system is GPT-6 Astra driven through a Codex runtime with tool calls, zero-shot, at reasoning effort xhigh, as the paper reports. On the real robot it also reads a pre-written “basic Skills” file that describes the available implementations and interfaces but, per the authors, contains no task strategies. The result therefore describes a model, a harness and a Skills file together, not the model alone. GPT-6 Astra is an OpenAI model that press reports say was released in early September 2026; the snapshot used, its specifications and its access route are the paper’s self-report, and we did not verify them against the vendor.

What the numbers do and do not show

The only clear pattern is at the top of the ladder: all four L3 tasks failed in all three trials in both environments (0/12 each). The drop from L1 to L2 is less certain. With 15 and 21 trials, and trials grouped within tasks, the difference between 12/15 and 10/21 gives a Fisher exact p of about 0.08 (our calculation, assuming independent trials; grouping would widen the uncertainty). The authors themselves list limited real-robot resources as a constraint on trial counts and statistical precision.

The authors also report that simulation and real results line up, with 45.8% real (22/48) against 41.7% simulated (20/48) overall. That agreement is on averages. Ten of 16 tasks had identical success counts in the two environments, six differed by one trial, and seven tasks were 0/3 in both. Within L2 the matching 10/21 hides offsetting task differences: Book Pulling was 1/3 in simulation and 0/3 real, Orange Handling 2/3 against 3/3, Drawer Pull and Return 1/3 against 2/3, Whiteboard Wiping 3/3 against 2/3. The authors state the correspondence is descriptive and specific to GPT-6 Astra.

Deep dive

1. Background

Contact-rich manipulation, meaning tasks where success depends on how forces and contacts evolve (turning a cap, wiping a surface, sliding a drawer), is a long-standing weak area for robots because small errors in contact geometry or force can change the outcome. Recent work asks whether general-purpose reasoning models, used as planners that call robot skills as tools, can handle physical tasks without task-specific training. PhysLadder contributes a structured way to ask that question: tasks are tiered by contact requirement, and each has a matched simulation scene (NVIDIA Isaac Sim with a cuRobo motion planner) and a real-robot setup (two ARX R5 arms with two-finger grippers, one external Orbbec Gemini Pro camera and two wrist Orbbec Gemini 305 cameras). By the authors’ description there is no contact force sensor and no depth image in the real setup. The simulated scenes were rebuilt from videos of the real scenes, with dimensions, masses and contact responses calibrated; the authors describe the physical agreement as qualitative, not quantitative.

2. What this study newly establishes

A graded, matched sim-real task set. The design contribution is the combination of a contact-demand hierarchy with paired simulated and real tasks, 48 trials per environment (16 tasks, 3 trials each). The authors do not claim this is the first such benchmark, and we have not checked the prior literature closely enough to say.

Reported results (authors’ claims, recomputed by us from the per-task table). Real: L1 12/15 (80.0%), L2 10/21 (47.6%), L3 0/12 (0.0%), total 22/48. Simulation: L1 10/15 (66.7%), L2 10/21 (47.6%), L3 0/12 (0.0%), total 20/48. The abstract describes real-world success declining from 80.0% to 47.6% to 0.0% across the levels.

Failure patterns. The authors describe four bottlenecks: spatial reasoning and execution precision, choice of contact configuration, force regulation, and coordinating contact with force. They state these are not mutually exclusive causes, and that comparing levels does not isolate causal factors.

Evaluation protocol. In simulation, the agent issues one structured action per call and cuRobo executes it. On the real robot there is no time or call limit and the model may end the episode itself; operator interventions and protective stops count as failures. We did not find a count of how often a human intervened.

3. Strengths and limitations

Strengths.

• The task hierarchy gives a more informative read than one pooled success rate, and L3 results isolate the hardest combination.
• Real-hardware trials sit next to a matched simulation, which is useful for locating where simulation and reality diverge.
• Code is public on GitHub under Apache-2.0, along with simulation assets (as a separate submodule with pinned hashes) and the automatic evaluator. Real-robot videos are linked from the project page. The repository notes it has been moved to a repository named LLM4Robot-Bench. We did not run the code.
• The authors state the limitations of trial count and of the descriptive sim-real correspondence themselves.

Limitations.

• Small, clustered sample. Levels have 15, 21 and 12 trials, formed from three repeats of 5, 7 and 4 tasks. Repeats within a task share objects and similar starting conditions, so the effective sample is closer to the number of tasks. We computed Wilson 95% intervals assuming independent trials (our calculation, not the paper’s): L1 about 55-93%, L2 about 28-68%, L3 about 0-24%. They overlap between L1 and L2, and clustering would widen them. At task level (real robot, at least one success), L1 was 4/5 against L3 0/4, with Fisher p of about 0.048 (our calculation), so even the L3 floor rests on four tasks.
• One model plus harness, no comparison. The paper reports a single frontier LLM with the Codex runtime and a Skills file. There is no robot-specific vision-language-action model, no other frontier model, and no ablation of the harness. What is measured is the zero-shot performance of one general frontier LLM with one agent harness.
• Hardware confound. With no force sensor, failures at L2 and L3 may reflect missing sensing as much as model limits. The study cannot separate the two.
• Asymmetric judging. Simulation is scored automatically with fixed geometric tolerances. Real trials were judged visually from video, stopping once alignment “appeared adequate” in the authors’ wording. We found no information on who judged or whether they were blinded, so the higher real-robot L1 result (80.0% vs 66.7%) could partly reflect lenient judging. Simulation start conditions vary, while real-robot scenes were reset by hand.
• Generous conditions. No time or call cap on the real robot is favourable to success compared with a deployment setting.
• Single objects and platform. One asset per object, one robot, one gripper type. No safety or long-run reliability measures.
• Preprint. Version 1 only, 31 pages, not peer reviewed. We read the HTML full text through the main sections and most of the appendix; part of the appendix, including simulation budget values, was not read.

4. Connections to neighboring domains

AI/ML agents. The evaluated system is a general-purpose reasoning model driven as a tool-calling agent, so the work belongs as much to the study of LLM agents as to robotics. The paper does not compare against robot-specific foundation models, so it says nothing about how general and specialised approaches rank on these tasks.

Laboratory automation (analogy only). The L3 tasks, unscrewing a bottle cap or a nut, involve the same kind of contact reconfiguration plus force control as opening caps and handling tubes in lab automation. The paper evaluates no laboratory tasks, so this is an analogy, not evidence about lab robots.

5. Commercialization and investment perspective

TRL 2. This is a measurement tool and evaluation methodology. The test platform is a research bimanual setup, not a product. A commercial timeline does not apply; whether the benchmark is adopted by others is unknown, and we found no evidence of adoption yet.

Related companies (facts only, no investment inference). The paper names GPT-6 Astra as an OpenAI model (privately held; access method and any financial relationship with the authors are not stated in the parts we read). NVIDIA Isaac Sim (NVIDIA, listed) is the simulation platform and cuRobo is the planner. The robot is an ARX R5, described as an ARX product; we did not check ARX’s ownership or listing status. Three author affiliations look like companies (Tuojing Intelligence, Sapient Intelligence, SimpleAI); we have not checked their business, listing status or vendor relationships.

6. Counterarguments

• The scope of the conclusion. It is tempting to read the zero on L3 as a limit of general-purpose models. The study tests one general frontier LLM with a Codex harness, zero-shot, on one robot. It supports only a statement about that combination.
• Chance and clustering. A drop from 80.0% to 47.6% sounds large, but with 15 versus 21 trials the difference is not separable from chance in our calculation, and the three repeats per task are not independent. Even the L3 result rests on four tasks.
• Sim-real agreement may be partly a floor effect. Seven of 16 tasks were 0/3 in both environments, which makes agreement easy to obtain but carries little information. Equal L2 averages hide task-level differences in both directions.
• Judging and conditions. Visual, non-blinded (as far as we can tell) real-robot judging, no call limit, and manual resets differ from the automatic simulation scoring, so the two columns are not measured the same way.
• Cause of failure. Without force sensing, and without a second model or a robot-specific baseline, the study cannot tell whether failures come from the model’s physical reasoning, the harness, the Skills interface or the hardware.
• Title framing. The preprint’s framing around how far a model can climb the ladder and around sim-real correspondence is larger than a three-trial-per-task design can bear, though the authors do qualify both points in the text.

7. Indicators to watch

• Re-evaluation with other frontier models and with robot-specific models on the same public tasks, using the released code.
• Larger trial counts per task (tens rather than three), with confidence intervals reported by task.
• Blinded or automated judging of real-robot trials, and reporting of human-intervention counts.
• Repeating L2 and L3 on hardware with force or tactile sensing, to separate sensing limits from model limits.
• Independent reproduction of the simulation results from the Apache-2.0 repository and assets.
• Whether the sim-real correspondence holds for models other than GPT-6 Astra, which the authors say remains to be tested.
• Peer-reviewed publication and any change between version 1 and later versions.

References

• Yu, Haibao, Jiaqi Meng, Zijian Zhang, et al. (19 authors). 2026. “How High Can GPT-6 Astra Climb in Contact-Rich Manipulation? PhysLadder: A Hierarchical, Matched Sim-Real Benchmark for Physical Interaction.” arXiv:2610.34527 [cs.RO], v1, 25 October 2026. https://doi.org/10.48550/arXiv.2610.34527; https://arxiv.org/abs/2610.34527. HTML full text: https://arxiv.org/html/2610.34527
• PhysLadder code and simulation assets (Apache-2.0): https://github.com/PhysLadder-Benchmark/PhysLadder
• Only the primary preprint and its public repository were available as verified sources for this piece, so the Tier 3 coverage is narrower than usual. Related benchmarks, robot-specific models and vendor material for GPT-6 Astra are not itemised because their bibliographic details were not verified.

Disclosure

• This article is for informational purposes only and is not investment advice.
• The author holds no position in any company named in this post.
• Conflict-of-interest notes:
  • The preprint lists 19 authors and 13 affiliations: Tuojing Intelligence; The University of Hong Kong; Chinese Academy of Sciences; Nanyang Technological University; University of Michigan, Ann Arbor; Southeast University; Sapient Intelligence; SimpleAI; Mohamed bin Zayed University of Artificial Intelligence; Stanford University; Imperial College London; Tsinghua University; Carnegie Mellon University. We could not confirm which author is linked to which affiliation, so none is attributed to individuals here.
  • Haibao Yu is the first author and the corresponding author; the paper marks Yu, Meng and Zhang as equal contributors.
  • Tuojing Intelligence, Sapient Intelligence and SimpleAI appear to be company affiliations; we did not verify their business, listing status or vendor relationships.
  • In the portions we read, we did not locate acknowledgments, funding, competing-interests or author-contribution statements; a small part of the appendix was not read.
  • The evaluated model, GPT-6 Astra, is an OpenAI model. Any financial relationship between OpenAI and the authors, and the way the authors accessed the model, were not stated in the parts we read.
  • The evaluation environment uses NVIDIA Isaac Sim, cuRobo, ARX R5 robots and Orbbec cameras; we found no stated interest between the authors and these suppliers, and did not verify any.
  • All quantitative results are the authors’ (preprint) claims and have not been independently reproduced. The model’s specifications and settings are the paper’s self-report. Wilson intervals and Fisher tests in this article are our calculations assuming independent trials, not figures from the paper.
• The figure in this article is an original drawing; no figure from the source was reproduced.