REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image

REST3D is single-image 3D reconstruction built for physical stability, not just visual plausibility.

REST3D means REconstructing physically STable 3D scenes. The central idea is brutal and simple: a 3D scene that looks correct is not enough if objects float, intersect, explode under gravity, or collapse in a simulator. REST3D uses physical scene understanding and physics-constrained optimization so the reconstructed scene can behave like a usable digital asset.

REST3D searchers want proof, demos, code status, metrics, and a fast explanation of why physical stability matters.

REST3D is a CMU computer-vision research framework, not a sleep supplement, mobile game, or generic 3D tool.

Searches for "REST3D" pull up several unrelated products and projects. To help visitors and search engines, we explicitly list what REST3D is NOT:

REST3D abstract: Reconstructing Physically Stable 3D Scenes from a Single Image

Reconstructing physically stable 3D scenes from a single RGB image enables casual images to be converted into simulation-ready digital assets for applications such as immersive interaction and content creation. However, existing single-image reconstruction methods fall short in capturing the physical structure of a scene. As a result, they often produce geometrically plausible but physically inconsistent results, including object floating and penetration, which lead to unstable behavior in physics simulations. Image-conditioned scene generation methods improve physical plausibility but often rely on strong scene priors, yielding plausible yet inaccurate object arrangements that fail to match the input image. We propose REST3D, a single-image reconstruction framework that can REconstruct physically STable 3D scenes by integrating physical scene understanding with physics-constrained refinement. We first introduce an agentic physical scene understanding technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective, providing a structural prior for reconstruction. Leveraging this structure, we initialize the scene using image-to-3D models, followed by scene-tree-guided alignment and physics-constrained optimization to resolve physical violations while preserving visual consistency with the input image. Experiments show that our method significantly reduces physical errors and improves simulation stability on both synthetic and real-world datasets while maintaining strong reconstruction quality. We further demonstrate the reconstructed scenes in VR-based human-object interaction, showing their potential for immersive applications.

From a single casual image to a visually consistent and physically stable interactive 3D scene.

Single-image 3D reconstruction often fails when gravity asks the obvious question: should this object stand, fall, or explode?

REST3D combines scene-tree construction, scene-tree-guided alignment, and physics-constrained optimization.

REST3D scene-tree construction models gravity-support relationships: ground, wall, ceiling, and ground-wall.

A REST3D scene tree is not a decorative hierarchy. It is the structural prior that says which object supports which object: table on ground, plant on table, poster attached to wall, radiator supported by ground-wall. This is the hidden skeleton that lets REST3D keep visual reconstruction and physical behavior aligned.

REST3D uses agentic physical scene understanding to identify objects, segment instances, and reason about spatial support.

REST3D initializes the 3D scene with image-to-3D models, then canonicalizes the layout so physics has a fighting chance.

REST3D starts with raw image-to-3D output, then uses the scene tree to correct coarse orientation, enforce support, and produce a structured initial scene. Canonicalization alone is not enough; it improves stability but still needs the full REST3D physics-constrained optimization stage.

REST3D physics-constrained optimization resolves physical violations while preserving visual consistency.

REST3D targets simulation-ready digital assets for immersive interaction, content creation, gaming, and embodied AI.

The high-value promise of REST3D is practical conversion: one casual image becomes a 3D scene with object meshes and world-frame layout that can be imported into physics simulation. For users searching REST3D, the phrase simulation-ready digital assets should appear early and repeatedly because that is the real difference from ordinary image-to-3D reconstruction.

REST3D Interactive 3D Physics Simulation in Isaac Gym

Explore the physics simulation of reconstructed scenes in Isaac Gym. Users can rotate by dragging, zoom by scrolling, inspect simulation, press Play, Reset, adjust Speed, and compare synchronized methods.

Painting

Room 1

WorldLab

Replica

Interactive 3D physics simulation in Isaac Gym — click any scene to view its simulation video. Use Play, Reset, Speed controls on the official project page.

REST3D highlights why baseline methods can explosively separate when gravity is applied.

Due to object interpenetration in baseline methods, applying gravity in a physics simulator can cause objects to explosively separate and become unstable. REST3D is built around the opposite expectation: reconstructed scenes should quickly settle into stable states.

REST3D results show high-resolution physics simulation of reconstructed scenes in Isaac Gym.

Objects are placed sequentially for clarity and then simulated jointly. REST3D reconstructed scenes are simulation-ready and quickly settle into stable states.

REST3D reconstructs an immersive and physically grounded 3D scene for VR hand-based interaction.

REST3D includes an interactive VR system that reconstructs an immersive, physically grounded 3D scene from a single image, enabling users to naturally interact with stable virtual objects through hand-based interactions. The demo was recorded with Meta Quest Pro and played back at 3× speed.

REST3D compares against DigitalCousins, Gen3DSR, SceneGen, and SAM3D.

The REST3D comparison focuses on physics simulation of reconstructed scenes in Isaac Gym. Existing methods struggle to balance reconstruction fidelity and physical stability, while REST3D produces stable, simulation-ready scenes that settle with only minor adjustments.

Ours — Simpson

DigitalCousins

Gen3DSR

SAM3D

SOTA comparison on Simpson Room — click each thumbnail to view the physics simulation video.

REST3D metric snapshot: low collision, high stability, low drift.

REST3D reports physical plausibility and geometric reconstruction quality.

REST3D differs from DigitalCousins by emphasizing input-faithful reconstruction plus physics stability.

DigitalCousins-style approaches can improve physical plausibility by retrieving and assembling 3D assets, but retrieval can be constrained by the asset database and may yield mismatched objects. REST3D instead uses image-to-3D priors and physics-constrained refinement to preserve visual consistency while reducing physical errors.

REST3D targets physical stability beyond divide-and-conquer scene reconstruction.

Gen3DSR is a strong single-view 3D scene reconstruction baseline. REST3D compares to Gen3DSR and focuses on the failure mode that matters in simulation: a scene can be reconstructed but still physically unstable under gravity.

REST3D prioritizes physical consistency with the observed image, while scene generation can trade accuracy for plausibility.

SceneGen-style methods synthesize multiple 3D assets and positions from a single scene image. REST3D argues that generation priors can be physically plausible yet inaccurate relative to the input. REST3D is framed as reconstruction: match the image and obey physics.

REST3D pushes beyond object-level reconstruction toward scene-level physical validity.

SAM3D can recover high-fidelity individual objects, but scene-level reconstruction also needs global orientation, wall attachment, support, collision handling, and stable contacts. REST3D explicitly focuses on those scene-level physical constraints.

REST3D use cases cluster around interactive 3D, VR, game content, robotics, and real-to-sim.

REST3D.org should make every high-intent tool one click away.

REST3D keyword clusters for headings, internal anchors, meta tags, and long-tail search coverage.

REST3D.org uses a Material Design 3 token-driven palette with M3 baseline colors, WCAG AAA contrast, and surface container hierarchy for spatial depth.

The REST3D audience expects a technical research interface, not a lifestyle landing page. This design uses Google Material Design 3 tokens (--md-sys-color-*) for all colors, M3 surface container hierarchy for depth, primary for interactive elements, tertiary for AI cues, and error/success/warning for semantic status. The page includes light/dark theme toggle, forced-colors support, and high-contrast text conforming to WCAG AAA standards for vision accessibility.

REST3D is strong, but not magic: VLM robustness and deformable objects remain future-work territory.

REST3D relies on the robustness of vision-language models for physical scene understanding and may fail in challenging cases. The current REST3D paper focuses on rigid objects and does not explicitly model deformable or non-rigid objects, leaving those cases for future work.

REST3D code repository exists, but the public README currently says code coming soon.

Cite REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image

@article{ma2026rest3d,
  title     = {REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image},
  author    = {Ma, Xiaoxuan and Wang, Jiashun and Ugrinovic, Nicol'{a}s and Litman, Yehonathan and Kitani, Kris},
  booktitle = {arXiv preprint arXiv:2605.30338},
  year      = {2026}
}

REST3D acknowledgement

The authors would like to thank Yuxuan Kuang, Yufei Wang, and Maxwell Jones for their insightful discussions.

REST3D FAQ for searchers, researchers, creators, and builders.

REST3D glossary: terms users search after they understand the headline.

REST3D primary links