Setting Up a 3DGS Simulation
Environment for Isaac Sim

01 The big picture — why a collider at all

3DGS is unbeatable at looking right. But that is the result of optimising to match photographs, not of getting the shape correct — and used raw in simulation, two failures follow.

First, floaters: gaussians left where nothing exists land in the robot's camera as view noise. Under this article's policy we accept them rather than delete them — deletion passes keep destroying real content (see the optional section). Second, the real wall, holes in collision: 3DGS has no surfaces, so collision comes from the scanner mesh, whose boundary-edge holes run to 4.86% indoors and 9.01% in Shibuya. Put a robot on a hole and this happens.

The architecture that fixes both is two layers: a floater-cleaned 3DGS for appearance, an invisible watertight collider for physics. NVIDIA×Niantic's pipeline of the same shape constrains geometry with AI-estimated depth — but that is what you do when a 360° camera is all you have. Holding measured LiDAR depth, you need neither the estimator nor any retraining: the five steps below build the same thing (quantitative comparison against learned methods is in the benchmark section).

02 STEP 1: Load the 3DGS into Isaac Sim

Convert the 3DGS PLY to USD (ParticleField) and load it — the basic conversion workflow is covered in a separate article. For a working setup, defuse two traps first.

• Align the coordinate frame. A Y-up 3DGS (e.g. from LCC2) just needs a -90° rotation about X after referencing — no data re-export required.
• Pin the renderer to RaytracedLighting before opening the stage. Path-tracing modes blow the self-emissive 3DGS out to white, and switching modes after a 10-million-gaussian scene is loaded is a reliable way to crash.

03 STEP 2: Patch only the ground holes in the mesh

Use the raw scanner mesh untouched as the collider. A voxel-healed watertight shell melts the geometry into wax, and patches cut from such a shell leave spiky fragments — both approaches lost. For thin poles and bollards alike, the raw mesh is the most accurate shape you have.

The only addition is ground patches: build a smooth ground field from the measured LiDAR surface and lay grid faces at field height only in cells where the raw mesh has no ground at all. Wall and facade holes never affect driving, so they stay. In Shibuya, nothing falls through anywhere in the drivable area. The holes a robot could fall through are sealed; not a single other triangle changes.

04 STEP 3: Stack the two layers and drive

Assemble one USD: define a physicsScene on a Z-up stage, reference the 3DGS file as-is and stand it up with a -90° X rotation (no baked re-export needed), embed the collider as a Mesh with CollisionAPI (approximation=none), and call MakeInvisible() on it. That completes the two-layer environment: untouched 3DGS for appearance, raw mesh + ground patches for physics.

For the vehicle, use PhysX Vehicle (omni.physx.vehicle). The wizard API (VehicleData + create_vehicle) generates a 4-wheel car with sprung suspension and Ackermann steering in a few lines, driven by accelerator (0–1) and steer (-1–1). The suspension absorbs the raw scan road's small bumps, so the ride is smooth without smoothing the data. Two traps: set rootVehiclePath and rootSharedPath on VehicleData explicitly, and register your collider in the generated GroundSurfaceCollisionGroup.

05 Optional: clean up with care

Deleting floaters and scanned-in pedestrians is possible, but deletion passes readily take real content — poles, signs, bollards — with them, so this procedure does not use any. If you must clean up: merge the raw LiDAR cloud into the reference surface, delete by silhouette rather than rectangles, and after every pass compare before/after renders from the same viewpoint and account for everything that disappeared. For collision and path-planning work, not deleting is the safest option.

06 Summary

Turning a LiDAR-fusion 3DGS scan into an Isaac Sim environment fits in three lines.

• STEP 1: reference the 3DGS untouched, stand it up with a -90° X rotation, and pin the renderer to RaytracedLighting before loading.
• STEP 2: collider = untouched raw mesh + ground-hole patches only. Wall holes stay.
• STEP 3: stack the two layers in one USD, make the collider invisible, and drive a PhysX Vehicle with real suspension. No deletion, no retraining.

Only two limits worth remembering: no surface exists where the LiDAR never looked, so keep routes inside the observed region; and under the untouched policy, scanned-in pedestrians and parked vehicles remain as real obstacles.

The basics of loading 3DGS into Isaac Sim are covered in a separate article.