Renderers¶
A family draws frames either with its upstream library's code or with one of the shared renderers in stable_envs.render. Each class table on a family page names the renderer it uses on CPU and on CUDA. Each section below takes one shared renderer and gives the families that use it and what it promises about backends, exactness and cost. The Scene2D IR gives the tensor layout of a 2D scene, and The compiled variation covers StyleTensor, the form of a variation that renderers read.
| Module | Draws | Used by | Devices |
|---|---|---|---|
scene, raster2d |
discs, convex polygons and capsules | SimplePointMaze; the navigation dot renderers reuse its tiling |
Torch on CPU, Triton on CUDA |
gfxdraw |
SDL_gfx primitives | classic control | Torch or Numba on CPU, Triton on CUDA |
raycast3d |
MuJoCo primitive scenes | OGBench mazes, fixed-camera DM Control tasks | Torch and Numba on CPU, Torch and Triton on CUDA |
cpu3d |
MuJoCo primitive scenes with a floor and a tracking camera | DM Control body-tracking tasks on CPU | Torch and Numba, CPU only |
nvdiff |
textured, lit triangle meshes | PyFlytRocketLanding |
CUDA only |
mujoco_raster |
MuJoCo scenes, as MuJoCo's OpenGL renderer draws them | OGBench Cube and Scene (backend="numba", the CPU default); Fetch render_semantic |
Numba, CPU only |
mjwarp |
MuJoCo models | DM Control body-tracking tasks on CUDA, Fetch on CUDA, OGBench Cube and Scene at supersample 2 or more |
CUDA only |
The other MuJoCo paths use MuJoCo's OpenGL renderer. The benchmarks measure the throughput of the ray caster and MJWarp.
2D scenes and rasterizer¶
stable_envs.render.scene defines Scene2D, a batch of discs, convex polygons and capsules in draw order, and SceneBuilder, which appends them. A family that draws in 2D returns a Scene2D from Env.scene, and stable_envs.render.raster2d draws it into (B, H, W, 3) uint8 frames.
import torch
from stable_envs.render.raster2d import render
from stable_envs.render.scene import SceneBuilder
b = SceneBuilder(batch=2, device="cpu")
b.add_rect(center=(0.5, 0.5), size=(0.6, 0.2), color=(40, 90, 200)) # shared by both rows
b.add_disc(center=torch.tensor([[0.3, 0.5], [0.7, 0.5]]), radius=0.1, # one center per row
color=(220, 30, 30))
scene = b.build(background=(255, 255, 255), world=(0.0, 0.0, 1.0, 1.0))
print(scene.batch, scene.n_prims) # 2 2
print(scene.kind.tolist()) # [[2, 1], [2, 1]]: a polygon, then a disc
frames = render(scene, out_size=64, supersample=2)
print(frames.shape, frames.dtype) # torch.Size([2, 64, 64, 3]) torch.uint8
print(frames[0, 32, 19].tolist(), frames[0, 2, 2].tolist()) # [220, 30, 30] [255, 255, 255]
Every SceneBuilder argument takes one value shared by the batch or one value per batch row. Coordinates are in the family's own world units, and world=(x0, y0, x1, y1) maps them onto the frame. Colors are RGB in 0 to 255. enabled turns a primitive off per row, and the scene keeps its shape when the number of visible objects changes. build raises ValueError when nothing was added. See The Scene2D IR for the tensor layout and the static-prefix cache that mark_static requests.
Primitives are composited in the order they were added. A static background is baked only when its geometry and appearance are the same for the whole batch. Output offsets are int64, which keeps the address calculation of large batches from overflowing. Outlines are unions of per-edge capsules with round corners, as in pygame's draw_fat_segment.
backend="auto" uses Triton when it's installed and the scene is on CUDA, and Torch otherwise. backend="triton" with a CPU scene raises ValueError. The Triton kernel computes each pixel's color in registers and writes only the result, while the Torch backend composites on a (B, 3, R, R) float canvas, one pass per primitive. Tests compare their pixels exactly. Both take out, a contiguous (B, H, W, 3) uint8 buffer that Env.render(out=...) passes through, and a training loop can reuse one buffer for every call. supersample=ss averages ss × ss subpixels per output pixel, which changes only edge pixels.
Building a Scene2D from state issues a few dozen small Torch operations from Python, and its cost changes little with batch size. The kernel is one launch that scales with output pixels. Below about a thousand environments, building the scene takes most of the time.
Batched 2-D scene description and the builder that fills it, consumed by raster2d.
Classes:
-
Scene2D–Batched 2-D primitives in draw order.
-
SceneBuilder–Append primitives in draw order and build a batched Scene2D.
Scene2D
dataclass
¶
Scene2D(kind: Tensor, geom: Tensor, fill: Tensor, line: Tensor, line_w: Tensor, background: Tensor, world: tuple[float, float, float, float] = (0.0, 0.0, 1.0, 1.0), flip_y: bool = False, static_prefix: int = 0)
Batched 2-D primitives in draw order.
world=(x0, y0, x1, y1) maps native coordinates onto the image and flip_y handles upward-positive
axes. KIND_NONE disables a primitive per row.
Methods:
-
to–Move arrays and optionally convert floating dtype, preserving kind dtype.
-
validate–Validate shapes, int32 kinds, and positive world extents; return self.
Attributes:
SceneBuilder
¶
SceneBuilder(batch: int, device: Any = 'cpu', vertices: int = 4, dtype: dtype = float32, style: Any = None)
Append primitives in draw order and build a batched Scene2D.
A width-sized value is shared, (B, width) ((B, 1) for scalars) is per environment, and plural add
methods also take (1, n, width) per-primitive values.
Methods:
-
add_disc–Add a filled disc with an optional concentric inner disc and independent color.
-
add_poly–Add a convex CCW polygon from
(V, 2)or(B, V, 2)vertices. -
add_segment–Add a capsule from endpoints and a half-width in world units.
-
add_rect–Add a rectangle with full width/height and optional rotation in radians.
-
add_segments–Add n capsules per row with endpoints (B, n, 2) and shared or per-primitive widths.
-
add_polys–Add convex CCW polygons (B, n, V, 2), rejecting excess vertices.
-
add_rects–Add rectangles with centers and full sizes (B, n, 2), optionally rotated per rectangle.
-
mark_static–Offer the primitives appended so far as a state-independent prefix for caching.
-
build–Build a validated nonempty scene.
add_disc
¶
add_disc(center: Batched, radius: Batched, color: Batched, inner_radius: Batched = 0.0, inner_color: Batched | None = None, enabled: Batched | None = None) -> None
Add a filled disc with an optional concentric inner disc and independent color.
Parameters:
-
center(Batched) –Center
(x, y)in world units. -
radius(Batched) –Radius in world units.
-
color(Batched) –RGB in 0 to 255.
-
inner_radius(Batched, default:0.0) –Radius of the inner disc (none if 0).
-
inner_color(Batched | None, default:None) –RGB of the inner disc.
-
enabled(Batched | None, default:None) –Whether each row draws the disc (all rows if None).
add_poly
¶
add_poly(verts: Batched, color: Batched, outline_color: Batched | None = None, outline_w: Batched = 0.0, enabled: Batched | None = None) -> None
Add a convex CCW polygon from (V, 2) or (B, V, 2) vertices.
Repeat the last vertex when padding; exceeding vertex capacity raises ValueError.
Parameters:
-
verts(Batched) –Vertices in world units, counterclockwise.
-
color(Batched) –Fill RGB in 0 to 255.
-
outline_color(Batched | None, default:None) –Outline RGB (no outline if None).
-
outline_w(Batched, default:0.0) –Outline width in world units.
-
enabled(Batched | None, default:None) –Whether each row draws the polygon (all rows if None).
add_segment
¶
add_segment(a: Batched, b: Batched, half_w: Batched, color: Batched, enabled: Batched | None = None) -> None
Add a capsule from endpoints and a half-width in world units.
Parameters:
-
a(Batched) –First endpoint
(x, y). -
b(Batched) –Second endpoint
(x, y). -
half_w(Batched) –Half the capsule width.
-
color(Batched) –RGB in 0 to 255.
-
enabled(Batched | None, default:None) –Whether each row draws the capsule (all rows if None).
add_rect
¶
add_rect(center: Batched, size: Batched, color: Batched, angle: Batched | None = None, outline_color: Batched | None = None, outline_w: Batched = 0.0, enabled: Batched | None = None) -> None
Add a rectangle with full width/height and optional rotation in radians.
Parameters:
-
center(Batched) –Center
(x, y)in world units. -
size(Batched) –Full
(width, height). -
color(Batched) –Fill RGB in 0 to 255.
-
angle(Batched | None, default:None) –Rotation in radians (none if None).
-
outline_color(Batched | None, default:None) –Outline RGB (no outline if None).
-
outline_w(Batched, default:0.0) –Outline width in world units.
-
enabled(Batched | None, default:None) –Whether each row draws the rectangle (all rows if None).
add_segments
¶
add_segments(a: Tensor, b: Tensor, half_w: Batched, color: Batched, enabled: Batched | None = None) -> None
Add n capsules per row with endpoints (B, n, 2) and shared or per-primitive widths.
add_polys
¶
add_polys(verts: Tensor, color: Batched, outline_color: Batched | None = None, outline_w: Batched = 0.0, enabled: Batched | None = None) -> None
Add convex CCW polygons (B, n, V, 2), rejecting excess vertices.
add_rects
¶
add_rects(center: Tensor, size: Tensor, color: Batched, angle: Batched | None = None, outline_color: Batched | None = None, outline_w: Batched = 0.0, enabled: Batched | None = None) -> None
Add rectangles with centers and full sizes (B, n, 2), optionally rotated per rectangle.
mark_static
¶
mark_static() -> SceneBuilder
Offer the primitives appended so far as a state-independent prefix for caching.
The rasterizer bakes the prefix only if its values are uniform across the batch.
Rasterize Scene2D discs, convex polygons, and capsules with Torch or a fused Triton kernel.
Functions:
-
render–Render uint8 RGB frames with Torch or Triton on the scene's device.
-
render_torch–Rasterize a Scene2D into
(B, H, W, 3)uint8 frames on the scene's device. -
render_triton–Rasterize a CUDA Scene2D into
(B, H, W, 3)uint8 frames. -
row_groups–Partition
(B, 4)pixel boxes into row groups with smaller union windows. -
tile_shape–Choose a
(height, width)tile within theinstancesandmax_widthbudgets.
render
¶
render(scene: Scene2D, out_size: int | tuple[int, int], supersample: int = 1, backend: str = 'auto', out: Tensor | None = None) -> Tensor
Render uint8 RGB frames with Torch or Triton on the scene's device.
Parameters:
-
scene(Scene2D) –The scene to draw.
-
out_size(int | tuple[int, int]) –Output size, an int or
(height, width). -
supersample(int, default:1) –Subpixels per output pixel along each axis.
-
backend(str, default:'auto') –"auto"(Triton for CUDA scenes when it's installed),"triton"or"torch". -
out(Tensor | None, default:None) –A contiguous
(B, H, W, 3)uint8 buffer to fill.
Returns:
-
Tensor–(B, H, W, 3)uint8 frames.
render_torch
¶
render_torch(scene: Scene2D, out_size: int | tuple[int, int], supersample: int = 1) -> Tensor
Rasterize a Scene2D into (B, H, W, 3) uint8 frames on the scene's device.
render_triton
¶
render_triton(scene: Scene2D, out_size: int | tuple[int, int], supersample: int = 1, out: Tensor | None = None, block: int | None = None, cull: bool | None = None) -> Tensor
Rasterize a CUDA Scene2D into (B, H, W, 3) uint8 frames.
block defaults to the raster2d_block setting and cull to a choice by primitive count. out must be
contiguous uint8 storage with exactly B*H*W*3 elements.
The Scene2D IR¶
Scene2D is the interface between a 2D environment and the rasterizer. The environment turns (state, variation) into a Scene2D, and one kernel turns that into pixels. The kernel sees only the scene, never an environment handle, simulator or Python callback.
from stable_envs.render import raster2d
from stable_envs.render.scene import SceneBuilder
agent = torch.tensor([[256.0, 256.0], [100.0, 400.0]]) # one position per batch row
sb = SceneBuilder(batch=2, device="cpu", vertices=4)
sb.add_segment((5, 506), (5, 5), 2.5, (0, 0, 0)) # a wall, shared by the batch
sb.mark_static() # request a bake of everything above
sb.add_disc(agent, 15.0, (65, 105, 225)) # (B, 2) centers, one per row
scene = sb.build(background=(255, 255, 255), world=(0, 0, 512, 512))
frames = raster2d.render_torch(scene, 96)
print(frames.shape, frames.dtype) # torch.Size([2, 96, 96, 3]) torch.uint8
The IR has three primitive kinds, discs, convex polygons and capsules, composited back to front with flat colors and packed into fixed-width tensors:
| Field | Shape | Meaning |
|---|---|---|
kind |
(B, N) int32 |
0 disabled, 1 disc, 2 convex polygon, 3 capsule |
geom |
(B, N, 2V) f32 |
disc: [cx, cy, r, inner_r, ...]. Polygon: [x0, y0, ... xV-1, yV-1] CCW. Capsule: [x0, y0, x1, y1, half_w, ...] |
fill |
(B, N, 3) f32 |
interior color, 0–255 |
line |
(B, N, 3) f32 |
outline color, also the disc's inner color |
line_w |
(B, N) f32 |
outline half-width in world units, where 0 disables it |
background |
(B, 3) f32 |
per-environment background color |
world |
4 floats | (x0, y0, x1, y1), the extent of the primitives |
flip_y |
bool | set when the environment's +y points up |
static_prefix |
int | number of leading primitives the environment asks to bake |
kind == 0disables a slot. A batch is one rectangular tensor: a scene whose primitive count depends on the state disables slots instead of changing shape, which allows CUDA graph capture and compile-time unrolling.- Draw order is index order, back to front, with no depth buffer.
- Padding repeats the last vertex of a polygon. That makes a zero-length edge, which both backends drop from the signed distance (edges with
L2 < 1e-12), and the outline stays correct.
Each environment keeps its own units, such as PushT's 512-unit board or CartPole's meters, and the rasterizer maps world onto the output image. Set flip_y instead of flipping coordinates in environment code. add_disc takes an optional inner_radius and inner_color, add_poly takes (B, V, 2) world vertices with an optional outline_color and outline_w, and add_rect wraps add_poly with an optional rotation. build() runs Scene2D.validate() once, which checks each tensor's shape, the int32 dtype of kind, and that world isn't degenerate.
Static prefix¶
SceneBuilder.mark_static() asks the rasterizer to draw the primitives added so far once into a cached base image that later frames start from. The whole batch shares the base image. It bakes only if the prefix is identical across batch rows, which one kernel checks against row 0, and otherwise draws every primitive per frame. The output is the same either way.
The cache key holds the exact bytes of the static primitives and the background color, the supersampled resolution, world, flip_y, the vertex count, the prefix length and the device. A variation that recolors a static primitive or the background therefore invalidates the bake. The rasterizer skips the bake when K/N × B × H × W (K static primitives out of N, at the supersampled resolution) is below raster2d._BAKE_MIN_WORK = 8e6 evaluations, and then no comparison or transfer happens. An environment that passes its StyleTensor to the builder (SceneBuilder(B, ..., style=sty)) also skips the comparison when the style differs per batch row.
Baking saves more work as the batch and the resolution grow. Below the work threshold the rasterizer does not bake, because the bake would cost more than it saves. The public PushT classes do not use this cache: they draw with their own Numba (CPU) and Triton (CUDA) rasterizers.
gfxdraw¶
stable_envs.render.gfxdraw is a batched Torch version of the SDL_gfx primitives behind pygame.gfxdraw, and classic control frames match Gymnasium's pygame drawing. On CUDA the Triton kernels in stable_envs.render.gfxdraw.fused draw, and on CPU a Numba twin in stable_envs.render.cpu_raster.classic draws when Numba is installed (classic extra), with the Torch functions as the fallback. With backend="auto", some CPU batches with the default appearance use Gymnasium's own renderer instead, and every choice gives the same frames.
It's separate from raster2d: SDL_gfx works in integers and takes Sint16 vertices. The module computes geometry in float64 and truncates every coordinate before drawing. Scanline fills, circle spans, clipping and packed-channel blending keep SDL's integer operation order, and the CUDA path repeats it. The conformance comparison found 0 differing pixels out of 250.9 M.
Batched SDL_gfx-style integer rasterization with Torch.
Compute geometry in float64 before truncating it, because float32 rounding can move a whole pixel row.
Classes:
-
Gfx–Cache drawing constants for one canvas size and device.
-
StyleCanvas–Canvas with per-environment color, antialiasing and visibility gates.
Functions:
-
polygon–Draw a convex polygon with an antialiased fringe and integer scanline fill.
-
circle–Draw ordered antialiased fringe passes, then the filled circle.
-
aaline–Clip and draw an antialiased edge with a solid initial pixel and no final endpoint.
-
fill_polygon_mask–Return a
(B, H, W)scanline mask for convex integer polygons; seepolygon_spans. -
fill_circle_mask–Return a (B, H, W) circle mask for integer (B,) centers and a cached span table.
-
blend–Apply SDL_gfx packed-channel alpha blending with unsigned 32-bit wrapping.
-
blit_over–Blend a per-pixel-alpha surface onto opaque RGB using SDL_BlitSurface arithmetic.
-
rotate–Apply pygame's
Vector2.rotate_radarithmetic before the canvas's final vertical flip. -
to_u8–Truncate style colors to uint8 using the reference's integer conversion rule.
Gfx
¶
Gfx(H: int, W: int, device)
Cache drawing constants for one canvas size and device.
Warm it up before CUDA graph capture, because creating its tensors during capture is invalid.
Methods:
StyleCanvas
¶
StyleCanvas(gfx: Gfx, B: int, bg)
Bases: Canvas
Canvas with per-environment color, antialiasing and visibility gates.
use selects (B, 3) colors and gated restricts the primitives that follow.
Methods:
-
use–Set (B, 3) colors for subsequent primitives and return this canvas.
-
gated–Set a (B,) boolean visibility gate for subsequent primitives and return this canvas.
-
paint–Fill a window-sized mask using per-environment colors and the current visibility gate.
-
paint_spans–Fill the inclusive (B, rows) column spans of the window's rows with per-environment colors.
gated
¶
gated(gate) -> StyleCanvas
Set a (B,) boolean visibility gate for subsequent primitives and return this canvas.
paint
¶
paint(mask, color=None, window=None) -> None
Fill a window-sized mask using per-environment colors and the current visibility gate.
paint_spans
¶
paint_spans(xa, xb, window=None, color=None) -> None
Fill the inclusive (B, rows) column spans of the window's rows with per-environment colors.
Only the span pixels are written, so a thin polygon costs its area rather than its window.
polygon
¶
polygon(cv: Canvas, verts, color, color_t, bidx, kmax, aa: bool = True, fill_first: bool = False, window=None) -> None
Draw a convex polygon with an antialiased fringe and integer scanline fill.
verts should be float64 and window must contain the polygon footprint. fill_first=True fills
before the fringe, which changes edge blending.
circle
¶
circle(cv: Canvas, cx, cy, r, color, color_t, bidx, hw_t, stencil, aa: bool = True, window=None, disc=None, instances=None) -> None
Draw ordered antialiased fringe passes, then the filled circle.
disc is a filled-disc stencil (hw_t and window are used otherwise). With instances, the
centers belong to those environments only.
aaline
¶
aaline(cv: Canvas, x1, y1, x2, y2, color_t, kmax, bidx) -> None
Clip and draw an antialiased edge with a solid initial pixel and no final endpoint.
fill_polygon_mask
¶
fill_polygon_mask(vx: Tensor, vy: Tensor, ys: Tensor, xs: Tensor, W: int) -> Tensor
Return a (B, H, W) scanline mask for convex integer polygons; see polygon_spans.
xs selects columns.
fill_circle_mask
¶
fill_circle_mask(cx: Tensor, cy: Tensor, r: int, ys: Tensor, xs: Tensor, hw_t: Tensor) -> Tensor
Return a (B, H, W) circle mask for integer (B,) centers and a cached span table.
blend
¶
blend(src: Tensor, dst: Tensor, w: Tensor, sh: Tensor) -> Tensor
Apply SDL_gfx packed-channel alpha blending with unsigned 32-bit wrapping.
Weight 256 is an opaque write; weight 255 still blends.
blit_over
¶
blit_over(dst: Tensor, src_rgb: Tensor, alpha: Tensor) -> Tensor
Blend a per-pixel-alpha surface onto opaque RGB using SDL_BlitSurface arithmetic.
This uses alpha directly; SDL_gfx's packed-channel pixel blend uses another rule.
Ray casting and CPU3D¶
stable_envs.render.raycast3d intersects rays with spheres, capsules, cylinders, ellipsoids, boxes and finite planes. stable_envs.render.cpu3d uses the same intersections to draw a whole primitive scene on CPU when a moving camera rules out a static background, adding sky and floor textures, directional shadows and specular light. Its constructor rejects unsupported geometry and shadow-floor setups.
Fixed-camera rendering bakes a static RGB and depth plate, casts the moving primitives inside conservative screen bounds, and shades only the covered samples. Three MuJoCo conventions decide silhouette and highlight pixels: the mono camera averages its two stereo eyes, the depth buffer measures perpendicular distance while the compositor compares radial ray parameters, and specular light is added before texture modulation.
backend="auto" picks Triton for packed float32 inputs on CUDA and Torch otherwise. An explicit "triton" needs CUDA float32 inputs on one device with packed trailing axes and raises ValueError otherwise. The Torch cast takes bounds=False to cast every pixel, for diagnostics. In tests, override the backend in a scoped fixture rather than with a process-wide switch. On CPU, DM Control uses Numba kernels when Numba imports: cpu_raster.rays3d repeats the float32 operation order of the Torch intersectors and tests each tile of rays only against primitives its rays can reach, cpu_raster.suite3d composites the fixed-camera plate, and cpu_raster.scene3d draws a CPU3D scene. They use the IEEE square root, while Torch's CPU square root can round through MKL. Rare pixels differ from the Torch backend as a result.
Tests require exact agreement between the optimized compositor and its full-window reference, between the Torch and Triton casts on float32 inputs, and between bounded and unbounded Torch casts. Agreement with the upstream library's OpenGL frames is measured per class.
Batched analytic rendering of MuJoCo primitive geometry.
Torch and Triton share intersection and shading conventions. Moving primitives are composited over a fixed RGB/depth plate; compact windows avoid casting the static background again. Float operation order is part of the pixel contract.
Classes:
-
Camera–Pinhole camera with the near-plane frustum reported by
mjvGLCamera. -
Lights–Directional lights with unit directions toward each light and scalar intensity weights.
Functions:
-
intersect–Return ray parameters and world-space normals, with
t=INFon misses. -
cast_nearest–Return (ray parameter, world normal, primitive index); see
cast_nearest_torch. -
cast_nearest_torch–Return nearest ray parameters
(B, N), normals(B, N, 3)and indices(B, N). -
any_blocking–Return a
(B, N)mask of hits before lim, excluding each ray's optional skip index. -
any_blocking_torch–Return a
(B, N)mask of hits with0 < t < lim, excluding each ray's skip index. -
cached_rays–Return cached camera rays, keyed by camera, resolution, device, and dtype.
-
shade–Return RGB
(..., 3)in [0, 1]; white specular is added before texture modulation. -
static_depth–Return nearest static-surface ray parameters
(H*W,), in radial units rather than MuJoCo depth. -
composite–Draw moving primitives over a static plate and return
(B, H, W, 3)uint8 frames.
Camera
dataclass
¶
Camera(pos: tuple, forward: tuple, up: tuple, near: float, top: float, bottom: float, center: float = 0.0)
Pinhole camera with the near-plane frustum reported by mjvGLCamera.
Keeping the frustum supports off-center and stereo cameras.
Methods:
Lights
dataclass
¶
Lights(dirs: ndarray, ambient: ndarray, diffuse: ndarray, specular: ndarray, exponent: float)
Directional lights with unit directions toward each light and scalar intensity weights.
intersect
¶
intersect(kind: int, size: Tensor, pos: Tensor, R: Tensor, o: Tensor, d: Tensor) -> tuple[Tensor, Tensor]
Return ray parameters and world-space normals, with t=INF on misses.
Poses (..., 3) and rotations (..., 3, 3) broadcast against rays (..., 3). Unsupported MuJoCo
geometry kinds raise ValueError.
cast_nearest
¶
cast_nearest(kinds: Kinds, sizes: Tensor, pos: Tensor, R: Tensor, o: Tensor, d: Tensor, backend: str = 'auto') -> Cast
Return (ray parameter, world normal, primitive index); see cast_nearest_torch.
auto uses Triton for packed float32 inputs on CUDA, otherwise Torch.
An unknown backend or unsupported explicit triton request raises ValueError.
cast_nearest_torch
¶
cast_nearest_torch(kinds: Kinds, sizes: Tensor, pos: Tensor, R: Tensor, o: Tensor, d: Tensor, *, bounds: bool = True) -> Cast
Return nearest ray parameters (B, N), normals (B, N, 3) and indices (B, N).
kinds holds K host codes; sizes and positions are (B, K, 3), rotations (B, K, 3, 3) and rays
(B, N, 3). Misses have t=INF.
any_blocking
¶
any_blocking(kinds: Kinds, sizes: Tensor, pos: Tensor, R: Tensor, o: Tensor, d: Tensor, lim: Tensor, skip: Tensor | None = None, backend: str = 'auto') -> Tensor
Return a (B, N) mask of hits before lim, excluding each ray's optional skip index.
Backend selection and errors follow cast_nearest.
any_blocking_torch
¶
any_blocking_torch(kinds: Kinds, sizes: Tensor, pos: Tensor, R: Tensor, o: Tensor, d: Tensor, lim: Tensor, skip: Tensor | None = None, *, bounds: bool = True) -> Tensor
Return a (B, N) mask of hits with 0 < t < lim, excluding each ray's skip index.
Inputs follow cast_nearest_torch; lim and skip are (B, N).
cached_rays
¶
cached_rays(cam: Camera, H: int, W: int, device, dtype=float32) -> tuple[Tensor, Tensor]
Return cached camera rays, keyed by camera, resolution, device, and dtype.
shade
¶
shade(n: Tensor, v: Tensor, ldir: Tensor, lamb: Tensor, ldif: Tensor, lspec: Tensor, exponent: float, mat: Tensor, tex: Tensor, spec: Tensor) -> Tensor
Return RGB (..., 3) in [0, 1]; white specular is added before texture modulation.
static_depth
¶
static_depth(cam: Camera, prims: Primitives, xpos, xmat, H: int, W: int, device) -> Tensor
Return nearest static-surface ray parameters (H*W,), in radial units rather than MuJoCo depth.
composite
¶
composite(plate: Tensor, plate_t: Tensor, cam: Camera, prims: Primitives, xpos: Tensor, xmat: Tensor, lights: Lights, H: int, W: int, supersample: int = 1, out: Tensor | None = None, tile: int | None = None, mat_rgb: Tensor | None = None, size_scale: Tensor | None = None, shader=None, cast: Sequence[int] | None = None) -> Tensor
Draw moving primitives over a static plate and return (B, H, W, 3) uint8 frames.
plate is (H, W, 3) or (B, H, W, 3) uint8 and plate_t its (H*W,) radial depth. Poses are
(B, P, 3) positions and (B, P, 3, 3) rotations; mat_rgb (B, P, 3) and size_scale (B,)
override appearance per environment. shader(rows, hit, normal, view, prim, mat) replaces lights and
returns colors in [0, 255]. cast limits the cast to these primitive indices, and the others must not
cover any sample these can cover.
nvdiffrast¶
stable_envs.render.nvdiff rasterizes textured, lit triangle meshes on CUDA with nvdiffrast and returns uint8 frames. The PyFlyt rocket reuses its scene and shading code with its own rasterizer, which follows Bullet's triangle coverage rules. DM Control, Fetch and CPU3D borrow its material and texture helpers, effective_rgba and texture_array, which run without nvdiffrast. nvdiffrast is installed from source.
Projection uses a fixed arithmetic order, and every batch size takes the same matrix-multiply rounding path. Texture origin, row direction and byte rounding are explicit. The backend copies MuJoCo view arrays before later simulator calls can change them. Mipmapping and optional anisotropic filtering control texture footprints, with one tap by default, and antialiasing and supersampling control edge coverage. Each setting has its own cost and needs its own source-image check. The output isn't bit-exact with upstream OpenGL or TinyRenderer images. Exact tests check only that frames stay the same across batch sizes. On x86 CPU, nvdiffrast has a one-ULP batch dependence in a sparse stage, and no registered class uses that path on CPU.
Batched nvdiffrast rendering of MuJoCo geometry with MuJoCo-style lighting, textures, and skybox.
Classes:
-
Scene–Static triangle geometry, appearance and textures on one device.
-
BatchRenderer–Render geom transforms to
(B, H, W, 3)uint8 frames with nvdiffrast on CUDA. -
Camera–Pinhole camera looking along local -z, with matrix columns right, up and backward.
-
Lights–Resolved
mjvScenelights, including the camera headlight; directions point away from the lights. -
TinyShading–Bullet TinyRenderer lighting parameters for a z-up world;
dirpoints toward the light.
Functions:
-
effective_rgba–Resolve geom RGBA, using its material only when geom RGBA matches the compiler default.
-
texture_array–Read texture
texof a compiled model as an(H, W, 3)uint8 array. -
close_cached_renderer–Close the renderer held by a family cache entry before eviction.
Scene
¶
Scene(m, groups=DEFAULT_GROUPS, device='cuda', nslice: int = NUMSLICES, nstack: int = NUMSTACKS, skip_alpha_zero: bool = True, tex_policy: str = 'pot')
Bases: Closeable
Static triangle geometry, appearance and textures on one device.
A renderer given this Scene borrows it by default, so close the Scene only after every borrower is done.
Methods:
-
from_triangles–Build a Scene from triangles in each geom's local frame.
-
clip–Return clip vertices
(B, V, 4), world normals(B, V, 3), and world positions.
from_triangles
classmethod
¶
from_triangles(vert, norm, face, vert_geom, rgba, *, uv=None, specular=None, shininess=None, emission=None, textures=None, tex_of_geom=None, device='cuda', tex_policy: str = 'pot') -> Scene
Build a Scene from triangles in each geom's local frame.
Inputs are vert/norm (V, 3), face (F, 3), vert_geom (V,) and rgba (G, 4). Every face lies in
one geom, tex_of_geom is -1 for untextured geoms, and render transforms use the same geom order.
clip
¶
clip(xpos, xmat, view, proj)
Return clip vertices (B, V, 4), world normals (B, V, 3), and world positions.
BatchRenderer
¶
BatchRenderer(m=None, res: int = 224, groups=DEFAULT_GROUPS, device: str = 'cuda', textures: bool = True, mipmap: bool = True, skybox: bool = True, quant: str = 'round', nslice: int = NUMSLICES, nstack: int = NUMSTACKS, supersample: int = 1, tex_policy: str = 'pot', aniso: int = 1, *, scene: Scene | None = None, shading: str = 'mujoco', tiny: TinyShading | None = None, tex_filter: str = 'linear', sample_offset=(0.0, 0.0))
Bases: Closeable
Render geom transforms to (B, H, W, 3) uint8 frames with nvdiffrast on CUDA.
Close it or use it as a context manager. A scene= argument is borrowed unless owns_scene=True.
shading="tiny" uses TinyShading parameters and ignores scene lights.
sample_offset shifts pixel coverage; tex_filter selects linear or
nearest sampling. Texture mipmapping and anisotropy are configured separately.
Methods:
-
set_skybox–Prepare the model's skybox texture when present.
-
resolve_from_mujoco–Resolve lights and camera through mjv_updateScene, including runtime defaults and the headlight.
-
render–Render device-resident
xpos(B, G, 3)andxmat(B, G, 3, 3)to uint8 RGB.
resolve_from_mujoco
¶
resolve_from_mujoco(m, d, camera=None, option=None)
Resolve lights and camera through mjv_updateScene, including runtime defaults and the headlight.
Camera
dataclass
¶
Camera(pos: ndarray, mat: ndarray, fovy: float, znear: float, zfar: float)
Pinhole camera looking along local -z, with matrix columns right, up and backward.
Position and orientation may carry a leading batch dimension.
Methods:
-
from_glcam–Build a mono camera by averaging the two resolved eyes, canceling their IPD offsets.
-
matrices–Return OpenGL view (..., 4, 4) and projection (4, 4) matrices.
Lights
dataclass
¶
Lights(pos: ndarray, dir: ndarray, directional: ndarray, ambient: ndarray, diffuse: ndarray, specular: ndarray, attenuation: ndarray, cutoff: ndarray, exponent: ndarray, headlight: ndarray)
Resolved mjvScene lights, including the camera headlight; directions point away from the lights.
TinyShading
dataclass
¶
TinyShading(dir: tuple = (-50.0, 30.0, 100.0), ambient: float = 0.6, diffuse: float = 0.35, specular: float = 0.05, exponent: float = 2.0, color: tuple = (1.0, 1.0, 1.0))
Bullet TinyRenderer lighting parameters for a z-up world; dir points toward the light.
effective_rgba
¶
effective_rgba(m, g: int) -> ndarray
Resolve geom RGBA, using its material only when geom RGBA matches the compiler default.
texture_array
¶
texture_array(m, tex: int) -> ndarray
Read texture tex of a compiled model as an (H, W, 3) uint8 array.
close_cached_renderer
¶
close_cached_renderer(c) -> None
Close the renderer held by a family cache entry before eviction.
MJWarp¶
stable_envs.render.mjwarp renders a batch of MuJoCo configurations on CUDA with MuJoCo Warp, each family with its own model preparation, static background and material handling. Model preparation selects the visible geom groups, moves invisible geometry out of them, sets shadow casting and records texture mappings MuJoCo Warp can't sample. The backend converts MuJoCo Warp's packed BGRA output to RGB, and tracking cameras need center-of-mass data.
- GL-compatible rounding must be active when the backend captures the render graph. A captured graph keeps the pack function it was captured with.
- Building a render context patches MuJoCo Warp's texture-upload and byte-packing helpers for the duration of the call. These patches aren't thread-safe: build contexts only while no other thread uses those helpers.
- Importing the module sets
WARP_CACHE_PATHto~/.cache/warpwhen it's unset. - The light specialization applies only to the MuJoCo Warp version and model it supports, and other configurations use stock MuJoCo Warp.
- Exact tests compare the backend with itself across changes, and texture filtering, coverage and shading leave a per-class MSE against the upstream library's OpenGL frames.
See OGBench for how Cube and Scene choose between the MuJoCo rasterizer, OpenGL and MJWarp.
MJWarp model preparation, texture uploads and batched CUDA rendering.
Importing it sets WARP_CACHE_PATH to ~/.cache/warp unless you've set it.
Classes:
-
BatchRenderer–Render
(B, nq)joint positions to(B, H, W, 3)uint8 RGB on CUDA. -
Prepared–Model edits and saved arrays needed to undo them.
Functions:
-
prepare_model–Prepare a model for MJWarp in place and return restoration data.
-
create_render_context–Create an MJWarp context with one upload of the model's texture blob.
-
release_render_context–Drop texture tensor keepalives attached to an MJWarp render context; repeated calls are safe.
-
release_cached_ctx–Drop texture keepalives from a cached family context before eviction.
-
gl_rounding–Temporarily round RGB channels before MJWarp packs them to uint8.
-
needs_com_pos–Return whether any camera or light tracks or targets a subtree center of mass.
BatchRenderer
¶
BatchRenderer(m, batch: int, res: int = 224, camera: int = 0, groups=DEFAULT_GROUPS, textures: bool = True, shadows: bool = True, skybox: bool = True, prepare: bool = True, capture: bool = True)
Bases: Closeable
Render (B, nq) joint positions to (B, H, W, 3) uint8 RGB on CUDA.
Close it or use it as a context manager. With prepare=True the caller's MuJoCo model stays modified;
restore_model with prepared undoes that.
Methods:
-
render–Render device-resident
(B, nq)positions and return uint8 RGB frames.
Prepared
dataclass
¶
Prepared(hidden_geoms: list[int], hidden_sites: list[int], groups: tuple[int, ...], restore: dict, neutralised_spots: list[int] = list())
prepare_model
¶
prepare_model(m, groups=DEFAULT_GROUPS, alpha_eps: float = 0.0, spot_semantics: bool = True) -> Prepared
Prepare a model for MJWarp in place and return restoration data.
Geoms and sites with alpha at most alpha_eps move to HIDDEN_GROUP, which the caller must not render.
create_render_context
¶
create_render_context(m, tex_lod: int = 1, **kw)
Create an MJWarp context with one upload of the model's texture blob.
tex_lod > 1 area-downsamples textures before upload, which is a visual approximation. Don't create
contexts from several threads at once, because this patches a process-global MJWarp helper.
release_render_context
¶
release_render_context(rc) -> None
Drop texture tensor keepalives attached to an MJWarp render context; repeated calls are safe.
The caller must also drop the context to release its device arrays.
release_cached_ctx
¶
release_cached_ctx(c) -> None
Drop texture keepalives from a cached family context before eviction.
gl_rounding
¶
gl_rounding()
Temporarily round RGB channels before MJWarp packs them to uint8.
Enclose graph capture as well as warmup. This patches process-global MJWarp internals, so don't render concurrently.
MuJoCo rasterizer¶
stable_envs.render.mujoco_raster draws MuJoCo scenes on CPU with a Numba rasterizer that follows MuJoCo's fixed-function OpenGL renderer (render_gl3.c). It reads one scene model and one set of per-frame inputs. It is the default CPU renderer of the OGBench Cube and Scene tasks (backend="numba"), and Fetch uses it for render_semantic only. backend="opengl" stays the exact reference. On CUDA, the Cube and Scene backend="numba" raises ValueError.
Scene model¶
stable_envs.render.mujoco_scene.extract reads one mjv_updateScene of a model. It gives the drawn geoms with the triangles of their display lists, the materials and texture mappings, the lights, the camera, the textures with their mipmaps, the skybox and the shadow-map settings. mujoco_scene.pack turns the scene into flat arrays: corners, triangles, materials with a table of specular powers, textures, light rows, the camera row and the shadow matrices.
The extraction records the MuJoCo minor version. mujoco_scene.VERIFIED lists the scene features that were compared with the OpenGL renderer of each version. A scene that needs another feature raises NotImplementedError. Fetch uses MuJoCo 3.11. Cube and Scene use MuJoCo 3.17.
Per-frame inputs¶
A family gives these arrays for each frame:
| Input | Shape | Content |
|---|---|---|
gpos |
(B, G, 3) | Geom positions |
gmat |
(B, G, 9) | Geom rotations, row major |
rgba |
(B, G, 4) | Geom colors; an alpha below 0.995 makes the geom translucent in that frame |
cams |
(B, 12) | Camera position, right, up and forward |
lights |
(B, L, 23) | Light rows; mujoco_scene.headlight_rows points the headlight along each camera |
sky |
(B, 3) | A flat sky color on the 0-255 scale, or -1 for the skybox |
Rendering contract¶
- Lighting is per vertex, in float64, with a local viewer, the headlight and the model lights.
- Each frame has 4 samples per pixel at the sample positions of Mesa, shaded at the pixel center.
- A shadow light adds its light where the shadow map has no nearer caster. The static geoms make one map per scene; the moving geoms make a map per frame.
- Translucent geoms blend after the opaque geoms and the skybox, sorted by distance to the camera less their bounding radius, once front to back and once back to front.
- Triangles are clipped at the near plane.
- A draw of the same inputs gives the same frame each time, alone or in any batch.
Plates¶
Frames with the same camera, light rows, sky color and colors of the static geoms share a plate. The plate holds the static opaque geoms, their shadows and the skybox. Each frame starts from its plate and draws the other geoms.
Segmentation and depth¶
mujoco_raster.semantic gives the geom of each pixel center and its depth along the optical axis, as the segmentation and depth rendering of MuJoCo give them. A pixel shows the nearest opaque geom, or a translucent geom in front of it. Fetch and Cube/Scene give render_semantic(rows): the segmentation as (objid, objtype) pairs in the model of the reference environment, and the depth in model units.
The compiled variation¶
stable_envs._core.style.StyleTensor is the compiled form of variation: one flat (B, variation_dim) float32 tensor plus a layout that maps columns to leaves such as cart.color. Application code uses variation_spec, default_variation, sample_variation and render(..., variation=) instead.
import stable_envs as snv
from stable_envs._core.style import StyleTensor
with snv.CartPole(num_envs=2, device="cpu") as env:
style = StyleTensor.from_space(env.variation_space, env.num_envs, device=env.device)
print(style.values.shape, style.values.dtype) # torch.Size([2, 22]) torch.float32
print(style.get("cart.color").shape) # torch.Size([2, 3])
print(style.for_batch(4).values.shape) # torch.Size([4, 22])
StyleTensor.for_batch is the only operation that may change the batch size of a compiled row. A shared style broadcasts its one row to the new batch size. A style whose rows differ raises ValueError, since nothing says which rows the smaller batch should keep. Slice the style together with the states instead.
Compiled appearance rows: a (B, style_dim) float32 tensor addressed by variation path.
StyleTensor.for_batch is the only operation that changes the batch size of a compiled row.
Classes:
-
StyleTensor–Appearance values of shape
(B, style_dim), indexed by variation path.
Functions:
-
rows_are_equal–Test equality across batch rows, reducing all comparisons before one host read.
StyleTensor
¶
StyleTensor(values: Tensor, layout: list[tuple[str, int, int]], shared: bool | None = None)
Appearance values of shape (B, style_dim), indexed by variation path.
layout lists (path, offset, width) for each leaf. shared records whether all
rows are known to be equal; None defers the check until :attr:shared is read.
Methods:
-
get–Return the leaf view for
path, ordefaultwhen the layout has no such leaf. -
to–Return a copy on
devicewithdtype, keeping the layout and known sharedness. -
for_batch–Return this style at
batchrows. -
from_space–Build float32 appearance rows from space defaults or supplied values.
Attributes:
shared
property
¶
shared: bool
Whether all rows match; the first query may synchronize with the device.
The result is cached. Do not mutate values after this property is read.
get
¶
get(path: str, default=None)
Return the leaf view for path, or default when the layout has no such leaf.
to
¶
to(device=None, dtype=None) -> StyleTensor
Return a copy on device with dtype, keeping the layout and known sharedness.
for_batch
¶
for_batch(batch: int, owner: str = 'render') -> StyleTensor
Return this style at batch rows.
A matching batch returns self. A shared style broadcasts its row; a style
with differing rows raises ValueError, since no row selection is implied.
from_space
classmethod
¶
from_space(space, batch: int, device='cuda', overrides: dict | None = None, values=None) -> StyleTensor
Build float32 appearance rows from space defaults or supplied values.
Values must contain one shared row or exactly batch rows. Overrides map
paths to either one leaf value or one value per batch row; they apply last.