Classic control¶
The classic control tasks of Gymnasium (Towers et al., 2024) balance a pole on a cart, swing up a pendulum or a chain of links, and drive a car up a hill. stable-envs steps the batch in NumPy on CPU and on the GPU on CUDA, and draws it with a compiled reproduction of Gymnasium's pygame drawing.
import stable_envs as snv
with snv.CartPole(num_envs=8, device="cpu") as env:
state = env.reset(seed=0)
print(env.render().shape) # torch.Size([8, 400, 600, 3])
Install¶
CartPole and Acrobot run with the core install. MountainCar, MountainCarContinuous and Pendulum draw part of their scene with pygame, which the classic extra installs along with Numba for the compiled CPU renderer. CUDA needs a CUDA build of torch; see What each family needs.
Task¶
Each class keeps Gymnasium's actions, rewards and termination, and Classes describes them task by task.
Simulation¶
The simulation_backend argument selects how the batch steps:
| Value | Steps |
|---|---|
"auto" (default) |
on CPU, "gym" for small batches and "numpy" for large batches. On CUDA, the CPU choice for small batches and "torch" for large batches |
"numpy" |
the whole batch in one NumPy operation on the host, with outputs moved to the environment device |
"numba" |
CartPole and MountainCar: the whole batch in one compiled call on the host, in place on the state |
"torch" |
the whole batch in one Torch operation on the environment device |
"gym" |
one Gymnasium environment per batch row |
None means "auto". CartPole and MountainCar step with "numba" on the host at every batch size, because it was the fastest host backend from 1 to 4096 environments. Each other class ships two switch points, measured on RTX 3090 and RTX A5000 hosts: the batch size from which the host steps in NumPy instead of Gymnasium, and the batch size from which CUDA batches step in Torch. The large_backend_min_envs tuning setting changes only the CUDA switch point. stable-envs tune --device cpu changes the host switch point, and stable-envs tune --device cuda the CUDA one (see Physics backends). On CUDA, the CPU choice steps on the host and moves the outputs and the state of each step to the GPU in one copy.
Gymnasium resets each environment and owns its PCG64 random stream for every backend, so a seed gives the same initial states on every backend and device. NumPy and Numba steps give state, observations, rewards and episode flags equal to Gymnasium's steps. env.simulation_backend reports the selected backend.
On CUDA, "torch" keeps the state on the GPU. One Triton kernel steps the batch and a CUDA graph replays it, so a step reads nothing back to the host. A reset draws the initial states with Gymnasium on the host and copies them to the GPU in one copy. The step doesn't check the range of discrete actions, because that check would read back to the host, and an action out of range gives undefined dynamics.
with snv.Acrobot(num_envs=256, simulation_backend="numpy", device="cpu") as env:
print(env.simulation_backend) # numpy
with snv.Acrobot(num_envs=256, simulation_backend="gym", device="cpu") as env:
print(env.simulation_backend) # gym
# CUDA only
with snv.Acrobot(num_envs=64, device="cuda", simulation_backend="torch") as env:
print(env.simulation_backend) # torch
with snv.tuning.override(large_backend_min_envs=256), snv.Acrobot(num_envs=64, device="cuda") as env:
print(env.simulation_backend) # gym
Rendering¶
The renderer reimplements Gymnasium's pygame gfxdraw drawing and gives Gymnasium's frame for the same state. A fused Triton kernel draws the frames on CUDA, and on CPU a Numba kernel draws the same primitive records in parallel and gives the same frames as the Triton kernel. The classic extra installs Numba. With backend="auto", a CPU batch of fewer than 8 states with the default appearance draws with Gymnasium's renderer instead, which is faster at that size. Without Numba, the CPU path draws with Gymnasium's renderer (default appearance, when pygame is installed) or with the Torch reimplementation.
State¶
The state folds its fields into one array for each dtype: float64, int64 and bool. The arrays are the leaves of the state, under the key of the class, for example cartpole. snv.get_field(state, name) reads one field by name, and snv.schema_of(state).names lists them. The simulation variables come first in the float64 array, for example x, x_dot, theta and theta_dot of CartPole. The other fields are lifecycle/elapsed_steps, the task fields under task/, the random number generator under rng/ and the physical constants under physics/.
with snv.CartPole(num_envs=4, device="cpu") as env:
state = env.reset(seed=0)
print(list(state.keys(True, True))) # [('cartpole', 'float64'), ('cartpole', 'int64'), ('cartpole', 'bool')]
print(snv.get_field(state, "theta").shape) # torch.Size([4, 1])
Variation¶
The physics/ fields of the state hold the physical constants of each environment, for example gravity and masses, and the variation can change them. Each class has physics.* leaves for these constants, a color leaf for each part of the scene and rendering.antialias.
Classes¶
CartPole¶
A pole stands on a cart that moves along a track, and the agent pushes the cart left or right to keep the pole upright. Action 0 pushes the cart to the left and action 1 pushes it to the right. The episode terminates when the pole angle is more than 12° from vertical or the cart leaves the track.
| Property | Value |
|---|---|
| Reference | CartPole-v1 |
| Physics | batched NumPy, or batched Torch for large CUDA batches, with Gymnasium resets |
| Devices | CPU, CUDA |
| Requires CUDA | no |
| Renderer on CPU | gfxdraw port (Numba; Torch or Gymnasium without Numba) |
| Renderer on CUDA | gfxdraw port (fused Triton kernel) |
| Default frame | 400 × 600 × 3, uint8 |
| Native size | 600 px, the only size |
| Action | int64 (1,), 2 choices |
| Clips | rendered on CUDA at 600 px, then resized |
State (1 top-level keys)
| Key | Shape | dtype |
|---|---|---|
cartpole |
3 leaves, 23 values | bool, float64, int64 |
Variation space (12 leaves)
| Path | Type | Bounds or choices | Default | Sampled at reset |
|---|---|---|---|---|
axle.color |
RGBBox (3,) | [0, 255] | [129, 132, 203] | |
background.color |
RGBBox (3,) | [0, 255] | 255 | |
cart.color |
RGBBox (3,) | [0, 255] | 0 | |
physics.force_mag |
Box (1,) | [5, 15] | 10 | |
physics.gravity |
Box (1,) | [4.9, 14.7] | 9.8 | |
physics.length |
Box (1,) | [0.25, 0.75] | 0.5 | |
physics.masscart |
Box (1,) | [0.5, 1.5] | 1 | |
physics.masspole |
Box (1,) | [0.05, 0.5] | 0.1 | |
physics.tau |
Box (1,) | [0.01, 0.04] | 0.02 | |
pole.color |
RGBBox (3,) | [0, 255] | [202, 152, 101] | |
rendering.antialias |
Discrete | 2 choices from 0 | 1 | |
track.color |
RGBBox (3,) | [0, 255] | 0 |
Pendulum¶
A pendulum hangs from a fixed pivot, and the agent applies a torque to swing it up and hold it upright. The task has no termination condition, and the reward penalizes the angle, the angular velocity and the torque.
| Property | Value |
|---|---|
| Reference | Pendulum-v1 |
| Physics | batched NumPy, or batched Torch for large CUDA batches, with Gymnasium resets |
| Devices | CPU, CUDA |
| Requires CUDA | no |
| Renderer on CPU | gfxdraw port (Numba; Torch or Gymnasium without Numba) |
| Renderer on CUDA | gfxdraw port (fused Triton kernel) |
| Default frame | 500 × 500 × 3, uint8 |
| Native size | 500 px, the only size |
| Action | float32 (1,), [-2, 2] |
| Clips | rendered on CUDA at 500 px, then resized |
State (1 top-level keys)
| Key | Shape | dtype |
|---|---|---|
pendulum |
3 leaves, 21 values | bool, float64, int64 |
Variation space (10 leaves)
| Path | Type | Bounds or choices | Default | Sampled at reset |
|---|---|---|---|---|
axle.color |
RGBBox (3,) | [0, 255] | 0 | |
background.color |
RGBBox (3,) | [0, 255] | 255 | |
physics.dt |
Box (1,) | [0.025, 0.1] | 0.05 | |
physics.g |
Box (1,) | [5, 15] | 10 | |
physics.l |
Box (1,) | [0.5, 1.5] | 1 | |
physics.m |
Box (1,) | [0.5, 1.5] | 1 | |
physics.max_speed |
Box (1,) | [4, 12] | 8 | |
physics.max_torque |
Box (1,) | [1, 4] | 2 | |
rendering.antialias |
Discrete | 2 choices from 0 | 1 | |
rod.color |
RGBBox (3,) | [0, 255] | [204, 77, 77] |
Acrobot¶
Acrobot is a chain of two links where only the joint between the links has an actuator. The three actions apply a torque of -1, 0 or +1, and the episode terminates when the free end rises above a target height.
| Property | Value |
|---|---|
| Reference | Acrobot-v1 |
| Physics | batched NumPy, or batched Torch for large CUDA batches, with Gymnasium resets |
| Devices | CPU, CUDA |
| Requires CUDA | no |
| Renderer on CPU | gfxdraw port (Numba; Torch or Gymnasium without Numba) |
| Renderer on CUDA | gfxdraw port (fused Triton kernel) |
| Default frame | 500 × 500 × 3, uint8 |
| Native size | 500 px, the only size |
| Action | int64 (1,), 3 choices |
| Clips | rendered on CUDA at 500 px, then resized |
State (1 top-level keys)
| Key | Shape | dtype |
|---|---|---|
acrobot |
2 leaves, 22 values | float64, int64 |
Variation space (12 leaves)
| Path | Type | Bounds or choices | Default | Sampled at reset |
|---|---|---|---|---|
background.color |
RGBBox (3,) | [0, 255] | 255 | |
ground.color |
RGBBox (3,) | [0, 255] | 0 | |
joint.color |
RGBBox (3,) | [0, 255] | [204, 204, 0] | |
link.color |
RGBBox (3,) | [0, 255] | [0, 204, 204] | |
physics.link_com_pos_1 |
Box (1,) | [0.25, 0.75] | 0.5 | |
physics.link_com_pos_2 |
Box (1,) | [0.25, 0.75] | 0.5 | |
physics.link_length_1 |
Box (1,) | [0.5, 1.5] | 1 | |
physics.link_length_2 |
Box (1,) | [0.5, 1.5] | 1 | |
physics.link_mass_1 |
Box (1,) | [0.5, 1.5] | 1 | |
physics.link_mass_2 |
Box (1,) | [0.5, 1.5] | 1 | |
physics.link_moi |
Box (1,) | [0.5, 1.5] | 1 | |
rendering.antialias |
Discrete | 2 choices from 0 | 1 |
MountainCar¶
A car sits in a valley between two hills, and its engine is too weak to drive straight up the right hill. The car must swing back and forth to gain momentum. The three actions accelerate left, do nothing or accelerate right. The episode terminates when the car reaches the flag at position 0.5.
| Property | Value |
|---|---|
| Reference | MountainCar-v0 |
| Physics | batched NumPy, or batched Torch for large CUDA batches, with Gymnasium resets |
| Devices | CPU, CUDA |
| Requires CUDA | no |
| Renderer on CPU | gfxdraw port (Numba; Torch or Gymnasium without Numba) |
| Renderer on CUDA | gfxdraw port (fused Triton kernel) |
| Default frame | 400 × 600 × 3, uint8 |
| Native size | 600 px, the only size |
| Action | int64 (1,), 3 choices |
| Clips | rendered on CUDA at 600 px, then resized |
State (1 top-level keys)
| Key | Shape | dtype |
|---|---|---|
mountain_car |
2 leaves, 16 values | float64, int64 |
Variation space (10 leaves)
| Path | Type | Bounds or choices | Default | Sampled at reset |
|---|---|---|---|---|
background.color |
RGBBox (3,) | [0, 255] | 255 | |
car.color |
RGBBox (3,) | [0, 255] | 0 | |
flag.color |
RGBBox (3,) | [0, 255] | [204, 204, 0] | |
flag.pole_color |
RGBBox (3,) | [0, 255] | 0 | |
physics.force |
Box (1,) | [0.0005, 0.0015] | 0.001 | |
physics.gravity |
Box (1,) | [0.00125, 0.00375] | 0.0025 | |
physics.max_speed |
Box (1,) | [0.035, 0.105] | 0.07 | |
rendering.antialias |
Discrete | 2 choices from 0 | 1 | |
terrain.color |
RGBBox (3,) | [0, 255] | 0 | |
wheel.color |
RGBBox (3,) | [0, 255] | 128 |
MountainCarContinuous¶
This class has the same terrain as MountainCar, but the action is a continuous force in [-1, 1] and the flag is at position 0.45. The reward gives 100 at the goal and subtracts a cost for each action.
| Property | Value |
|---|---|
| Reference | MountainCarContinuous-v0 |
| Physics | batched NumPy, or batched Torch for large CUDA batches, with Gymnasium resets |
| Devices | CPU, CUDA |
| Requires CUDA | no |
| Renderer on CPU | gfxdraw port (Numba; Torch or Gymnasium without Numba) |
| Renderer on CUDA | gfxdraw port (fused Triton kernel) |
| Default frame | 400 × 600 × 3, uint8 |
| Native size | 600 px, the only size |
| Action | float32 (1,), [-1, 1] |
| Clips | rendered on CUDA at 600 px, then resized |
State (1 top-level keys)
| Key | Shape | dtype |
|---|---|---|
mountain_car_continuous |
3 leaves, 16 values | bool, float64, int64 |
Variation space (9 leaves)
| Path | Type | Bounds or choices | Default | Sampled at reset |
|---|---|---|---|---|
background.color |
RGBBox (3,) | [0, 255] | 255 | |
car.color |
RGBBox (3,) | [0, 255] | 0 | |
flag.color |
RGBBox (3,) | [0, 255] | [204, 204, 0] | |
flag.pole_color |
RGBBox (3,) | [0, 255] | 0 | |
physics.max_speed |
Box (1,) | [0.035, 0.105] | 0.07 | |
physics.power |
Box (1,) | [0.00075, 0.00225] | 0.0015 | |
rendering.antialias |
Discrete | 2 choices from 0 | 1 | |
terrain.color |
RGBBox (3,) | [0, 255] | 0 | |
wheel.color |
RGBBox (3,) | [0, 255] | 128 |
Limitations¶
- Each class renders only at the native size of its Gymnasium renderer. Resize the output for a smaller frame.
- Torch steps agree with Gymnasium to rounding, not bit for bit: Torch computes
sin,cosand powers with its own code.