SADL world models with shared BatchNorm

Checkpoints and datasets for SADL: On the Demand-Supply Geometry of Representation Collapse. Project page.

Checkpoints and training data for the SADL training and evaluation code. Models are trained on PushT, Cube, Reacher and TwoRoom with one BatchNorm population shared by all tasks and GPUs, for two objectives:

  • INTACT: latent prediction and goal-conditioned action (planners: Direct, Guarded A);
  • LeWM: latent prediction only (planner: CEM).

Contents

checkpoints/<model>/<training>/seed<seed>/<task>/
    weights_epoch_1.pt     model weights and BatchNorm statistics
    config.yaml            model configuration
    action_stats.json      action normalization of the task
    training_config.json   training configuration of the run
datasets/
    pusht_expert_train.h5  cube_single_expert.h5  reacher.h5  tworoom.h5
    SHA256SUMS
CHECKPOINTS_SHA256.txt

<model> is intact or lewm; <training> is four_task (one model trained on all four tasks; the four task folders share every weight and normalization statistic and differ only in action dimension and action scaling) or single_task (one model per task); seeds are 3072, 3073 and 3074. Every checkpoint is trained for one epoch.

JEPA glyph checkpoints

Training checkpoints for J, E, P, and A are available separately. Each letter is an epoch-1 INTACT-Lite model trained jointly on PushT, Cube, Reacher, and TwoRoom.

Usage

git clone https://github.com/zju3dv/SADL.git && cd SADL
pip install -r requirements.txt
hf download DavidSunok/SADL --local-dir hf
python evaluate.py --checkpoint hf/checkpoints/intact/four_task/seed3072/reacher --task reacher \
    --data-dir hf/datasets --planner direct
python evaluate.py --checkpoint hf/checkpoints/lewm/single_task/seed3072/pusht --task pusht \
    --data-dir hf/datasets --planner cem

Results

Success rate (%), mean +- SD over the three training seeds; each seed is evaluated with seeds 0, 1 and 42, 100 episodes each, on an NVIDIA RTX 4090 (goal 25 steps ahead, at most 50 actions, horizon 5, replanning every 5 actions).

Model Training Planner PushT Cube Reacher TwoRoom Mean
INTACT four tasks Direct 85.7 +- 0.9 100.0 +- 0.0 93.4 +- 1.7 100.0 +- 0.0 94.8 +- 0.4
INTACT four tasks Guarded A 88.1 +- 0.8 99.6 +- 0.4 93.1 +- 0.7 99.8 +- 0.4 95.1 +- 0.3
INTACT single task Direct 85.1 +- 0.4 100.0 +- 0.0 94.9 +- 2.0 99.7 +- 0.6 94.9 +- 0.5
INTACT single task Guarded A 89.8 +- 0.2 99.6 +- 0.2 93.7 +- 3.2 99.3 +- 0.9 95.6 +- 0.9
LeWM four tasks CEM 64.7 +- 18.5 59.6 +- 1.3 46.7 +- 26.1 94.4 +- 1.0 66.3 +- 9.2
LeWM single task CEM 83.1 +- 1.6 58.9 +- 1.4 81.0 +- 4.3 91.3 +- 1.5 78.6 +- 1.9

Datasets

The HDF5 files hold expert demonstrations with uint8 RGB frames (224 x 224) and actions. PushT, Cube, Reacher and TwoRoom come from the LeWorldModel benchmark. The original dataset cards specify the MIT license: PushT, Cube, Reacher, and TwoRoom. The upstream copyright and permission notice is included in datasets/LeWM-MIT-LICENSE.txt.

License

The MIT license applies to the released weights. The upstream dataset license is linked above.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading