Learn navigation dynamics from action-free video. Adapt them to physical world.
News · Setup · Data · Models · Evaluation · Training · TODO List
- 2026-11-03 — The paper is available on arXiv.
- 2026-11-01 — Updated the paper figure and documentation. Code, models, and NavAnywhere data are available.
Requires Linux, Conda, and an NVIDIA GPU compatible with CUDA 12.4. Run from the repository root; LAM and NWM use separate environments.
git clone https://github.com/OpenSpace-Lab/OpenNWM.git
cd OpenNWM
for module in lam nwm; do
conda create -n "open-$module" python=3.10 -y
conda run -n "open-$module" python -m pip install torch==2.6.0 torchvision==0.21.0 \
--index-url https://download.pytorch.org/whl/cu124
conda run -n "open-$module" python -m pip install -r "$module/requirements.txt"
done
conda run -n open-nwm python setup_paths.py \
--data-root /data/navigation --work-root /data/opennwm --nav-profile 15 \
--lam-python "$(conda run -n open-lam python -c 'import sys; print(sys.executable)')" \
--nwm-python "$(conda run -n open-nwm python -c 'import sys; print(sys.executable)')"
source .envReplace /data/navigation and /data/opennwm with your data and work directories. Run source .env in each new terminal. To reuse existing splits, add --split-root /path/to/splits to setup_paths.py.
Download NavAnywhere from Hugging Face or ModelScope and extract it to $NWM_DATA_ROOT. The archives total 1.93 TB; extraction requires tar, zstd, and additional disk space.
Download and extract with ModelScope
"$NWM_PYTHON" -m pip install modelscope
conda run -n open-nwm ms-hub download LittleBoss/NavAnywhere --repo-type dataset \
--local-dir "$OPENNWM_WORK_ROOT/downloads/NavAnywhere" --max-workers 8
(
cd "$OPENNWM_WORK_ROOT/downloads/NavAnywhere" || exit 1
mkdir -p "$NWM_DATA_ROOT"
find NavAnywhere -path '*/packs/*.tar.zst' -print0 \
| xargs -0 -r -I{} tar --use-compress-program=unzstd -xf '{}' -C "$NWM_DATA_ROOT"
for first in LAM-Data-{recon,scand,sacson}-*.tar.zst.part-00000; do
[ -f "$first" ] || continue
cat "${first%.part-00000}.part-"* \
| tar --use-compress-program=unzstd -xf - -C "$NWM_DATA_ROOT"
done
)The action-data archives retain their LAM-Data- filenames. Expected layout:
$NWM_DATA_ROOT/
├── recon/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── sacson/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── scand/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── tartan/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
└── NavAnywhere/<source>/<trajectory>/{0.jpg,1.jpg,...}
The release includes RECON, HuRoN (sacson), SCAND, and NavAnywhere. TartanDrive requires a separate download for world model post-training and default LAM evaluation; follow the NWM data instructions and place it in tartan/ (split name: tartan_drive). setup_paths.py prepares the fixed train/validation/test lists and NavAnywhere recipe.
Download from Hugging Face or ModelScope into $OPENNWM_WORK_ROOT/models/OpenNWM, including configs/.
| Checkpoint | Use |
|---|---|
lam-nav15-pixel-action-step60k.ckpt |
Latent action extraction |
opennwm-nav15-stage1-pretrain60k.pth.tar |
World model pretraining checkpoint |
opennwm-nav15-stage2-joint100k.pth.tar |
Physical-action prediction and planning |
Download and verify with ModelScope
"$NWM_PYTHON" -m pip install modelscope
conda run -n open-nwm ms-hub download LittleBoss/OpenNWM --repo-type model \
--local-dir "$OPENNWM_WORK_ROOT/models/OpenNWM" --max-workers 4
(cd "$OPENNWM_WORK_ROOT/models/OpenNWM" && sha256sum -c SHA256SUMS)NWM downloads SD-VAE on first use; set NWM_VAE_PATH in .env to use a local copy. DINOv3 is needed only for LAM variant 3.
After preparing test data, import the Stage 2 checkpoint and run prediction:
"$NWM_PYTHON" run.py nwm import-checkpoint \
--checkpoint "$OPENNWM_WORK_ROOT/models/OpenNWM/opennwm-nav15-stage2-joint100k.pth.tar" \
--config "$OPENNWM_WORK_ROOT/models/OpenNWM/configs/opennwm-nav15-stage2.yaml" \
--run-dir "$OPENNWM_WORK_ROOT/runs/released-stage2"
"$NWM_PYTHON" run.py nwm infer \
exp_dir="$OPENNWM_WORK_ROOT/runs/released-stage2" ckp=latest eval_type=time \
prediction_dir="$OPENNWM_WORK_ROOT/eval/stage2/pred"Metrics, rollouts, and planning: NWM evaluation. Latent action reconstruction: LAM evaluation.
Follow LAM training → cache preparation → world model pretraining and adaptation. Use the NavAnywhere 15-source profile and matching LAM checkpoints for caches. For the released model, set LAM_PIXEL_ACTION_CHECKPOINT in .env to $OPENNWM_WORK_ROOT/models/OpenNWM/lam-nav15-pixel-action-step60k.ckpt.
- Release code, data, and pretrained models.
- Publish the paper, project page, and demos.
- Provide training and evaluation guides.
- Support closed-loop navigation on Unitree Go2 (deployment plan).
We thank NWM, CompACT, DreamDojo, and the dataset authors.
If you use OpenNWM, please cite:
@misc{nie2026opennwmgeneralistnavigationworld,
title={OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining},
author={Dujun Nie and Ruiqi Song and Zixuan Wang and Hanqing Wang and Yifan Han and Baiyong Ding and Wenzhao Zheng and Hangbin Wu and Long Chen},
year={2026},
eprint={2611.01844},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2611.01844},
}NWM-derived components retain CC BY-NC 4.0; DreamDojo LAM components retain Apache 2.0. Datasets and external models retain their own licenses.