Skip to content
OpenSpace-LabPublic

Repository files navigation

OpenNWM

A Generalist Navigation World Model
with Latent Action Pretraining

OpenSpace Lab

Learn navigation dynamics from action-free video. Adapt them to physical world.

Project Page arXiv:2611.01844 Models ModelScope Models Data ModelScope Data

OpenNWM method overview: latent action learning and navigation world modeling.

News · Setup · Data · Models · Evaluation · Training · TODO List

News

  • 2026-11-03 — The paper is available on arXiv.
  • 2026-11-01 — Updated the paper figure and documentation. Code, models, and NavAnywhere data are available.

Environment Setup

Requires Linux, Conda, and an NVIDIA GPU compatible with CUDA 12.4. Run from the repository root; LAM and NWM use separate environments.

git clone https://github.com/OpenSpace-Lab/OpenNWM.git
cd OpenNWM
for module in lam nwm; do
  conda create -n "open-$module" python=3.10 -y
  conda run -n "open-$module" python -m pip install torch==2.6.0 torchvision==0.21.0 \
    --index-url https://download.pytorch.org/whl/cu124
  conda run -n "open-$module" python -m pip install -r "$module/requirements.txt"
done

conda run -n open-nwm python setup_paths.py \
  --data-root /data/navigation --work-root /data/opennwm --nav-profile 15 \
  --lam-python "$(conda run -n open-lam python -c 'import sys; print(sys.executable)')" \
  --nwm-python "$(conda run -n open-nwm python -c 'import sys; print(sys.executable)')"
source .env

Replace /data/navigation and /data/opennwm with your data and work directories. Run source .env in each new terminal. To reuse existing splits, add --split-root /path/to/splits to setup_paths.py.

Data Preparation

Download NavAnywhere from Hugging Face or ModelScope and extract it to $NWM_DATA_ROOT. The archives total 1.93 TB; extraction requires tar, zstd, and additional disk space.

Download and extract with ModelScope
"$NWM_PYTHON" -m pip install modelscope
conda run -n open-nwm ms-hub download LittleBoss/NavAnywhere --repo-type dataset \
  --local-dir "$OPENNWM_WORK_ROOT/downloads/NavAnywhere" --max-workers 8
(
  cd "$OPENNWM_WORK_ROOT/downloads/NavAnywhere" || exit 1
  mkdir -p "$NWM_DATA_ROOT"
  find NavAnywhere -path '*/packs/*.tar.zst' -print0 \
    | xargs -0 -r -I{} tar --use-compress-program=unzstd -xf '{}' -C "$NWM_DATA_ROOT"
  for first in LAM-Data-{recon,scand,sacson}-*.tar.zst.part-00000; do
    [ -f "$first" ] || continue
    cat "${first%.part-00000}.part-"* \
      | tar --use-compress-program=unzstd -xf - -C "$NWM_DATA_ROOT"
  done
)

The action-data archives retain their LAM-Data- filenames. Expected layout:

$NWM_DATA_ROOT/
├── recon/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── sacson/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── scand/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
├── tartan/<trajectory>/{0.jpg,1.jpg,...,traj_data.pkl}
└── NavAnywhere/<source>/<trajectory>/{0.jpg,1.jpg,...}

The release includes RECON, HuRoN (sacson), SCAND, and NavAnywhere. TartanDrive requires a separate download for world model post-training and default LAM evaluation; follow the NWM data instructions and place it in tartan/ (split name: tartan_drive). setup_paths.py prepares the fixed train/validation/test lists and NavAnywhere recipe.

Pretrained Models

Download from Hugging Face or ModelScope into $OPENNWM_WORK_ROOT/models/OpenNWM, including configs/.

Checkpoint Use
lam-nav15-pixel-action-step60k.ckpt Latent action extraction
opennwm-nav15-stage1-pretrain60k.pth.tar World model pretraining checkpoint
opennwm-nav15-stage2-joint100k.pth.tar Physical-action prediction and planning
Download and verify with ModelScope
"$NWM_PYTHON" -m pip install modelscope
conda run -n open-nwm ms-hub download LittleBoss/OpenNWM --repo-type model \
  --local-dir "$OPENNWM_WORK_ROOT/models/OpenNWM" --max-workers 4
(cd "$OPENNWM_WORK_ROOT/models/OpenNWM" && sha256sum -c SHA256SUMS)

NWM downloads SD-VAE on first use; set NWM_VAE_PATH in .env to use a local copy. DINOv3 is needed only for LAM variant 3.

Evaluation

After preparing test data, import the Stage 2 checkpoint and run prediction:

"$NWM_PYTHON" run.py nwm import-checkpoint \
  --checkpoint "$OPENNWM_WORK_ROOT/models/OpenNWM/opennwm-nav15-stage2-joint100k.pth.tar" \
  --config "$OPENNWM_WORK_ROOT/models/OpenNWM/configs/opennwm-nav15-stage2.yaml" \
  --run-dir "$OPENNWM_WORK_ROOT/runs/released-stage2"
"$NWM_PYTHON" run.py nwm infer \
  exp_dir="$OPENNWM_WORK_ROOT/runs/released-stage2" ckp=latest eval_type=time \
  prediction_dir="$OPENNWM_WORK_ROOT/eval/stage2/pred"

Metrics, rollouts, and planning: NWM evaluation. Latent action reconstruction: LAM evaluation.

Training

Follow LAM training → cache preparation → world model pretraining and adaptation. Use the NavAnywhere 15-source profile and matching LAM checkpoints for caches. For the released model, set LAM_PIXEL_ACTION_CHECKPOINT in .env to $OPENNWM_WORK_ROOT/models/OpenNWM/lam-nav15-pixel-action-step60k.ckpt.

TODO List

  • Release code, data, and pretrained models.
  • Publish the paper, project page, and demos.
  • Provide training and evaluation guides.
  • Support closed-loop navigation on Unitree Go2 (deployment plan).

Acknowledgements & Citation

We thank NWM, CompACT, DreamDojo, and the dataset authors.

If you use OpenNWM, please cite:

@misc{nie2026opennwmgeneralistnavigationworld,
      title={OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining},
      author={Dujun Nie and Ruiqi Song and Zixuan Wang and Hanqing Wang and Yifan Han and Baiyong Ding and Wenzhao Zheng and Hangbin Wu and Long Chen},
      year={2026},
      eprint={2611.01844},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2611.01844},
}

NWM-derived components retain CC BY-NC 4.0; DreamDojo LAM components retain Apache 2.0. Datasets and external models retain their own licenses.

Releases

Packages

Contributors

Languages