diff --git a/README.md b/README.md index 4a9036c396d..23afbbe5300 100644 --- a/README.md +++ b/README.md @@ -223,6 +223,13 @@ docker run -it \ ``` > This image has used `uv` to install all GPU-related dependencies of Trinity-RFT. The virtual environment will be automatically activated upon entering the container (you can also manually activate it via `source /opt/venv/bin/activate` if needed). You can use `uv pip install` to add extra packages as necessary. +**Using the Installation Script (uv)** + +```bash +bash scripts/install/install_trinity.sh +source .venv/bin/activate +``` + **Using Conda** ```bash @@ -235,7 +242,7 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # If you encounter issues when installing flash-attn, try: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # for development like linting and debugging ``` @@ -251,34 +258,38 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # If you encounter issues when installing flash-attn, try: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # for development like linting and debugging ``` -**Using uv** +**Using uv manually** ```bash -uv sync --extra vllm --extra dev --extra flash_attn +uv venv .venv + +uv pip install \ + --python .venv/bin/python \ + --overrides scripts/install/trinity-overrides.txt \ + -e ".[vllm,dev]" + +.venv/bin/python scripts/install/install_flash_attn.py --uv # If you have no GPU, try to use Tinker instead: -# uv sync --extra tinker --extra dev +# uv pip install --python .venv/bin/python -e ".[tinker,dev]" + +source .venv/bin/activate ``` +If conflicts remain, add or adjust versions in `scripts/install/trinity-overrides.txt`. + #### Via PyPI If you just want to use the package without modifying the code: ```bash pip install trinity-rft -pip install flash-attn==2.8.1 -``` - -Or with `uv`: - -```bash -uv pip install trinity-rft -uv pip install flash-attn==2.8.1 +pip install "flash-attn>=2.8.3" --no-build-isolation ``` > For training with **Megatron-LM**, please refer to [Megatron-LM Backend](https://agentscope-ai.github.io/Trinity-RFT/en/main/tutorial/example_megatron.html). @@ -363,9 +374,10 @@ export WANDB_API_KEY= wandb login ``` -For command-line users, run the RFT process: +For command-line users, activate the environment and run the RFT process: ```shell +source .venv/bin/activate # If using venv or uv trinity run --config ``` diff --git a/README_zh.md b/README_zh.md index 663fa027be8..f9f08a1b509 100644 --- a/README_zh.md +++ b/README_zh.md @@ -245,6 +245,13 @@ docker run -it \ > 该镜像已经通过 `uv` 安装了 Trinity-RFT 以及所有 GPU 相关依赖,且会自动激活虚拟环境(也可通过 `source /opt/venv/bin/activate` 手动激活)。必要时可使用 `uv pip install` 添加额外的包。 +**使用安装脚本(uv)** + +```bash +bash scripts/install/install_trinity.sh +source .venv/bin/activate +``` + **使用 Conda** ```bash @@ -257,7 +264,7 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # 如果安装 flash-attn 时遇到问题,可尝试: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # 用于调试和开发 ``` @@ -274,36 +281,41 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # 如果安装 flash-attn 时遇到问题,可尝试: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # 用于调试和开发 ``` -**使用 uv** +**手动使用 uv** [`uv`](https://github.com/astral-sh/uv) 是现代的 Python 包管理工具。 ```bash -uv sync --extra vllm --extra dev --extra flash_attn +uv venv .venv + +uv pip install \ + --python .venv/bin/python \ + --overrides scripts/install/trinity-overrides.txt \ + -e ".[vllm,dev]" + +.venv/bin/python scripts/install/install_flash_attn.py --uv -# 如果没有GPU,可以改为使用Tinker: -# uv sync --extra tinker --extra dev +# 如果没有 GPU,可以改用 Tinker: +# uv pip install --python .venv/bin/python -e ".[tinker,dev]" + +source .venv/bin/activate ``` +脚本会安装匹配当前环境的第三方 [flash-attn wheel](https://github.com/mjun0812/flash-attention-prebuild-wheels)。 +如果仍有依赖冲突,可在 `scripts/install/trinity-overrides.txt` 中新增或调整版本约束。 + #### 通过 PyPI 安装 如果您只需使用 Trinity-RFT 而不打算修改代码: ```bash pip install trinity-rft -pip install flash-attn==2.8.1 -``` - -或使用 `uv`: - -```bash -uv pip install trinity-rft -uv pip install flash-attn==2.8.1 +pip install "flash-attn>=2.8.3" --no-build-isolation ``` > 如需使用 **Megatron-LM** 进行训练,请参考 [Megatron-LM 支持](https://agentscope-ai.github.io/Trinity-RFT/zh/main/tutorial/example_megatron.html) @@ -398,9 +410,10 @@ export WANDB_API_KEY= wandb login ``` -对于命令行用户,运行 RFT 流程: +对于命令行用户,先激活环境,再运行 RFT 流程: ```shell +source .venv/bin/activate # 使用 venv 或 uv 时 trinity run --config ``` diff --git a/docs/sphinx_doc/source/tutorial/faq.md b/docs/sphinx_doc/source/tutorial/faq.md index 87626ae1cbb..adbe809a18f 100644 --- a/docs/sphinx_doc/source/tutorial/faq.md +++ b/docs/sphinx_doc/source/tutorial/faq.md @@ -71,7 +71,7 @@ File ".../flash_attn/flash_attn_interface.py", line 15, in ‹module> ImportError: ... ``` -**A:** The `flash-attn` module is not properly installed. Try to fix it by running `pip install flash-attn==2.8.1` or `pip install flash-attn==2.8.1 -v --no-build-isolation`. +**A:** The `flash-attn` module is not properly installed. Try to fix it by running `pip install "flash-attn>=2.8.3"` or `pip install "flash-attn>=2.8.3" -v --no-build-isolation`. --- diff --git a/docs/sphinx_doc/source/tutorial/trinity_installation.md b/docs/sphinx_doc/source/tutorial/trinity_installation.md index 11ff36a5d89..b401dff387d 100644 --- a/docs/sphinx_doc/source/tutorial/trinity_installation.md +++ b/docs/sphinx_doc/source/tutorial/trinity_installation.md @@ -49,7 +49,7 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # If you encounter issues when installing flash-attn, try: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # for development like linting and debugging ``` @@ -66,7 +66,7 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # If you encounter issues when installing flash-attn, try: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # for development like linting and debugging ``` @@ -146,14 +146,14 @@ If you just want to use the package without modifying the code: ```bash pip install trinity-rft -pip install flash-attn==2.8.1 --no-build-isolation +pip install "flash-attn>=2.8.3" --no-build-isolation ``` Or with `uv`: ```bash uv pip install trinity-rft -uv pip install flash-attn==2.8.1 --no-build-isolation +uv pip install "flash-attn>=2.8.3" --no-build-isolation ``` --- diff --git a/docs/sphinx_doc/source_zh/tutorial/faq.md b/docs/sphinx_doc/source_zh/tutorial/faq.md index 67da2ee6459..578ab77da9e 100644 --- a/docs/sphinx_doc/source_zh/tutorial/faq.md +++ b/docs/sphinx_doc/source_zh/tutorial/faq.md @@ -70,7 +70,7 @@ File ".../flash_attn/flash_attn_interface.py", line 15, in ‹module> ImportError: ... ``` -**A:** `flash-attn` 模块未正确安装。请尝试运行 `pip install flash-attn==2.8.1` 或 `pip install flash-attn==2.8.1 -v --no-build-isolation` 进行修复。 +**A:** `flash-attn` 模块未正确安装。请尝试运行 `pip install "flash-attn>=2.8.3"` 或 `pip install "flash-attn>=2.8.3" -v --no-build-isolation` 进行修复。 --- diff --git a/docs/sphinx_doc/source_zh/tutorial/trinity_installation.md b/docs/sphinx_doc/source_zh/tutorial/trinity_installation.md index 42730f003b2..7c51634f53b 100644 --- a/docs/sphinx_doc/source_zh/tutorial/trinity_installation.md +++ b/docs/sphinx_doc/source_zh/tutorial/trinity_installation.md @@ -49,7 +49,7 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # 如果安装 flash-attn 时遇到问题,可尝试: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # 用于调试和开发 ``` @@ -66,7 +66,7 @@ pip install -e ".[vllm,flash_attn]" # pip install -e ".[tinker]" # 如果安装 flash-attn 时遇到问题,可尝试: -# pip install flash-attn==2.8.1 --no-build-isolation +# pip install "flash-attn>=2.8.3" --no-build-isolation pip install -e ".[dev]" # 用于调试和开发 ``` @@ -147,14 +147,14 @@ docker run -it \ ```bash pip install trinity-rft -pip install flash-attn==2.8.1 --no-build-isolation +pip install "flash-attn>=2.8.3" --no-build-isolation ``` 或使用 `uv`: ```bash uv pip install trinity-rft -uv pip install flash-attn==2.8.1 --no-build-isolation +uv pip install "flash-attn>=2.8.3" --no-build-isolation ``` --- diff --git a/pyproject.toml b/pyproject.toml index 008c00ab23f..a5d4f0a6f0c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -57,8 +57,7 @@ trinity = "trinity.cli.launcher:main" [project.optional-dependencies] vllm = [ - # for routed_experts support, please install vllm 0.22.0 or above - "vllm>=0.19.1,<=0.23.0", + "vllm>=0.22.0,<=0.23.0", ] sglang = [ "sglang==0.5.13", diff --git a/scripts/install/install_flash_attn.py b/scripts/install/install_flash_attn.py index 777fd883911..299adc3eae6 100644 --- a/scripts/install/install_flash_attn.py +++ b/scripts/install/install_flash_attn.py @@ -1,80 +1,125 @@ -"""This script is used to install flash-attn from a pre-built wheel hosted on an OSS bucket. -Useful for mainland China users who have difficulty installing flash-attn from PyPI due to network issues. -""" +"""Install a compatible third-party flash-attn wheel from a community index.""" + import os -import platform +import re import subprocess import sys import tempfile +from importlib.metadata import PackageNotFoundError +from importlib.metadata import version as package_version +from urllib.parse import unquote import torch import typer +from packaging.specifiers import SpecifierSet +from packaging.tags import sys_tags +from packaging.utils import InvalidWheelFilename, parse_wheel_filename +from packaging.version import InvalidVersion, Version app = typer.Typer() -FLASH_VERSION = "2.8.1" +FLASH_ATTN_RANGE = SpecifierSet(">=2.8.3") +WHEEL_INDEX_URL = ( + "https://raw.githubusercontent.com/mjun0812/" + "flash-attention-prebuild-wheels/main/doc/packages.md" +) +WHEEL_URL_PATTERN = re.compile( + r"(https://github\.com/mjun0812/flash-attention-prebuild-wheels/" + r"releases/download/[^ )]+/flash_attn-[^ )]+\.whl)" +) + + +def _get_cuda_torch_tag(): + # torch.version.hip/cuda are runtime attributes not in type stubs + is_rocm = ( + hasattr(torch.version, "hip") + and torch.version.hip is not None # type: ignore[attr-defined] + ) + if is_rocm: + raise RuntimeError("ROCm wheels are not supported.") + + torch_cuda_version = torch.version.cuda # type: ignore[attr-defined] + if torch_cuda_version is None: + raise RuntimeError("The installed PyTorch does not include CUDA support.") + + torch_major, torch_minor = torch.__version__.split("+", 1)[0].split(".")[:2] + return f"cu{torch_cuda_version.replace('.', '')}torch{torch_major}.{torch_minor}" def check_flash_attn_installed(): try: - import flash_attn - - print(f"flash_attn version: {flash_attn.__version__}") + installed_version = package_version("flash-attn") + print(f"flash_attn version: {installed_version}") + version = Version(installed_version) + if version not in FLASH_ATTN_RANGE: + return False + if re.fullmatch(r"cu\d+torch\d+\.\d+", version.local or "") and ( + version.local != _get_cuda_torch_tag() + ): + return False + __import__("flash_attn") return True - except ImportError: + except (ImportError, InvalidVersion, OSError, PackageNotFoundError, RuntimeError): return False -def install_flash_attn(uv: bool = False, keep_wheel: bool = False): - # Get torch version - TORCH_VERSION_RAW = torch.__version__ - torch_major, torch_minor = TORCH_VERSION_RAW.split(".")[:2] - torch_version = f"{torch_major}.{torch_minor}" - - # Get python version - python_version = f"cp{sys.version_info.major}{sys.version_info.minor}" +def find_wheel_url(): + cuda_torch_tag = _get_cuda_torch_tag() + + print(f"Detected: Python {sys.version_info.major}.{sys.version_info.minor} {cuda_torch_tag}") + + wheel_index = subprocess.run( + ["curl", "-fL", "--retry", "5", "--retry-all-errors", WHEEL_INDEX_URL], + check=True, + capture_output=True, + text=True, + ).stdout + supported_tags = set(sys_tags()) + matches = [] + for wheel_url in WHEEL_URL_PATTERN.findall(wheel_index): + wheel_url = unquote(wheel_url) + try: + distribution, version, _, wheel_tags = parse_wheel_filename( + wheel_url.rsplit("/", 1)[-1] + ) + except InvalidWheelFilename: + continue + if ( + distribution == "flash-attn" + and version in FLASH_ATTN_RANGE + and version.local == cuda_torch_tag + and supported_tags.intersection(wheel_tags) + ): + matches.append((version, wheel_url)) + if matches: + return max(matches, key=lambda item: item[0])[1] + + raise RuntimeError(f"No matching flash-attn wheel found for {cuda_torch_tag}.") - # Get platform name - platform_name = platform.system().lower() + "_" + platform.machine() - # Get cxx11_abi - cxx11_abi = str(torch._C._GLIBCXX_USE_CXX11_ABI).upper() - - # Is ROCM - # torch.version.hip/cuda are runtime attributes not in type stubs - IS_ROCM = hasattr(torch.version, "hip") and torch.version.hip is not None # type: ignore[attr-defined] - - if IS_ROCM: - print("We currently do not host ROCm wheels for flash-attn.") - sys.exit(1) - else: - torch_cuda_version = torch.version.cuda # type: ignore[attr-defined] - cuda_major = torch_cuda_version.split(".")[0] if torch_cuda_version else None - if cuda_major != "12": - print("Only CUDA 12 wheels are hosted for flash-attn.") - sys.exit(1) - cuda_version = "12" - wheel_filename = ( - f"flash_attn-{FLASH_VERSION}%2Bcu{cuda_version}torch{torch_version}" - f"cxx11abi{cxx11_abi}-{python_version}-{python_version}-{platform_name}.whl" - ) - local_filename = ( - f"flash_attn-{FLASH_VERSION}-{python_version}-{python_version}-{platform_name}.whl" - ) - - wheel_url = ( - "https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com" - f"/AgentScope/download/flash-attn/{FLASH_VERSION}/{wheel_filename}" - ) +def install_flash_attn(uv: bool = False, keep_wheel: bool = False): + wheel_url = find_wheel_url() + local_filename = wheel_url.rsplit("/", 1)[-1] print(f"wheel_url: {wheel_url}") - print(f"target_local_file: {local_filename}") def _install_helper(local_path: str): - subprocess.run(["wget", wheel_url, "-O", local_path], check=True) + subprocess.run( + [ + "curl", + "-fL", + "--retry", + "5", + "--retry-all-errors", + "-o", + local_path, + wheel_url, + ], + check=True, + ) install_cmd = ( - ["uv", "pip", "install", local_path] + ["uv", "pip", "install", "--python", sys.executable, "--no-deps", local_path] if uv - else [sys.executable, "-m", "pip", "install", local_path] + else [sys.executable, "-m", "pip", "install", "--no-deps", local_path] ) subprocess.run(install_cmd, check=True) @@ -99,7 +144,7 @@ def main( False, help="Keep the downloaded wheel file in current directory" ), ): - """Install flash-attn from a pre-built wheel.""" + """Install flash-attn from a matching pre-built wheel.""" if check_flash_attn_installed(): print("flash_attn is already installed. Skipping installation.") return diff --git a/scripts/install/install_trinity.sh b/scripts/install/install_trinity.sh new file mode 100644 index 00000000000..cf907c046f4 --- /dev/null +++ b/scripts/install/install_trinity.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." && pwd)" +VENV="${REPO_ROOT}/.venv" +PYTHON_BIN="${VENV}/bin/python" + +cd "${REPO_ROOT}" + +[[ -x "${PYTHON_BIN}" ]] || uv venv "${VENV}" + +uv pip install \ + --python "${PYTHON_BIN}" \ + --overrides "${SCRIPT_DIR}/trinity-overrides.txt" \ + -e "${REPO_ROOT}[vllm,dev]" + +"${PYTHON_BIN}" "${SCRIPT_DIR}/install_flash_attn.py" --uv + +echo "Activate with: source ${VENV}/bin/activate" diff --git a/scripts/install/trinity-overrides.txt b/scripts/install/trinity-overrides.txt new file mode 100644 index 00000000000..e70ae0b6877 --- /dev/null +++ b/scripts/install/trinity-overrides.txt @@ -0,0 +1,2 @@ +numpy>=2 +transformers>=5.12.1