DeepSpeed-FastGen support ascend npu? #960

Activity

1. loadams commented on Mar 13, 2025

   

   loadams

   on Mar 13, 2025

   Contributor

   More actions

   @RyanOvO - DeepSpeed supports the Ascend NPU, but I don't believe FastGen has been tested there. @hipudding or @xuedinge233, do you know?
2. hipudding commented on Mar 14, 2025

   

   hipudding

   on Mar 14, 2025

   More actions

   We have not tested it before. Will test it now.
3. xuedinge233 commented on Mar 24, 2025

   

   xuedinge233

   on Mar 24, 2025

   Last edited by xuedinge233

   More actions

   [2025-03-24 06:12:22,548] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to npu (auto detect)
   torch npu:  True
   [2025-03-24 06:12:25,338] [INFO] [comm.py:658:init_distributed] cdb=None
   [2025-03-24 06:12:25,338] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend hccl
   [2025-03-24 06:12:35,567] [INFO] [engine_v2.py:82:__init__] Building model...
   ...
   Config max_tokens=768 type='rms_norm' channels=4096 residual_dtype=torch.bfloat16 input_dtype=torch.bfloat16 output_dtype=torch.bfloat16 eps=1e-05 is not supported by <class 'deepspeed.inference.v2.modules.implementations.pre_norm.cuda_pre_rms.DSPreRMSCUDAModule'>
   

   Currently, Deepspeed-mii only supports CPU and CUDA, NPU does not support it.