Need for HGEMM and HGEMV?

[ChipKerchner]

Since ARM has added support for BGEMM and BGEMV, do we need to add support for HGEMM and HEGMV as well?

@Mousius I assume the differences between BGEMM and SBGEMM is the input/output vector and that the operations themselves are the same? Only diff is the input/output is converted to BF16

[Mousius]

You could technically accumulate into BF16 if you had instructions to do so, though the accuracy would likely suffer. For the BGEMM/BGEMV kernels I implemented they are the exact same internals though.

Since ARM has added support for BGEMM and BGEMV, do we need to add support for HGEMM and HEGMV as well?

We managed to get a pretty good speed up from replacing this block in PyTorch with bgemm_:
https://github.com/pytorch/pytorch/blob/a2e2e1d8c026951baa345f0dd17668bd1718eda5/aten/src/ATen/native/CPUBlas.cpp#L356-L383

Which avoids all the copying and conversions, because it's done on-the-fly by the kernel. So I would say it is probably similarly worthwhile to have hgemm and therefore hgemv to forward to internally.

[ChipKerchner]

As far as I see, there are vector instructions that accumulate directly in FP16.