Data conversion when packing #5647

Activity

1. Mousius commented on Feb 18, 2026

   

   Mousius

   on Feb 18, 2026

   Contributor

   More actions

   Could we use this for an sgemm that uses bf16 internally? I.e. the memory shrinks during copy? 😸
2. martin-frbg commented on Mar 5, 2026

   

   martin-frbg

   on Mar 5, 2026

   Collaborator

   More actions

   I think I understand how that would be useful for some hardware, but it's not immediately clear to me how to integrate this into the current structure ?
   OTOH, an "sgemm that uses bf16 internally" sounds decidedly evil - surely this would be losing precision left and right
3. ChipKerchner commented on Mar 5, 2026

   

   ChipKerchner

   on Mar 5, 2026

   Last edited by ChipKerchner

   ContributorAuthor

   More actions

   Actually since the conversions would be done from BF16 or FP16 to FP32, I don't think there would be loss in precision. The GEMM would then be done in FP32
4. ChipKerchner commented on Mar 5, 2026

   

   ChipKerchner

   on Mar 5, 2026

   Last edited by ChipKerchner

   ContributorAuthor

   More actions

   Maybe we have a function that decides whether the conversion (packing) should be done in the input resolution or the output resolution. And then selects the correct GEMM based on that? I guess the packing would need to know the output resolution
5. MathieuCAISSA commented on Nov 2, 2026

   

   MathieuCAISSA

   on Nov 2, 2026

   More actions

   A data point for this, measured on x86-64 without bf16 instructions (Intel Core Ultra 7 155U, no AVX512_BF16 or AMX), where SBGEMM takes the generic path. Since 0.3.31 it runs about 20 times slower than before.

   cblas_sbgemm, 512 × 512 × 512, one thread, each release built with make TARGET=HASWELL BUILD_BFLOAT16=1 NO_LAPACK=1 NO_FORTRAN=1 USE_OPENMP=0:

   Version	GFLOP/s	c[0] (expected 512)
   0.3.28	1.84	512
   0.3.29	2.02	512
   0.3.30	1.87	512
   0.3.31	0.10	256
   0.3.32	0.11	256
   0.3.33	0.10	512
   0.3.34	0.10	512
   develop (0404990)	0.11	512

   The wrong results of 0.3.31 and 0.3.32 are gone since #5712. The speed has not come back. It matches what this issue describes: converting during packing (M×K + K×N conversions) rather than in the kernel (M×N×K) would recover it on hardware without bf16 instructions.

   For comparison, MKL's cblas_gemm_bf16bf16f32 on the same machine, at the same size and on one thread, reaches about 43 GFLOP/s at its fastest of 10 calls (4–14 GFLOP/s on a single first call, as the reproducer makes, which pays MKL's initialization). OpenBLAS 0.3.34 gives 0.12 GFLOP/s either way.

   Reproducer:

   #define _POSIX_C_SOURCE 200809L
   #include <stdio.h>
   #include <stdlib.h>
   #include <stdint.h>
   #include <time.h>
   #include <cblas.h>
   int main(void) {
       enum { n = 512 };
       uint16_t *a = malloc(n * n * 2), *b = malloc(n * n * 2);
       float *c = calloc(n * n, 4);
       for (int i = 0; i < n * n; i++) { a[i] = 0x3f80; b[i] = 0x3f80; }  /* 1.0 */
       openblas_set_num_threads(1);
       struct timespec t0, t1;
       clock_gettime(CLOCK_MONOTONIC, &t0);
       cblas_sbgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, n, n, n, 1.0f, a, n, b, n, 0.0f, c, n);
       clock_gettime(CLOCK_MONOTONIC, &t1);
       double s = (t1.tv_sec - t0.tv_sec) + 1e-9 * (t1.tv_nsec - t0.tv_nsec);
       printf("%s: c[0]=%g, %.3f s, %.2f GFLOP/s\n", openblas_get_config(), c[0], s, 2.0 * n * n * n / s / 1e9);
       return 0;
   }

   Found with blas-microbenchmark's sbgemm benchmark.
6. martin-frbg commented on Nov 2, 2026

   

   martin-frbg

   on Nov 2, 2026

   Collaborator

   More actions

   Thanks for the datapoint - however this looks suspiciously like more fallout from the introduction of BGEMM into the fallback gemm kernel that happened in 0.3.31. Chances are that the reuse of existing code introduced even more spurious type conversions into SBGEMM than #5712 removed
7. martin-frbg commented on Nov 3, 2026

   

   martin-frbg

   on Nov 3, 2026

   Collaborator

   More actions

   Bisecting between 0.3.30 and 0.3.31 revealed #5373 as the culprit - while mainly about NeoverseV1 BGEMM, it also replaced the local function for bf16/f32 conversion with a call to the sbf16tos utility function in a different translation unit.