It seems impossible to pin the threads

[rongcuid]

I have tried many methods to pin OpenBLAS threads (for microbenchmark purpose), but none of the implied methods work. I have tried every combination of:

• NO_AFFINITY=1 or NO_AFFINITY=0
• OPENBLAS_NUM_THREADS
• GOTOBLAS_MAIN_FREE, OPENBLAS_MAIN_FREE
• openblas_set_num_threads before any call, right before set affinity, and after set affinity.
• openblas_setaffinity with cpuset_t that are known to be correct on other microbenchmarks.
• Whether I pin the main thread or not.

No matter what I do, OpenBLAS uses more threads than I allow it, except in the one case of setting OPENBLAS_NUM_THREADS. It seems like it just automatically increases the amount of threads to use all hyperthreads on the physical cores, sometimes more.

How exactly does the OpenBlas thread affinity work? Am I supposed to set NO_AFFINITY to 1 or 0? Does openblas_set_num_threads have any effects in run time?

Here's my config:

OpenBLAS 0.3.30 NO_AFFINITY HASWELL MAX_THREADS=28

This is how I pin the threads (using hwloc):

    for (int i = 0; i < threads; ++i) {
      int core = i;
      int ht = pin_offset;
      hwloc_obj_t core_obj = hwloc_get_obj_by_type(topo, HWLOC_OBJ_CORE, core);
      if (!core_obj) {
        fprintf(stderr, "Failed to get core\n");
        exit(1);
      }
      hwloc_obj_t ht_obj = hwloc_get_obj_below_by_type(topo, HWLOC_OBJ_CORE,
                                                       core, HWLOC_OBJ_PU, ht);
      if (!ht_obj) {
        fprintf(stderr, "Failed to get thread\n");
        exit(1);
      }
      fprintf(stderr, "Logical core [%d:%d] is physical [%d:%d]\n", core, ht,
              core_obj->os_index, ht_obj->os_index);
      cpu_set_t cpu_set;
      CPU_ZERO(&cpu_set);
      hwloc_cpuset_to_glibc_sched_affinity(topo, ht_obj->cpuset, &cpu_set,
                                           sizeof(cpu_set));
      if (openblas_setaffinity(i, sizeof(cpu_set_t), &cpu_set)) {
        perror("openblas_setaffinity()");
        exit(1);
      }
    }

Is this a bug? A misuse? I have no idea.

[martin-frbg]

Thread count is not directly related to thread affinity, but OpenBLAS should honor openblas_set_num_threads(). Only when this was not called, will OpenBLAS try to use all available threads up to a maximum of OPENBLAS_NUM_THREADS in most situations. (Even with openblas_set_num_threads, you may see unused threads hang around for a short while - configurable via the environment variable OPENBLAS_THREAD_TIMEOUT - see Makefile.rule)

What platform is this, and are you certain that the library is not built with USE_OPENMP=1 (where setting affinity via openblas_setaffinity would not work, and you'd need to use the equivalent OMP environment variables to pin your threads to cores) ? The build-time parameter NO_AFFINITY only determines whether OpenBLAS should automatically pin its threads to the cores it sees on startup - this is usually a bad idea as it will constrain what the calling program and any libraries loaded by it later on can use, so NO_AFFINITY=1 is the default. Lastly, the openblas_setaffinity() helper function is just an interface to the pthread_setaffinity_np() system call - with a corresponding openblas_getaffinity to inquire

[rongcuid]

I've tried setting openblas_set_num_threads to exactly the number I want. I have not tried OPENBLAS_THREAD_TIMEOUT, I will try it.

I am certain I built with USE_OPENMP=0, because I have checked the reported config: OpenBLAS 0.3.30 NO_AFFINITY HASWELL MAX_THREADS=28

I am certain I have tried both options of NO_AFFINITY. Neither works. In fact, now that I look at my data again, I am not sure if OPENBLAS_NUM_THREADS is behaving correctly either.

All my other benchmarks use pthread_setaffinity_np under the hood, with the same CPU masks. They respect the CPU mask, OpenBLAS did not.

[martin-frbg]

seems to work fine at least with a trivial cpuset defined via CPU_SET, haven't had time to build hwloc

[rongcuid]

Can you try to use perf stat -e cpu-cycles,duration_time to verify? The cpu-cycles counter counts the total amount of cycles for all active CPUs, while the duration_time measures the wall clock in nanoseconds. If you use a CPU_SET that enables one hyperthread each on 2 cores, using the formula cycles / duration_time / num_threads, you should get your frequency. In my case, I almost always get double the expected amount, indicating that double the number of cores are active. Nothing I tried prevents that.