perf: add contiguous fast path to complex add/subtract fallback loop by intel-python-devops · Pull Request #278 · IntelPython/mkl_umath

intel-python-devops · 2026-10-05T18:34:03Z

Note

This pull request is AI-generated. Please review thoroughly.

Adds a contiguous fast path to the complex add/subtract loops (CFLOAT, CDOUBLE) for calls that don't take the oneMKL VM path, i.e. contiguous arrays with n <= VML_ASM_THRESHOLD (100000). Before this change those calls fell through to the strided BINARY_LOOP.

The branch is gated on can_vectorize: contiguous, and each input either disjoint from the output or exactly the output. Partially overlapping operands, such as accumulate/cumsum (out == in1 + 1), still take BINARY_LOOP.
Real and imaginary parts are processed as one flat run of 2*n scalars. Since +/- is componentwise and each element is computed independently, results are identical to the previous loop for the layouts this branch accepts.
out is in1 and out is in2 get their own two-pointer loops. With a single three-pointer loop, the compiler's runtime alias check sees out == in as an overlap and runs its scalar fallback, which made in-place calls slower than main (icx, complex64: 0.54–0.66x at n = 1k–100k).
No vectorization pragma. NPY_PRAGMA_VECTOR is #pragma GCC ivdep under GCC, which is unsafe for partial overlap. Under icx it did not change codegen and added two -Wpass-failed warnings for CDOUBLE. The icx warning count is now the same as main (91).

Codegen (icx 2026.1.1, -qopt-report=3, conda-forge CFLAGS -march=nocona -mtune=haswell -O2):

complex64: all three loops are vectorized (VL 4), with a runtime dependence check.
complex128: not loop-vectorized ("vectorization possible but seems inefficient" at 128-bit width); icx unrolls and SLP-vectorizes it instead. The complex128 gains below come from that.

Tests: test_complex_contig covers out-of-place, out=a and out=b at n = 1, 7, 100, 1001 with a bitwise comparison to NumPy. test_accumulate_complex covers the partial-overlap path; it fails on a GCC build of the previous revision of this PR (acced6b) and passes with this one.

jharlow-intel · 2026-10-05T18:46:38Z

unsure if auto-generated PRs like this should be putting these changes into the CHANGELOG or not. Thoughts?

ndgrigorian · 2026-10-05T18:59:16Z

unsure if auto-generated PRs like this should be putting these changes into the CHANGELOG or not. Thoughts?

Tough call, I would say ideally yes but I would want to make sure we have some kind of anti-AI-slop-text skill or guard before that

antonwolfy · 2026-10-06T11:54:37Z

            *oi @OP@= ri;
            return;
        }
+        else if (contig) {


Wrong results for complex accumulate/cumsum in GCC builds.

What NumPy sends this branch:

np.add.accumulate and np.cumsum call the inner loop with out = in1 + 1 element and all strides contiguous (NumPy v2.5.3 ufunc_object.c L3043–3078).

So contig is 1 and can_vectorize is 0. That lands in the new branch, where each output depends on the result two scalars earlier.

Why that breaks under GCC:

NPY_PRAGMA_VECTOR becomes #pragma GCC ivdep, which promises the compiler there is no such dependency. GCC then drops its runtime overlap check: -fopt-info-vec shows the new loop vectorized with no versioning, while the old loop was "versioned … because of possible aliasing".

The three existing uses of this pragma (lines 355, 514, 668) all run behind DISJOINT_OR_SAME checks. fast_loop_macros.h:205 says ivdep must only be used after that kind of check. The new comment itself says operands may be "overlapping".

The repro: a GCC build of the PR gives mkl_umath.add.accumulate on complex64 [1..8] = [1, 3, -3.62, 0.38, 5, 11, 7, 15]. Expected is [1, 3, 6, 10, …]. The result includes uninitialized output memory, so it's nondeterministic.

Shipped builds use icx, so they're fine today. But meson.build accepts GCC, the README says "a C compiler", and a plain pip install . on Linux would use GCC.

Suggested change

else if (contig) {

else if (can_vectorize) {

This matches the other three pragma sites and the overlap guard NumPy itself uses before its fast path. It still covers np.add(Z, C, Z).

Reproduced on GCC 9.4. Fixed: now gated on can_vectorize, pragma removed. test_accumulate_complex covers it.

antonwolfy · 2026-10-06T12:00:20Z

            *oi @OP@= ri;
            return;
        }
+        else if (contig) {


We need a real benchmark numbers in the PR before it merges, including both out-of-place with no overlap and in-place paths

Added to the description. In-place numbers showed a regression on icx (0.54–0.66x), so in-place calls now get their own loops (1.2–2.1x).

antonwolfy · 2026-10-06T12:02:50Z

            *oi @OP@= ri;
            return;
        }
+        else if (contig) {


The PR text and code comment don't match what was measured:

"Bit-for-bit identical for every stride, overlap…": false.

"Lets the compiler autovectorize": only true for complex64. For complex128, icx reports "seems inefficient" and only unrolls the loop. This adds 2 new -Wpass-failed warnings for the CDOUBLE add and subtract loops.

No measurements: the PR gives no benchmark numbers.

np.add(Z, C, Z) isn't an overlap case: the output is the same array as an input, which can_vectorize already allows.

Agreed. Description and comment are rewritten. Removing the pragma also removes the two -Wpass-failed warnings, and complex128 gains are attributed to unroll + SLP.

antonwolfy · 2026-10-06T12:04:05Z

            *oi @OP@= ri;
            return;
        }
+        else if (contig) {


Missing tests coverage for the new branch

Added test_complex_contig and test_accumulate_complex.

antonwolfy · 2026-10-06T12:04:29Z

+            const @ftype@ *ip1 = (@ftype@ *)args[0];
+            const @ftype@ *ip2 = (@ftype@ *)args[1];
+            @ftype@ *op1 = (@ftype@ *)args[2];


Suggested change

const @ftype@ *ip1 = (@ftype@ *)args[0];

const @ftype@ *ip2 = (@ftype@ *)args[1];

@ftype@ *op1 = (@ftype@ *)args[2];

const @ftype@ *ip1 = (const @ftype@ *)args[0];

const @ftype@ *ip2 = (const @ftype@ *)args[1];

@ftype@ *op1 = (const @ftype@ *)args[2];

Applied to ip1/ip2. op1 stays non-const since it's written through.

antonwolfy · 2026-10-06T12:04:54Z

            *oi @OP@= ri;
            return;
        }
+        else if (contig) {


Need to populate the changelog

- Take the flat loop only when operands are disjoint or exactly in-place; accumulate (out == in1 + 1) goes back to the strided loop. Under GCC, NPY_PRAGMA_VECTOR (ivdep) gave wrong accumulate results. - Drop the pragma; it did not change icx codegen and added two -Wpass-failed warnings for CDOUBLE. - Give out == in1 and out == in2 their own loops so the runtime alias check does not send in-place calls to the scalar fallback. - Add tests for the contiguous and accumulate paths, and a changelog entry.

perf: add contiguous fast path to complex add/subtract fallback loop

acced6b

intel-python-devops requested review from antonwolfy, jharlow-intel, ndgrigorian, vlad-perevezentsev and xaleryb as code owners October 5, 2026 18:34

jharlow-intel self-assigned this Oct 5, 2026

antonwolfy reviewed Oct 6, 2026

View reviewed changes

Conversation

intel-python-devops commented Oct 5, 2026 • edited by jharlow-intel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jharlow-intel commented Oct 5, 2026

Uh oh!

ndgrigorian commented Oct 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

intel-python-devops commented Oct 5, 2026 •

edited by jharlow-intel

Loading