Description of performance issue
array_has(array, element) returns, for each row, whether array contains element.
When element is a scalar literal, DataFusion has a fast path (added in #20374). But when element is an column, e.g. array_has(t1.tags, t2.key) used as a join filter, execution goes through array_has_dispatch_for_array (the ColumnarValue::Array needle branch in datafusion/functions nested/src/array_has.rs).
That branch compares each row by invoking the Arrow eq comparison kernel once per row.
Every invocation allocates a BooleanArray and pays downcast + dispatch overhead, so for an N-row batch it is N kernel calls and N allocations, the cost is dominated by fixed per-row overhead, not the element comparison itself.
This is a common shape in real workloads: matching tags/labels/keys between two tables lowers to an array-needle array_has join filter (e.g. a NestedLoopJoinExec with filter=array_has(tags, key)). It was a visible fraction of one such profile even after the join's deep-copy was fixed in #18070 / #18161.
Describe the solution you'd like
Performance improvement for at least primitives, non-nulls, and/or more
Proposition/draft of improvement
PR with bench: #23335
PR with perf and bench results: #23337 (leaving on draft until get agreement that this should be optimised)
Additional context
Related: #20374, #18070 / #18161, #18727.
Description of performance issue
array_has(array, element)returns, for each row, whetherarraycontainselement.When
elementis a scalar literal, DataFusion has a fast path (added in #20374). But whenelementis an column, e.g.array_has(t1.tags, t2.key)used as a join filter, execution goes througharray_has_dispatch_for_array(theColumnarValue::Arrayneedle branch indatafusion/functions nested/src/array_has.rs).That branch compares each row by invoking the Arrow
eqcomparison kernel once per row.Every invocation allocates a
BooleanArrayand pays downcast + dispatch overhead, so for anN-row batch it isNkernel calls andNallocations, the cost is dominated by fixed per-row overhead, not the element comparison itself.This is a common shape in real workloads: matching tags/labels/keys between two tables lowers to an array-needle
array_hasjoin filter (e.g. aNestedLoopJoinExecwithfilter=array_has(tags, key)). It was a visible fraction of one such profile even after the join's deep-copy was fixed in #18070 / #18161.Describe the solution you'd like
Performance improvement for at least primitives, non-nulls, and/or more
Proposition/draft of improvement
PR with bench: #23335
PR with perf and bench results: #23337 (leaving on draft until get agreement that this should be optimised)
Additional context
Related: #20374, #18070 / #18161, #18727.