Describe the bug
restricted_column added in #24520 counts the distinct values in an IN list using Vec::contains making restricted_column quadratic.
ScalarValue already implements Hash and Eq so count can be performed via HashSet making restricted_column run linearly.
+ This happens even for tables without statistics since in unique_match_limit restricted_columnis called before the check on whether the column's statistics say each value is unique. Adding another check on all columns can allow skipping counting altogether:
fn unique_match_limit(
predicate: &Arc<dyn PhysicalExpr>,
statistics: &Statistics,
) -> Option<usize> {
if !statistics
.column_statistics
.iter()
.any(|column| holds_each_value_once(column, &statistics.num_rows))
{
return None;
}
let mut limit: Option<usize> = None;
...
To Reproduce
Plan with IN list of different sizes and observe quadratic runtime.
Expected behavior
Planning time grows linearly with the size of IN-list.
Additional context
No response
Describe the bug
restricted_columnadded in #24520 counts the distinct values in anIN listusingVec::containsmakingrestricted_columnquadratic.ScalarValuealready implementsHashandEqso count can be performed via HashSet makingrestricted_columnrun linearly.+ This happens even for tables without statistics since in
unique_match_limitrestricted_columnis called before the check on whether the column's statistics say each value is unique. Adding another check on all columns can allow skipping counting altogether:To Reproduce
Plan with
IN listof different sizes and observe quadratic runtime.Expected behavior
Planning time grows linearly with the size of
IN-list.Additional context
No response