[SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference#57183
[SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference#57183HyukjinKwon wants to merge 2 commits into
Conversation
dongjoon-hyun
left a comment
There was a problem hiding this comment.
This PR seems to break the CI at describe function unit test.
2026-07-10T10:04:14.2381680Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m- describe function *** FAILED *** (115 milliseconds)�[0m�[0m
2026-07-10T10:04:14.2390933Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m Results do not match for query:�[0m�[0m
2026-07-10T10:04:14.2436987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=311,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]�[0m�[0m
2026-07-10T10:04:14.2449058Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m Timezone Env: �[0m�[0m
2026-07-10T10:04:14.2450581Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m �[0m
2026-07-10T10:04:14.2451468Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m == Parsed Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2452454Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m 'DescribeFunction true�[0m�[0m
2026-07-10T10:04:14.2453548Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m +- 'UnresolvedFunctionName [^], DESCRIBE FUNCTION�[0m�[0m
2026-07-10T10:04:14.2454477Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m �[0m
2026-07-10T10:04:14.2455282Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m == Analyzed Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2456358Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m function_desc: string�[0m�[0m
2026-07-10T10:04:14.2458012Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m DescribeFunctionCommand org.apache.spark.sql.catalyst.expressions.ExpressionInfo@13541f24, true�[0m�[0m
2026-07-10T10:04:14.2459987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m �[0m
2026-07-10T10:04:14.2460987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m == Optimized Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2463845Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m CommandResult [function_desc#1140035], Execute DescribeFunctionCommand, [[Function: ^], [Class: org.apache.spark.sql.catalyst.expressions.BitwiseXor], [Usage: expr1 ^ expr2 - Returns the result of bitwise exclusive OR of `expr1` and `expr2`.], [Extended Usage:�[0m�[0m
2026-07-10T10:04:14.2466350Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m Arguments:�[0m�[0m
2026-07-10T10:04:14.2467543Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m * expr1 - The first operand of the bitwise exclusive OR.�[0m�[0m
2026-07-10T10:04:14.2468867Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m An expression that evaluates to an integral.�[0m�[0m
2026-07-10T10:04:14.2470447Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m * expr2 - The second operand of the bitwise exclusive OR.�[0m�[0m
2026-07-10T10:04:14.2471784Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m An expression that evaluates to an integral.�[0m�[0m
2026-07-10T10:04:14.2472804Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m �[0m
2026-07-10T10:04:14.2473892Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m Examples:�[0m�[0m
2026-07-10T10:04:14.2474855Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m > SELECT 3 ^ 5;�[0m�[0m
2026-07-10T10:04:14.2475761Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m 6�[0m�[0m
2026-07-10T10:04:14.2476546Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m �[0m
2026-07-10T10:04:14.2477361Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m Since: 1.4.0�[0m�[0m
2026-07-10T10:04:14.2478191Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m ]]�[0m�[0m
2026-07-10T10:04:14.2479891Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m +- DescribeFunctionCommand org.apache.spark.sql.catalyst.expressions.ExpressionInfo@13541f24, true�[0m�[0m
2026-07-10T10:04:14.2481555Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m �[0m
…unctions in the SQL function reference ### What changes were proposed in this pull request? This documents, for every built-in function argument, which input types it accepts, in the SQL `@ExpressionDescription` `arguments` blocks (rendered on the SQL functions reference page), using a numeric umbrella (`a numeric` / `a numeric or interval`) where numeric subtypes cast in, and `timestamp` for TIMESTAMP_NTZ in type phrases. Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only the SQL surface. ### Why are the changes needed? The SQL function reference previously had no per-argument type information for most built-in functions. This fills that gap. ### Does this PR introduce _any_ user-facing change? Yes, documentation only. No behavioral or API change. ### How was this patch tested? Existing `ExpressionInfoSuite` and the SPARK-32870 argument-block-format tests pass. Types were derived from and verified against the analyzer's accept/reject behavior and each expression's `inputTypes` / `checkInputDataTypes`. Co-authored-by: Isaac
c6cd128 to
99fbac9
Compare
MaxGekk
left a comment
There was a problem hiding this comment.
0 blocking, 2 non-blocking, 0 nits.
Careful, high-quality docs pass. I built the SQL functions API page (api/sql/) and confirmed the arguments blocks this PR populates render there, so the rationale holds. Two minor consistency items in the added text.
Correctness (1)
- datetimeExpressions.scala:1158:
year/quarter/month/dayofmonth/dayofweek/dayofyearprose says "date or timestamp" but the type phrase says only "a date"; timestamp is accepted (verified) — see inline
Suggestions (1)
- Between.scala:38:
upper(andarrays_zipa2, tuple-sketchlgNomEntries/mode) omit the type line their sibling args got — see inline
Verification
Built the SQL functions API page via sql/gen-sql-api-docs.py (the generator behind spark.apache.org/docs/latest/api/sql/): its ExpressionInfo carries an arguments field and it emits a Arguments: block reproducing the text verbatim — rendered blocks appear (44 in datetime, 22 in string, etc.), so the type sentences this PR adds will show there once compiled. Separately confirmed year(TIMESTAMP), dayofyear, weekofyear all evaluate against a local build, so "a date" is narrower than the accepted types.
…g arg type lines (Between.upper, arrays_zip, tuple sketch, kll k), fix indentation, and qualify to_char constant note Co-authored-by: Isaac
What changes were proposed in this pull request?
This documents, for every built-in function argument, which input types it accepts, in the SQL
@ExpressionDescriptionargumentsblocks (rendered on the SQL functions reference page), using anumeric umbrella (
a numeric/a numeric or interval) where numeric subtypes cast in, andtimestampfor TIMESTAMP_NTZ in type phrases.Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only
the SQL surface.
Why are the changes needed?
The SQL function reference previously had no per-argument type information for most built-in
functions. This fills that gap.
Does this PR introduce any user-facing change?
Yes, documentation only. No behavioral or API change.
How was this patch tested?
Existing
ExpressionInfoSuiteand the SPARK-32870 argument-block-format tests pass. Types werederived from and verified against the analyzer's accept/reject behavior and each expression's
inputTypes/checkInputDataTypes.Co-authored-by: Isaac