Skip to content

[SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference#57183

Open
HyukjinKwon wants to merge 2 commits into
apache:masterfrom
HyukjinKwon:split-func-types-sql
Open

[SPARK-58085][SQL][DOCS] Document accepted input types for built-in functions in the SQL function reference#57183
HyukjinKwon wants to merge 2 commits into
apache:masterfrom
HyukjinKwon:split-func-types-sql

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This documents, for every built-in function argument, which input types it accepts, in the SQL
@ExpressionDescription arguments blocks (rendered on the SQL functions reference page), using a
numeric umbrella (a numeric / a numeric or interval) where numeric subtypes cast in, and
timestamp for TIMESTAMP_NTZ in type phrases.

Subtask of SPARK-57999, split out from #57079 for reviewability. This PR covers only
the SQL surface.

Why are the changes needed?

The SQL function reference previously had no per-argument type information for most built-in
functions. This fills that gap.

Does this PR introduce any user-facing change?

Yes, documentation only. No behavioral or API change.

How was this patch tested?

Existing ExpressionInfoSuite and the SPARK-32870 argument-block-format tests pass. Types were
derived from and verified against the analyzer's accept/reject behavior and each expression's
inputTypes / checkInputDataTypes.

Co-authored-by: Isaac

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR seems to break the CI at describe function unit test.

2026-07-10T10:04:14.2381680Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m- describe function *** FAILED *** (115 milliseconds)�[0m�[0m
2026-07-10T10:04:14.2390933Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  Results do not match for query:�[0m�[0m
2026-07-10T10:04:14.2436987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=311,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]�[0m�[0m
2026-07-10T10:04:14.2449058Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  Timezone Env: �[0m�[0m
2026-07-10T10:04:14.2450581Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m
2026-07-10T10:04:14.2451468Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  == Parsed Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2452454Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  'DescribeFunction true�[0m�[0m
2026-07-10T10:04:14.2453548Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  +- 'UnresolvedFunctionName [^], DESCRIBE FUNCTION�[0m�[0m
2026-07-10T10:04:14.2454477Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m
2026-07-10T10:04:14.2455282Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  == Analyzed Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2456358Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  function_desc: string�[0m�[0m
2026-07-10T10:04:14.2458012Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  DescribeFunctionCommand org.apache.spark.sql.catalyst.expressions.ExpressionInfo@13541f24, true�[0m�[0m
2026-07-10T10:04:14.2459987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m
2026-07-10T10:04:14.2460987Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  == Optimized Logical Plan ==�[0m�[0m
2026-07-10T10:04:14.2463845Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  CommandResult [function_desc#1140035], Execute DescribeFunctionCommand, [[Function: ^], [Class: org.apache.spark.sql.catalyst.expressions.BitwiseXor], [Usage: expr1 ^ expr2 - Returns the result of bitwise exclusive OR of `expr1` and `expr2`.], [Extended Usage:�[0m�[0m
2026-07-10T10:04:14.2466350Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m      Arguments:�[0m�[0m
2026-07-10T10:04:14.2467543Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m        * expr1 - The first operand of the bitwise exclusive OR.�[0m�[0m
2026-07-10T10:04:14.2468867Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m          An expression that evaluates to an integral.�[0m�[0m
2026-07-10T10:04:14.2470447Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m        * expr2 - The second operand of the bitwise exclusive OR.�[0m�[0m
2026-07-10T10:04:14.2471784Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m          An expression that evaluates to an integral.�[0m�[0m
2026-07-10T10:04:14.2472804Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m    �[0m
2026-07-10T10:04:14.2473892Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m      Examples:�[0m�[0m
2026-07-10T10:04:14.2474855Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m        > SELECT 3 ^ 5;�[0m�[0m
2026-07-10T10:04:14.2475761Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m         6�[0m�[0m
2026-07-10T10:04:14.2476546Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m    �[0m
2026-07-10T10:04:14.2477361Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m      Since: 1.4.0�[0m�[0m
2026-07-10T10:04:14.2478191Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m  ]]�[0m�[0m
2026-07-10T10:04:14.2479891Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m     +- DescribeFunctionCommand org.apache.spark.sql.catalyst.expressions.ExpressionInfo@13541f24, true�[0m�[0m
2026-07-10T10:04:14.2481555Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m  �[0m

…unctions in the SQL function reference

### What changes were proposed in this pull request?

This documents, for every built-in function argument, which input types it accepts, in the SQL
`@ExpressionDescription` `arguments` blocks (rendered on the SQL functions reference page), using a
numeric umbrella (`a numeric` / `a numeric or interval`) where numeric subtypes cast in, and
`timestamp` for TIMESTAMP_NTZ in type phrases.

Subtask of SPARK-57999, split out from apache#57079 for reviewability. This PR covers only
the SQL surface.

### Why are the changes needed?

The SQL function reference previously had no per-argument type information for most built-in
functions. This fills that gap.

### Does this PR introduce _any_ user-facing change?

Yes, documentation only. No behavioral or API change.

### How was this patch tested?

Existing `ExpressionInfoSuite` and the SPARK-32870 argument-block-format tests pass. Types were
derived from and verified against the analyzer's accept/reject behavior and each expression's
`inputTypes` / `checkInputDataTypes`.

Co-authored-by: Isaac
@HyukjinKwon HyukjinKwon marked this pull request as draft July 12, 2026 22:09
@HyukjinKwon HyukjinKwon force-pushed the split-func-types-sql branch from c6cd128 to 99fbac9 Compare July 12, 2026 22:18
@HyukjinKwon HyukjinKwon marked this pull request as ready for review July 13, 2026 06:26

@MaxGekk MaxGekk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 2 non-blocking, 0 nits.
Careful, high-quality docs pass. I built the SQL functions API page (api/sql/) and confirmed the arguments blocks this PR populates render there, so the rationale holds. Two minor consistency items in the added text.

Correctness (1)

  • datetimeExpressions.scala:1158: year/quarter/month/dayofmonth/dayofweek/dayofyear prose says "date or timestamp" but the type phrase says only "a date"; timestamp is accepted (verified) — see inline

Suggestions (1)

  • Between.scala:38: upper (and arrays_zip a2, tuple-sketch lgNomEntries/mode) omit the type line their sibling args got — see inline

Verification

Built the SQL functions API page via sql/gen-sql-api-docs.py (the generator behind spark.apache.org/docs/latest/api/sql/): its ExpressionInfo carries an arguments field and it emits a Arguments: block reproducing the text verbatim — rendered blocks appear (44 in datetime, 22 in string, etc.), so the type sentences this PR adds will show there once compiled. Separately confirmed year(TIMESTAMP), dayofyear, weekofyear all evaluate against a local build, so "a date" is narrower than the accepted types.

…g arg type lines (Between.upper, arrays_zip, tuple sketch, kll k), fix indentation, and qualify to_char constant note

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants