[MINOR][PYTHON][DOCS] Add example to Column.outer docstring#57199
[MINOR][PYTHON][DOCS] Add example to Column.outer docstring#57199CynicDog wants to merge 3 commits into
Conversation
| |name|salary|department_id| | ||
| +----+------+-------------+ | ||
| | Bob| 6000| 101| | ||
| +----+------+-------------+ |
There was a problem hiding this comment.
Nit: the 3-row dataset puts a single employee (Charlie, 4500) in dept 102, so avg == 4500 and Charlie fails the strict > filter, leaving dept 102 with no qualifying row. The shown output is a single row (Bob), which a reader may find slightly opaque (why are two of three names absent?). Consider mirroring DataFrame.scalar() Example 2's richer dataset so both departments yield at least one qualifying row. The math is correct, this is just a readability suggestion.
There was a problem hiding this comment.
Good catch, done — switched to DataFrame.scalar()s Example 2 dataset, both departments now have a qualifying row.
uros-b
left a comment
uros-b
left a comment
There was a problem hiding this comment.
Left just one comment, otherwise LGTM
Column.outer had no worked example showing how it's used to build a correlated subquery, unlike the other Column methods in this file.
The 3-row dataset left department 102 with only one employee, whose salary equaled the department average, so it never passed the filter. Mirror DataFrame.scalar()'s Example 2 dataset instead, so both departments have a qualifying row and the output isn't just one line.
6914653 to
ac9304c
Compare
|
@HyukjinKwon, @uros-b All checks are green! Thanks for the review and the positive feedback 😊 |
3 participants
What changes were proposed in this pull request?
Adds an
Examplessection toColumn.outer()'s docstring inpython/pyspark/sql/column.py. Every otherColumnmethod in this file has a worked example;outer()was the one exception.The example builds a small
employeesDataFrame and uses.outer()inside a correlated scalar subquery to filter employees earning above their department's average salary.Why are the changes needed?
outer()is easy to misunderstand from the description alone, since its effect (marking a column so it resolves against an outer query) only makes sense in the context of a correlated subquery. A concrete example makes that immediately clear, consistent with the rest of the file.Does this PR introduce any user-facing change?
No. Documentation-only change (a docstring example).
How was this patch tested?
Built Spark locally (
build/sbt -Phive package) and ran theColumn.outerdoctest directly against the built jars to confirm the example's output matches exactly.Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored. The code example and documentation text were authored by me, but I used Claude as a peer-review to validate the clarity of the example and check for edge cases before submission.