[SPARK-52146][SQL] Detect cyclic function references in SQL UDFs #51626
+534
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
allisonwang-db
changed the title
allisonwang-db
commented
Jul 23, 2025
Comment on lines
+1671
to
+1675
| val outer = expr.transform { | ||
| case a: Attribute if a.resolved => OuterReference(a) | ||
| case o: OuterReference => OuterReference(o) | ||
| } | ||
| Alias(Cast(outer, param.dataType), param.name)( |
There was a problem hiding this comment.
cc @cloud-fan this is needed to make the plan structure valid when using OneRowRelation.
allisonwang-db
changed the title
allisonwang-db
force-pushed
the
spark-52146-cyc-func-usage
branch
from
30c74a8 to
eaee0a7
Compare
cloud-fan
reviewed
Jul 24, 2025
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
Show resolved
Hide resolved
cloud-fan
reviewed
Jul 24, 2025
| } | ||
| // Check cyclic reference using qualified function names. | ||
| val newPath = path :+ f.function.name | ||
| if (f.function.name == name) { |
There was a problem hiding this comment.
shall we consider case sensitivity?
There was a problem hiding this comment.
and is this name qualified?
There was a problem hiding this comment.
Function names are always case insensitive, and the name here is qualified. I've added a few more tests to cover both scenarios.
…log/SessionCatalog.scala Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
|
need to fix |
cloud-fan
approved these changes
Jul 25, 2025
| @@ -367,6 +373,61 @@ case class CreateSQLFunctionCommand( | |||
| } | |||
| } | |||
|
|
|||
| /** | |||
There was a problem hiding this comment.
Could you fix the indentation, @allisonwang-db ?
| CREATE OR REPLACE FUNCTION foo3_4a(x INT) RETURN FoO3_4b(x); | ||
| CREATE OR REPLACE FUNCTION foo3_4a(x INT) RETURNS INT RETURN SELECT SUM(a) FROM foo3_4e(x); | ||
| CREATE OR REPLACE FUNCTION foo3_4e(x INT) RETURNS TABLE (c INT) RETURN SELECT * FROM foo3_4f(x); | ||
| CREATE OR REPLACE FUNCTION foo3_4e(x INT) RETURNS TABLE RETURN SELECT * FROM fOo3_4F(x); |
There was a problem hiding this comment.
Thank you for adding this test coverage.
dongjoon-hyun
approved these changes
Jul 27, 2025
|
the last commit just fixed indentation, thanks, merging to master/4.0! |
cloud-fan
added a commit
that referenced
this pull request
Jul 28, 2025
This change adds logic to detect cyclic function references when creating SQL UDFs to prevent infinite analysis. To improve SQL UDF usability No New tests No Closes #51626 from allisonwang-db/spark-52146-cyc-func-usage. Lead-authored-by: Allison Wang <allison.wang@databricks.com> Co-authored-by: Allison Wang <allisonwang@apache.org> Co-authored-by: Wenchen Fan <cloud0fan@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 3ff28ae) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
|
Late LGTM. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This change adds logic to detect cyclic function references when creating SQL UDFs to prevent infinite analysis.
Why are the changes needed?
To improve SQL UDF usability
Does this PR introduce any user-facing change?
No
How was this patch tested?
New tests
Was this patch authored or co-authored using generative AI tooling?
No