[PECOBLR-330] Support for complex params #559

jprakash-db · 2025-05-19T04:44:06Z

Description

This PR introduces comprehensive support for complex parameter types (ARRAY and MAP) in the Databricks SQL Python client.

Key Changes

Added new parameter classes: ArrayParameter and MapParameter
These classes recursively wrap Python lists and dictionaries and serialize them into the appropriate Thrift structures.
The type inference logic dbsql_parameter_from_primitive is updated to recognize and construct these parameter types, including for nested complex values.
Added the TSparkParamValueArgs which is necessary to be passed as arguements for complex types

Testing

e2e

Added tests to insert and compare the results for nested structures ARRAY<ARRAY<STRING>>, ARRAY<MAP<STRING,INTEGER>>, MAP<STRING,ARRAY<STRING>>

unit

Added tests to verify the convertion of the params into the respective server contract format

`IGNORE THE FAILING e2e Tests`

github-actions · 2025-05-19T04:49:57Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-19T04:53:11Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Merged Main

github-actions · 2025-05-19T04:54:39Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-19T18:34:08Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-20T07:13:26Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-20T10:26:28Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Working e2e tests prototype

github-actions · 2025-05-22T05:04:33Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-05-22T16:04:53Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

src/databricks/sql/parameters/native.py

shivam2680 · 2025-06-06T11:47:24Z

src/databricks/sql/utils.py

@@ -429,7 +429,7 @@ def user_friendly_error_message(self, no_retry_reason, attempt, elapsed):
 # Taken from PyHive
 class ParamEscaper:
    _DATE_FORMAT = "%Y-%m-%d"
-    _TIME_FORMAT = "%H:%M:%S.%f"
+    _TIME_FORMAT = "%H:%M:%S.%f %z"


Timezone will not be there for TIMESTAMP_NTZ param.

+1, would like to know how we've accounted/tested for NTZ

If it is not there it will be an empty space, there is already a test suite that inserts NTZ and none NTZ and reads back to compare whether it is equal or not

@vikrantpuppala @shivam2680 There are already existing tests that insert NTZ and non NTZ values and reads back from table to ensure everything is working as expected -

databricks-sql-python/tests/e2e/test_parameterized_queries.py

Line 283 in 3842583

def test_dbsqlparameter_single(

src/databricks/sql/utils.py

tests/e2e/test_parameterized_queries.py

vikrantpuppala · 2025-06-09T05:00:07Z

src/databricks/sql/utils.py

@@ -429,7 +429,7 @@ def user_friendly_error_message(self, no_retry_reason, attempt, elapsed):
 # Taken from PyHive
 class ParamEscaper:
    _DATE_FORMAT = "%Y-%m-%d"
-    _TIME_FORMAT = "%H:%M:%S.%f"
+    _TIME_FORMAT = "%H:%M:%S.%f %z"


+1, would like to know how we've accounted/tested for NTZ

gopalldb

Is this not an opt-in feature, can't user disable this in favor of existing behavior?

github-actions · 2025-06-13T09:58:32Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jprakash-db · 2025-06-13T10:05:48Z

@gopalldb We are just adding support for more types, don't think there is anything to opt in. If the users don't use complex types they won't need this

github-actions · 2025-06-13T11:06:25Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

github-actions · 2025-06-13T12:11:30Z

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

jprakash-db added 5 commits April 17, 2025 00:57

Basic testing

65fe00d

testing examples

cfc58d6

Basic working prototype

7271866

ttypes fix

7c42faa

Refractored the ttypes

8e5038c

jprakash-db had a problem deploying to azure-prod May 19, 2025 04:49 — with GitHub Actions Failure

nit

ddbe54e

jprakash-db had a problem deploying to azure-prod May 19, 2025 04:53 — with GitHub Actions Failure

Merge branch 'main' into jprakash-db/complex-param

4dd26d9

Merged Main

jprakash-db had a problem deploying to azure-prod May 19, 2025 04:54 — with GitHub Actions Failure

Added inline support

71be1ed

jprakash-db had a problem deploying to azure-prod May 19, 2025 18:33 — with GitHub Actions Failure

Reordered boolean to be above int

b032c3f

jprakash-db temporarily deployed to azure-prod May 20, 2025 07:13 — with GitHub Actions Inactive

jprakash-db temporarily deployed to azure-prod May 20, 2025 10:26 — with GitHub Actions Inactive

jprakash-db temporarily deployed to azure-prod May 21, 2025 05:49 — with GitHub Actions Inactive

Check

37c89b8

Working e2e tests prototype

jprakash-db force-pushed the jprakash-db/complex-param branch from 3026b6f to 37c89b8 Compare May 22, 2025 05:04

jprakash-db had a problem deploying to azure-prod May 22, 2025 05:04 — with GitHub Actions Failure

More tests

b2b8a2a

jprakash-db had a problem deploying to azure-prod May 22, 2025 16:04 — with GitHub Actions Failure

Added unit tests

4c36d99

jprakash-db self-assigned this Jun 6, 2025

jprakash-db marked this pull request as ready for review June 6, 2025 06:48

jprakash-db requested review from deeksha-db, samikshya-db, jackyhu-db, madhav-db, gopalldb, jayantsing-db, vikrantpuppala and shivam2680 as code owners June 6, 2025 06:48

jprakash-db changed the title ~~Support for complex params~~ [PECOBLR-330] Support for complex params Jun 6, 2025

shivam2680 reviewed Jun 6, 2025

View reviewed changes

vikrantpuppala reviewed Jun 9, 2025

View reviewed changes

gopalldb reviewed Jun 9, 2025

View reviewed changes

nit

8346de6

jprakash-db temporarily deployed to azure-prod June 13, 2025 09:57 — with GitHub Actions Inactive

Merge branch 'main' into jprakash-db/complex-param

c22f6c7

jprakash-db had a problem deploying to azure-prod June 13, 2025 09:58 — with GitHub Actions Error

jprakash-db had a problem deploying to azure-prod June 13, 2025 10:10 — with GitHub Actions Failure

nit

1631142

jprakash-db had a problem deploying to azure-prod June 13, 2025 11:06 — with GitHub Actions Failure

jprakash-db requested review from vikrantpuppala, gopalldb and shivam2680 June 13, 2025 11:17

nit

3178332

jprakash-db deployed to azure-prod June 13, 2025 12:11 — with GitHub Actions Active

[PECOBLR-330] Support for complex params #559

Are you sure you want to change the base?

[PECOBLR-330] Support for complex params #559

Uh oh!

Conversation

jprakash-db commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Testing

e2e

unit

IGNORE THE FAILING e2e Tests

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 20, 2025

Uh oh!

github-actions bot commented May 20, 2025

Uh oh!

github-actions bot commented May 22, 2025

Uh oh!

github-actions bot commented May 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shivam2680 Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

vikrantpuppala Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

jprakash-db Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

jprakash-db Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vikrantpuppala Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

gopalldb left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 13, 2025

Uh oh!

jprakash-db commented Jun 13, 2025

Uh oh!

github-actions bot commented Jun 13, 2025

Uh oh!

github-actions bot commented Jun 13, 2025

Uh oh!

Uh oh!

jprakash-db commented May 19, 2025 •

edited

Loading

`IGNORE THE FAILING e2e Tests`