Complete SeaDatabricksClient (Execution Phase) #580

varun-edachali-dbx · 2025-06-02T05:02:25Z

What type of PR is this?

Refactor
Feature
Bug Fix
Other

Description

Implement execution relevant methods (as defined in the backend interface) for the SEA client.

How is this tested?

Unit tests
E2E Tests
Manually
N/A

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing
https://databricks.atlassian.net/browse/PECOBLR-483?atlOrigin=eyJpIjoiODU3ODU0ZjEwN2VkNDI1Y2E4ODFiZDBlMDY4MmNjYjMiLCJwIjoiaiJ9

* Modified the gitignore file to not have .idea file * [PECO-1803] Splitting the PySql connector into the core and the non core part (#417) * Implemented ColumnQueue to test the fetchall without pyarrow Removed token removed token * order of fields in row corrected * Changed the folder structure and tested the basic setup to work * Refractored the code to make connector to work * Basic Setup of connector, core and sqlalchemy is working * Basic integration of core, connect and sqlalchemy is working * Setup working dynamic change from ColumnQueue to ArrowQueue * Refractored the test code and moved to respective folders * Added the unit test for column_queue Fixed __version__ Fix * venv_main added to git ignore * Added code for merging columnar table * Merging code for columnar * Fixed the retry_close sesssion test issue with logging * Fixed the databricks_sqlalchemy tests and introduced pytest.ini for the sqla_testing * Added pyarrow_test mark on pytest * Fixed databricks.sqlalchemy to databricks_sqlalchemy imports * Added poetry.lock * Added dist folder * Changed the pyproject.toml * Minor Fix * Added the pyarrow skip tag on unit tests and tested their working * Fixed the Decimal and timestamp conversion issue in non arrow pipeline * Removed not required files and reformatted * Fixed test_retry error * Changed the folder structure to src / databricks * Removed the columnar non arrow flow to another PR * Moved the README to the root * removed columnQueue instance * Revmoved databricks_sqlalchemy dependency in core * Changed the pysql_supports_arrow predicate, introduced changes in the pyproject.toml * Ran the black formatter with the original version * Extra .py removed from all the __init__.py files names * Undo formatting check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * Check * BIG UPDATE * Refeactor code * Refractor * Fixed versioning * Minor refractoring * Minor refractoring * Changed the folder structure such that sqlalchemy has not reference here * Fixed README.md and CONTRIBUTING.md * Added manual publish * On push trigger added * Manually setting the publish step * Changed versioning in pyproject.toml * Bumped up the version to 4.0.0.b3 and also changed the structure to have pyarrow as optional * Removed the sqlalchemy tests from integration.yml file * [PECO-1803] Print warning message if pyarrow is not installed (#468) Print warning message if pyarrow is not installed Signed-off-by: Jacky Hu <[email protected]> * [PECO-1803] Remove sqlalchemy and update README.md (#469) Remove sqlalchemy and update README.md Signed-off-by: Jacky Hu <[email protected]> * Removed all sqlalchemy related stuff * generated the lock file * Fixed failing tests * removed poetry.lock * Updated the lock file * Fixed poetry numpy 2.2.2 issue * Workflow fixes --------- Signed-off-by: Jacky Hu <[email protected]> Co-authored-by: Jacky Hu <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

* Removed python3.8 support * Minor fix Signed-off-by: varun-edachali-dbx <[email protected]>

Support for Py till 3.12 Signed-off-by: varun-edachali-dbx <[email protected]>

* Increased the number of retry attempts allowed (#486) Updated the number of attempts allowed * bump version to 3.7.1 (#487) bumped up version * Refractore * Minor change Signed-off-by: varun-edachali-dbx <[email protected]>

bumped up the version Signed-off-by: varun-edachali-dbx <[email protected]>

* Remove upper caps on dependencies (#452) * Remove upper caps on numpy and pyarrow versions Signed-off-by: David Black <[email protected]> * Added CI CD upto python 3.13 Signed-off-by: David Black <[email protected]> * Specify pandas 2.2.3 as the lower bound for python 3.13 Signed-off-by: David Black <[email protected]> * Specify pyarrow 18.0.0 as the lower bound for python 3.13 Signed-off-by: David Black <[email protected]> * Move `numpy` to dev dependencies Signed-off-by: Dave Hirschfeld <[email protected]> * Updated lockfile Signed-off-by: Dave Hirschfeld <[email protected]> --------- Signed-off-by: David Black <[email protected]> Signed-off-by: Dave Hirschfeld <[email protected]> Co-authored-by: David Black <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

* Improve debugging + add PR review template * case sensitivity of PR template Signed-off-by: varun-edachali-dbx <[email protected]>

* Base changes * Black formatter * Cache version fix * Added the changed test_retry.py file * retry_test_mixins changes Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

Updated the codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: Shivam Raj <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

* Added check for 2 formats * Wrote unit tests * Added more supporting formats * Added the T format datetime * Added more timestamp formats * Added python-dateutil library Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: Shivam Raj <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>

Changed bound for python-datetutil Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

Added examples and fixed the async execute not working without pyarrow Signed-off-by: varun-edachali-dbx <[email protected]>

* Added version check * Removed packaging Signed-off-by: varun-edachali-dbx <[email protected]>

Updated the version to 4.0.3 Signed-off-by: varun-edachali-dbx <[email protected]>

ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

… through Connection Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

as in CONTRIBUTING.md Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

in case the openSession takes long, the initialisation of the session will not complete immediately. This could make the session attribute inaccessible. If the Connection is deleted in this time, the open() check will throw because the session attribute does not exist. Thus, we default to the Connection being closed in this case. This was not an issue before because open was a direct attribute of the Connection class. Caught in the integration tests. Signed-off-by: varun-edachali-dbx <[email protected]>

WARNING: incomplete - result set still aligned heavily with thrift and the necessity of some defined functions is to be validated Signed-off-by: varun-edachali-dbx <[email protected]>

Signed-off-by: varun-edachali-dbx <[email protected]>

new codeowners Signed-off-by: varun-edachali-dbx <[email protected]>

github-actions · 2025-06-09T04:51:16Z