Skip to content

Complete SeaDatabricksClient (Execution Phase) #580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 355 commits into
base: sea-migration
Choose a base branch
from
Draft

Complete SeaDatabricksClient (Execution Phase) #580

wants to merge 355 commits into from

Conversation

varun-edachali-dbx
Copy link
Collaborator

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

Implement execution relevant methods (as defined in the backend interface) for the SEA client.

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing
https://databricks.atlassian.net/browse/PECOBLR-483?atlOrigin=eyJpIjoiODU3ODU0ZjEwN2VkNDI1Y2E4ODFiZDBlMDY4MmNjYjMiLCJwIjoiaiJ9

jprakash-db and others added 30 commits May 28, 2025 03:42
@jprakash-db @jackyhu-db @varun-edachali-dbx
PySQL Connector split into connector and sqlalchemy (#444)
904c304 
* Modified the gitignore file to not have .idea file

* [PECO-1803] Splitting the PySql connector into the core and the non core part (#417)

* Implemented ColumnQueue to test the fetchall without pyarrow

Removed token

removed token

* order of fields in row corrected

* Changed the folder structure and tested the basic setup to work

* Refractored the code to make connector to work

* Basic Setup of connector, core and sqlalchemy is working

* Basic integration of core, connect and sqlalchemy is working

* Setup working dynamic change from ColumnQueue to ArrowQueue

* Refractored the test code and moved to respective folders

* Added the unit test for column_queue

Fixed __version__

Fix

* venv_main added to git ignore

* Added code for merging columnar table

* Merging code for columnar

* Fixed the retry_close sesssion test issue with logging

* Fixed the databricks_sqlalchemy tests and introduced pytest.ini for the sqla_testing

* Added pyarrow_test mark on pytest

* Fixed databricks.sqlalchemy to databricks_sqlalchemy imports

* Added poetry.lock

* Added dist folder

* Changed the pyproject.toml

* Minor Fix

* Added the pyarrow skip tag on unit tests and tested their working

* Fixed the Decimal and timestamp conversion issue in non arrow pipeline

* Removed not required files and reformatted

* Fixed test_retry error

* Changed the folder structure to src / databricks

* Removed the columnar non arrow flow to another PR

* Moved the README to the root

* removed columnQueue instance

* Revmoved databricks_sqlalchemy dependency in core

* Changed the pysql_supports_arrow predicate, introduced changes in the pyproject.toml

* Ran the black formatter with the original version

* Extra .py removed from all the __init__.py files names

* Undo formatting check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* Check

* BIG UPDATE

* Refeactor code

* Refractor

* Fixed versioning

* Minor refractoring

* Minor refractoring

* Changed the folder structure such that sqlalchemy has not reference here

* Fixed README.md and CONTRIBUTING.md

* Added manual publish

* On push trigger added

* Manually setting the publish step

* Changed versioning in pyproject.toml

* Bumped up the version to 4.0.0.b3 and also changed the structure to have pyarrow as optional

* Removed the sqlalchemy tests from integration.yml file

* [PECO-1803] Print warning message if pyarrow is not installed (#468)

Print warning message if pyarrow is not installed

Signed-off-by: Jacky Hu <[email protected]>

* [PECO-1803] Remove sqlalchemy and update README.md (#469)

Remove sqlalchemy and update README.md

Signed-off-by: Jacky Hu <[email protected]>

* Removed all sqlalchemy related stuff

* generated the lock file

* Fixed failing tests

* removed poetry.lock

* Updated the lock file

* Fixed poetry numpy 2.2.2 issue

* Workflow fixes

---------

Signed-off-by: Jacky Hu <[email protected]>
Co-authored-by: Jacky Hu <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Removed CI CD for python3.8 (#490)
e5ac8c6 
* Removed python3.8 support

* Minor fix

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Added CI CD upto python 3.12 (#491)
e06fd2f 
Support for Py till 3.12

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Merging changes from v3.7.1 release (#488)
7126f97 
* Increased the number of retry attempts allowed  (#486)

Updated the number of attempts allowed

* bump version to 3.7.1 (#487)

bumped up version

* Refractore

* Minor change

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Bumped up to version 4.0.0 (#493)
b1245da 
bumped up the version

Signed-off-by: varun-edachali-dbx <[email protected]>
@dhirschfeld @dbaxa @varun-edachali-dbx
Support Python 3.13 and update deps (#510)
b54b04a 
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
@samikshya-db @varun-edachali-dbx
Improve debugging + fix PR review template (#514)
e1d7f71 
* Improve debugging + add PR review template

* case sensitivity of PR template

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Forward porting all changes into 4.x.x. uptil v3.7.3 (#529)
1c721e0 
* Base changes

* Black formatter

* Cache version fix

* Added the changed test_retry.py file

* retry_test_mixins changes

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Updated the actions/cache version (#532)
f1fa67a 
Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Updated the CODEOWNERS (#531)
512d37c 
Updated the codeowners

Signed-off-by: varun-edachali-dbx <[email protected]>
@shivam2680 @varun-edachali-dbx
Add version check for urllib3 in backoff calculation (#526)
ff0ec64 
Signed-off-by: Shivam Raj <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Support multiple timestamp formats in non arrow flow (#533)
c5900c9 
* Added check for 2 formats

* Wrote unit tests

* Added more supporting formats

* Added the T format datetime

* Added more timestamp formats

* Added python-dateutil library

Signed-off-by: varun-edachali-dbx <[email protected]>
@shivam2680 @varun-edachali-dbx
prepare release for v4.0.1 (#534)
c86e99b 
Signed-off-by: Shivam Raj <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Relaxed bound for python-dateutil (#538)
bcd6f01 
Changed bound for python-datetutil

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Bumped up the version for 4.0.2 (#539)
ca51d1d 
Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Added example for async execute query (#537)
d403fcd 
Added examples and fixed the async execute not working without pyarrow

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Added urllib3 version check (#547)
1b4154c 
* Added version check

* Removed packaging

Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Bump version to 4.0.3 (#549)
eac0433 
Updated the version to 4.0.3

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
decouple session class from existing Connection
1b0fc9b 
ensure maintenance of current APIs of Connection while delegating
responsibility

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
add open property to Connection to ensure maintenance of existing API
b45871e 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
update unit tests to address ThriftBackend through session instead of…
561c351 
… through Connection

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
chore: move session specific tests from test_client to test_session
a3188ad 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
formatting (black)
ff7abf6 
as in CONTRIBUTING.md

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
use connection open property instead of long chain through session
e7e2333 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
trigger integration workflow
fd4fae6 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
fix: ensure open attribute of Connection never fails
677e66a 
in case the openSession takes long, the initialisation of the session
will not complete immediately. This could make the session attribute
inaccessible. If the Connection is deleted in this time, the open()
check will throw because the session attribute does not exist. Thus, we
default to the Connection being closed in this case. This was not an
issue before because open was a direct attribute of the Connection
class. Caught in the integration tests.

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
introduce databricksClient interface and thrift backend implementation
4ec8703 
WARNING: incomplete - result set still aligned heavily with thrift and
the necessity of some defined functions is to be validated

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
change names of ThriftBackend -> ThriftDatabricksClient in tests
6ecb6bf 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
formatting: black + re-organise backend into new dir
fe2ce17 
Signed-off-by: varun-edachali-dbx <[email protected]>
@jprakash-db @varun-edachali-dbx
Update CODEOWNERS (#562)
568b1f4 
new codeowners

Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx varun-edachali-dbx changed the base branch from main to sea-migration June 9, 2025 04:00
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
fix imports in tests
d3d3fc2 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
remove accidental changes in workflows
53bc6ac 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
formatting (black)
40f15a7 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
account for new exec resp
aba27ab 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
account for new exec resp
4148aac 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
op_state -> status in ResultSet
0d51d1b 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
formatting (black)
d80ea49 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
change test name in example script
ac381f3 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx
cleaner, organised test scripts
92b778c 
Signed-off-by: varun-edachali-dbx <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 9, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deeksha-db deeksha-db Awaiting requested review from deeksha-db deeksha-db will be requested when the pull request is marked ready for review deeksha-db is a code owner

@samikshya-db samikshya-db Awaiting requested review from samikshya-db samikshya-db will be requested when the pull request is marked ready for review samikshya-db is a code owner

@jprakash-db jprakash-db Awaiting requested review from jprakash-db jprakash-db will be requested when the pull request is marked ready for review jprakash-db is a code owner

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db jackyhu-db will be requested when the pull request is marked ready for review jackyhu-db is a code owner

@madhav-db madhav-db Awaiting requested review from madhav-db madhav-db will be requested when the pull request is marked ready for review madhav-db is a code owner

@gopalldb gopalldb Awaiting requested review from gopalldb gopalldb will be requested when the pull request is marked ready for review gopalldb is a code owner

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db jayantsing-db will be requested when the pull request is marked ready for review jayantsing-db is a code owner

@vikrantpuppala vikrantpuppala Awaiting requested review from vikrantpuppala vikrantpuppala will be requested when the pull request is marked ready for review vikrantpuppala is a code owner

@shivam2680 shivam2680 Awaiting requested review from shivam2680 shivam2680 will be requested when the pull request is marked ready for review shivam2680 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.