Skip to content

Remove upper caps on dependencies #452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 5, 2024

Conversation

arredond
Copy link
Contributor

@arredond arredond commented Oct 7, 2024

What

Like many Poetry-managed projects, this package sets upper caps on dependencies both via caret (^) and lesser-than signs (<=). This PR relaxes these requirements to just establishing lower bounds, not upper ones.

Why

Poetry heavily believes in SemVer but many projects do not strictly follow it. For instance, pyarrow has bumped major versions eight times in the last 2 years without really making breaking changes for the vast majority of users.

Many well-regarded devs in the Python ecosystem are calling against upper-bounds in projects due to the complications it generates for end users (here's an excellent discussion on the topic from a PyPa member).

Personally, I find myself in "dependency hell" in a project where we use SQLAlchemy to interface with many kinds of databases (the area where SQLAlchemy excels, for sure). The latest dialects we're trying to support require SQLAlchemy >= 2.0, but our current databricks-sql-connectorversion of 2.9.3 doesn't allow it. We also can't upgrade to the latest 3.4.0 because an upper limit on pyarrow has been set (which didn't exist before). A similar experience happens with Numpy >= 2.0.

Related issues / PRs

Many versions of this issue have popped up before for this package (especially for pyarrow and numpy):
#427
#55
#428
#74
#88
#432

Note

If this PR seems too ambitious, I'm happy to provide another one that only removes or, at least, bumps the upper cap for pyarrow and numpy. However, I believe removing upper caps in a general sense is the way to go. Major version bumps in Python are very frequently backwards compatible, and it's a much better experience for users to fix problems when they occur than to prevent them from upgrading (and eventually ditching your package altogether).

Here's a quote in the linked article from Brett Cannon, Python Steering Council Member and packaging expert:

Libraries/packages should be setting a floor, and if necessary excluding known buggy versions, but otherwise don’t cap the maximum version as you can’t predict future compatibility

Or another quote from the author of the article himself:

If you absolutely must set upper limits, you should release a new version as soon as possible with a higher cap when a dependency updates (ideally before the dependency releases the update). If you are committing to this, why not just quickly release a patch release with caps only after an actual conflict happens? It will be less common, and will help you quickly sort out and fix incompatibilities, rather than hiding your true compatibilities and delaying updates. You want users to use the latest versions of your libraries if there’s a problem, so why can’t you offer the same consideration to the libraries you depend on and use?

@arredond
Copy link
Contributor Author

@andrefurlan-db @benc-db @jackyhu-db @kravets-levko @rcypher-databricks @yunbodeng-db pinging any of you in case this has gone unseen

Copy link
Contributor

@susodapop susodapop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change LGTM and the motivation is a solid one.

@hauntsaninja
Copy link

@jackyhu-db any chance you could take a look at this? I'm interested in removing the preemptive cap for numpy 2 (see also #432 )

@jackyhu-db
Copy link
Collaborator

@arredond thanks for making the change. Can you reduce the scope of the change? Only change the pyarrow and numpy?

@arredond
Copy link
Contributor Author

arredond commented Dec 5, 2024

Of course, I'll make a change to this PR reducing the scope.

However, at the risk of repeating myself and being annoying, please consider removing all upper caps in the future. Numpy and pyarrow have been the problem this time but it's very likely that in the near future you'll be seeing this same issue with, say, requests.

Once again copying that quote from Brett Cannon in the PR description, since these well-established folks make the case much better than I do:

Libraries/packages should be setting a floor, and if necessary excluding known buggy versions, but otherwise don’t cap the maximum version as you can’t predict future compatibility

@jackyhu-db jackyhu-db requested review from jprakash-db and removed request for kravets-levko December 5, 2024 17:38
@jackyhu-db jackyhu-db merged commit d690516 into databricks:main Dec 5, 2024
10 of 11 checks passed
@arredond arredond deleted the remove-pyarrow-cap branch December 8, 2024 18:12
BaukJ pushed a commit to BaukJ/databricks-sql-python that referenced this pull request Dec 16, 2024
* Remove upper caps on numpy and pyarrow versions
alexmalins added a commit to alexmalins/harlequin that referenced this pull request Jan 7, 2025
I think the deps issues on locking could have come
from upper bounds on the pyarrow & numpy version
set by the databricks-sql-connector

Those upper bounds have gone in the latest
databricks-sql-connector release (3.7.0), which is a
requirement of the latest harlequin-databricks release
(0.5.2)

databricks/databricks-sql-python#452
tconbeer added a commit to tconbeer/harlequin that referenced this pull request Jan 7, 2025
* Bump target python version to 3.9 for ruff/mypy

* Restore harlequin-databricks as optional dep

I think the deps issues on locking could have come
from upper bounds on the pyarrow & numpy version
set by the databricks-sql-connector

Those upper bounds have gone in the latest
databricks-sql-connector release (3.7.0), which is a
requirement of the latest harlequin-databricks release
(0.5.2)

databricks/databricks-sql-python#452

* Regenerate lock file

* Fix changelog

* fix: regenerate lockfile with poetry 1.8

* fix: bump pytest snapshot plugin version

---------

Co-authored-by: Alex Malins <[email protected]>
dbaxa added a commit to dbaxa/databricks-sql-python that referenced this pull request Feb 11, 2025
* Remove upper caps on numpy and pyarrow versions
dbaxa added a commit to dbaxa/databricks-sql-python that referenced this pull request Feb 11, 2025
* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>
dbaxa added a commit to dbaxa/databricks-sql-python that referenced this pull request Feb 12, 2025
* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>
samikshya-db pushed a commit that referenced this pull request Feb 27, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
wyattscarpenter pushed a commit to wyattscarpenter/databricks-sql-python that referenced this pull request Mar 11, 2025
* Remove upper caps on dependencies (databricks#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: wyattscarpenter <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 27, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 28, 2025
* Remove upper caps on numpy and pyarrow versions

Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 28, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 28, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 28, 2025
* Remove upper caps on numpy and pyarrow versions

Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 28, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
varun-edachali-dbx pushed a commit that referenced this pull request May 28, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
saishreeeee pushed a commit that referenced this pull request Jun 4, 2025
* Remove upper caps on numpy and pyarrow versions
saishreeeee pushed a commit that referenced this pull request Jun 4, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
saishreeeee pushed a commit that referenced this pull request Jun 4, 2025
* Remove upper caps on numpy and pyarrow versions

Signed-off-by: Sai Shree Pradhan <[email protected]>
saishreeeee pushed a commit that referenced this pull request Jun 4, 2025
* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: David Black <[email protected]>

* Added CI CD upto python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pandas 2.2.3 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Specify pyarrow 18.0.0 as the lower bound for python 3.13

Signed-off-by: David Black <[email protected]>

* Move `numpy` to dev dependencies

Signed-off-by: Dave Hirschfeld <[email protected]>

* Updated lockfile

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: David Black <[email protected]>
Signed-off-by: Dave Hirschfeld <[email protected]>
Co-authored-by: David Black <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
saishreeeee added a commit that referenced this pull request Jun 10, 2025
* [ES-402013] Close cursors before closing connection (#38)

* Add test: cursors are closed when connection closes

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.0.5 and improve CHANGELOG (#40)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* fix dco issue

Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* fix dco issue

Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* dco tunning

Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* dco tunning

Signed-off-by: Moe Derakhshani <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Github workflows: run checks on pull requests from forks (#47)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* OAuth implementation (#15)

This PR:
* Adds the foundation for OAuth against Databricks account on AWS with BYOIDP.
* It copies one internal module that Steve Weis @sweisdb wrote for Databricks CLI (oauth.py). Once ecosystem-dev team (Serge, Pieter) build a python sdk core we will move this code to their repo as a dependency.
* the PR provides authenticators with visitor pattern format for stamping auth-token which later is intended to be moved to the repo owned by Serge @nfx and and Pieter @pietern

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Automate deploys to Pypi (#48)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-205] Add functional examples (#52)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.1.0 (#54)

Bump to v2.1.0 and update changelog

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [SC-110400] Enabling compression in Python SQL Connector (#49)

Signed-off-by: Mohit Singla <[email protected]>
Co-authored-by: Moe Derakhshani <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add tests for parameter sanitisation / escaping (#46)

* Refactor so we can unit test `inject_parameters`
* Add unit tests for inject_parameters
* Remove inaccurate comment. Per #51, spark sql does not support escaping a single quote with a second single quote.
* Closes #51 and adds unit tests plus the integration test provided in #56

Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Courtney Holcomb (@courtneyholcomb)
Co-authored-by: @mcannamela
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump thrift dependency to 0.16.0 (#65)

Addresses https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.2.0 (#66)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Support Python 3.11 (#60)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.2.1  (#70)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add none check on _oauth_persistence in DatabricksOAuthProvider (#71)

Add none check on _oauth_persistence in DatabricksOAuthProvider to avoid app crash when _oauth_persistence is None.

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Support custom oauth client id and redirect port (#75)

* Support custom oauth client id and rediret port range

PySQL is used by other tools/CLIs which have own oauth client id,
we need to expose oauth_client_id and oauth_redirect_port_range
as the connection parameters to support this customization.

Signed-off-by: Jacky Hu <[email protected]>

* Change oauth redirect port range to port

Signed-off-by: Jacky Hu <[email protected]>

* Fix type check issue

Signed-off-by: Jacky Hu <[email protected]>

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.2.2 (#76)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Jesse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Merge staging ingestion into main (#78)

Follow up to #67 and #64

* Regenerate TCLIService using latest TCLIService.thrift from DBR (#64)
* SI: Implement GET, PUT, and REMOVE (#67)
* Re-lock dependencies after merging `main`

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.3.0 and update changelog (#80)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add pkgutil-style for the package (#84)

Since the package is under databricks namespace. pip install this package will cause issue importing other packages under the same namespace like automl and feature store. Adding pkgutil style to resolve the issue.

Signed-off-by: lu-wang-dl <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add SQLAlchemy Dialect (#57)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 2.4.0(#89)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix syntax in examples in root readme. (#92)

Do this because the environment variable pulls did not have closing quotes on their string literals.

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Less strict numpy and pyarrow dependencies (#90)

Signed-off-by: Thomas Newton <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Thomas Newton <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update example in docstring so query output is valid Spark SQL (#95)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.4.1 (#96)

Per the sermver.org spec, updating the projects dependencies is considered a compatible change.

https: //semver.org/#what-should-i-do-if-i-update-my-own-dependencies-without-changing-the-public-api

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update CODEOWNERS (#97)

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add Andre to CODEOWNERS (#98)

* Add Andre. Signed-off-by: Yunbo Deng <[email protected]>

Signed-off-by: yunbodeng-db <[email protected]>

* Revert the change temporarily so I can sign off. Signed-off-by: Yunbo Deng <[email protected]>

Signed-off-by: yunbodeng-db <[email protected]>

* Add Andre and sign off. Signed-off-by: Yunbo Deng <[email protected]>

Signed-off-by: yunbodeng-db <[email protected]>

* Remove redundant line

Signed-off-by: yunbodeng-db <[email protected]>

---------

Signed-off-by: yunbodeng-db <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add external auth provider + example (#101)

Signed-off-by: Andre Furlan <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>

Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Retry on connection timeout (#103)

A lot of the time we see the error `[Errno 110] Connection timed out`. This happens a lot in Azure, particularly. In this PR I make it a retryable error as it is safe

Signed-off-by: Andre Furlan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-244] Make http proxies work (#81)

Override thrift's proxy header encoding function.  Uses the fix identified in https://github.com/apache/thrift/pull/2565

H/T @pspeter

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 2.5.0 (#104)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix changelog release date for version 2.5.0

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Relax sqlalchemy requirement (#113)

* Plus update docs about how to change dependency spec

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update to version 2.5.1 (#114)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix SQLAlchemy timestamp converter + docs (#117)

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Relax pandas and alembic requirements (#119)

Update dependencies for alembic and pandas per customer request

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 2.5.2 (#118)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Use urllib3 for thrift transport + reuse http connections (#131)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Default socket timeout to 15 min (#137)

Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.6.0 (#139)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix: some thrift RPCs failed with BadStatusLine (#141)

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.6.1 (#142)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [ES-706907] Retry GetOperationStatus for http errors (#145)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.6.2 (#147)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-626] Support OAuth flow for Databricks Azure (#86)

## Summary

Support OAuth flow for Databricks Azure

## Background

Some OAuth endpoints (e.g. Open ID Configuration) and scopes are different between Databricks Azure and AWS. Current code only supports OAuth flow on Databricks in AWS

## What changes are proposed in this pull request?

- Change `OAuthManager` to decouple Databricks AWS specific configuration from OAuth flow
- Add `sql/auth/endpoint.py` that implements cloud specific OAuth endpoint configuration
- Change `DatabricksOAuthProvider` to work with the OAuth configurations in different Databricks cloud (AWS, Azure)
- Add the corresponding unit tests

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Use a separate logger for unsafe thrift responses (#153)

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Improve e2e test development ergonomics  (#155)

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Don't raise exception when closing a stale Thrift session (#159)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 2.7.0 (#161)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Cloud Fetch download handler (#127)

* Cloud Fetch download handler

Signed-off-by: Matthew Kim <[email protected]>

* Issue fix: final result link compressed data has multiple LZ4 end-of-frame markers

Signed-off-by: Matthew Kim <[email protected]>

* Addressing PR comments
 - Linting
 - Type annotations
 - Use response.ok
 - Log exception
 - Remove semaphore and only use threading.event
 - reset() flags method
 - Fix tests after removing semaphore
 - Link expiry logic should be in secs
 - Decompress data static function
 - link_expiry_buffer and static public methods
 - Docstrings and comments

Signed-off-by: Matthew Kim <[email protected]>

* Changing logger.debug to remove url

Signed-off-by: Matthew Kim <[email protected]>

* _reset() comment to docstring

Signed-off-by: Matthew Kim <[email protected]>

* link_expiry_buffer -> link_expiry_buffer_secs

Signed-off-by: Matthew Kim <[email protected]>

---------

Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Cloud Fetch download manager (#146)

* Cloud Fetch download manager

Signed-off-by: Matthew Kim <[email protected]>

* Bug fix: submit handler.run

Signed-off-by: Matthew Kim <[email protected]>

* Type annotations

Signed-off-by: Matthew Kim <[email protected]>

* Namedtuple -> dataclass

Signed-off-by: Matthew Kim <[email protected]>

* Shutdown thread pool and clear handlers

Signed-off-by: Matthew Kim <[email protected]>

* Docstrings and comments

Signed-off-by: Matthew Kim <[email protected]>

* handler.run is the correct call

Signed-off-by: Matthew Kim <[email protected]>

* Link expiry buffer in secs

Signed-off-by: Matthew Kim <[email protected]>

* Adding type annotations for download_handlers and downloadable_result_settings

Signed-off-by: Matthew Kim <[email protected]>

* Move DownloadableResultSettings to downloader.py to avoid circular import

Signed-off-by: Matthew Kim <[email protected]>

* Black linting

Signed-off-by: Matthew Kim <[email protected]>

* Timeout is never None

Signed-off-by: Matthew Kim <[email protected]>

---------

Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Cloud fetch queue and integration (#151)

* Cloud fetch queue and integration

Signed-off-by: Matthew Kim <[email protected]>

* Enable cloudfetch with direct results

Signed-off-by: Matthew Kim <[email protected]>

* Typing and style changes

Signed-off-by: Matthew Kim <[email protected]>

* Client-settable max_download_threads

Signed-off-by: Matthew Kim <[email protected]>

* Docstrings and comments

Signed-off-by: Matthew Kim <[email protected]>

* Increase default buffer size bytes to 104857600

Signed-off-by: Matthew Kim <[email protected]>

* Move max_download_threads to kwargs of ThriftBackend, fix unit tests

Signed-off-by: Matthew Kim <[email protected]>

* Fix tests: staticmethod make_arrow_table mock not callable

Signed-off-by: Matthew Kim <[email protected]>

* cancel_futures in shutdown() only available in python >=3.9.0

Signed-off-by: Matthew Kim <[email protected]>

* Black linting

Signed-off-by: Matthew Kim <[email protected]>

* Fix typing errors

Signed-off-by: Matthew Kim <[email protected]>

---------

Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Cloud Fetch e2e tests (#154)

* Cloud Fetch e2e tests

Signed-off-by: Matthew Kim <[email protected]>

* Test case works for e2-dogfood shared unity catalog

Signed-off-by: Matthew Kim <[email protected]>

* Moving test to LargeQueriesSuite and setting catalog to hive_metastore

Signed-off-by: Matthew Kim <[email protected]>

* Align default value of buffer_size_bytes in driver tests

Signed-off-by: Matthew Kim <[email protected]>

* Adding comment to specify what's needed to run successfully

Signed-off-by: Matthew Kim <[email protected]>

---------

Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update changelog for cloudfetch (#172)

Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Improve sqlalchemy backward compatibility with 1.3.24 (#173)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* OAuth: don't override auth headers with contents of .netrc file (#122)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix proxy connection pool creation (#158)

Signed-off-by: Sebastian Eckweiler <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Sebastian Eckweiler <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Relax pandas dependency constraint to allow ^2.0.0 (#164)

Signed-off-by: Daniel Segesdi <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Use hex string version of operation ID instead of bytes (#170)

---------
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy: fix has_table so it honours schema= argument (#174)

---------
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix socket timeout test (#144)

Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Disable non_native_boolean_check_constraint (#120)

---------
Signed-off-by: Bogdan Kyryliuk <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Remove unused import for SQLAlchemy 2 compatibility (#128)

Signed-off-by: William Gentry <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.8.0 (#178)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix typo in python README quick start example (#186)

---------

Co-authored-by: Jesse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Configure autospec for mocked Client objects (#188)

Resolves #187

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Use urllib3 for retries (#182)

Behaviour is gated behind `enable_v3_retries` config. This will be removed and become the default behaviour in a subsequent release.

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.9.0 (#189)

* Add note to changelog about using cloud_fetch
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Explicitly add urllib3 dependency (#191)

Signed-off-by: Jacobus Herman <[email protected]>

Co-authored-by: Jesse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to 2.9.1 (#195)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Make backwards compatible with urllib3~=1.0 (#197)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Convenience improvements to v3 retry logic (#199)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.9.2 (#201)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Github Actions Fix: poetry install fails for python 3.7 tests (#208)

snok/install-poetry@v1 installs the latest version of Poetry

The latest version of poetry released on 20 August 2023 (four days ago
as of this commit) which drops support for Python 3.7, causing our
github action to fail.

Until we complete #207 we need to conditionally install the last version
of poetry that supports Python 3.7 (poetry==1.5.1)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Make backwards compatible with urllib3~=1.0 [Follow up #197]  (#206)

* Make retry policy backwards compatible with urllib3~=1.0.0

We already implement the equivalent of backoff_max so the behaviour will
be the same for urllib3==1.x and urllib3==2.x

We do not implement backoff jitter so the behaviour for urllib3==1.x will
NOT include backoff jitter whereas urllib3==2.x WILL include jitter.

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump version to 2.9.3 (#209)

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add note to sqlalchemy example: IDENTITY isn't supported yet (#212)

ES-842237

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1029] Updated thrift compiler version (#216)

* Updated thrift definitions

Signed-off-by: nithinkdb <[email protected]>

* Tried with a different thrift installation

Signed-off-by: nithinkdb <[email protected]>

* Reverted TCLI to previous

Signed-off-by: nithinkdb <[email protected]>

* Reverted to older thrift

Signed-off-by: nithinkdb <[email protected]>

* Updated version again

Signed-off-by: nithinkdb <[email protected]>

* Upgraded thrift

Signed-off-by: nithinkdb <[email protected]>

* Final commit

Signed-off-by: nithinkdb <[email protected]>

---------

Signed-off-by: nithinkdb <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1055] Updated thrift defs to allow Tsparkparameters (#220)

Updated thrift defs to most recent versions

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update changelog to indicate that 2.9.1 and 2.9.2 have been yanked. (#222)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix changelog typo: _enable_v3_retries (#225)

Closes #219

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Introduce SQLAlchemy reusable dialog tests (#125)

Signed-off-by: Jim Fulton <[email protected]>
Co-Authored-By: Jesse Whitehouse <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>

Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1026] Add Parameterized Query support to Python (#217)

* Initial commit

Signed-off-by: nithinkdb <[email protected]>

* Added tsparkparam handling

Signed-off-by: nithinkdb <[email protected]>

* Added basic test

Signed-off-by: nithinkdb <[email protected]>

* Addressed comments

Signed-off-by: nithinkdb <[email protected]>

* Addressed missed comments

Signed-off-by: nithinkdb <[email protected]>

* Resolved comments

---------

Signed-off-by: nithinkdb <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Parameterized queries: Add e2e tests for inference (#227)

Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1109] Parameterized Query: add suport for inferring decimal types (#228)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: reorganise dialect files into a single directory (#231)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1083] Updated thrift files and added check for protocol version (#229)

* Updated thrift files and added check for protocol version

Signed-off-by: nithinkdb <[email protected]>

* Made error message more clear

Signed-off-by: nithinkdb <[email protected]>

* Changed name of fn

Signed-off-by: nithinkdb <[email protected]>

* Ran linter

Signed-off-by: nithinkdb <[email protected]>

* Update src/databricks/sql/client.py

Co-authored-by: Jesse <[email protected]>

---------

Signed-off-by: nithinkdb <[email protected]>
Co-authored-by: Jesse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-840] Port staging ingestion behaviour to new UC Volumes (#235)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Query parameters: implement support for binding NoneType parameters (#233)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Bump dependency version and update e2e tests for existing behaviour (#236)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Revert "[PECO-1083] Updated thrift files and added check for protocol version" (#237)

Reverts #229 as it causes all of our e2e tests to fail on some versions of DBR.

We'll reimplement the protocol version check in a follow-up.

This reverts commit 241e934a96737d506c2a1f77c7012e1ab8de967b.

Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: add type compilation for all CamelCase types (#238)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: add type compilation for uppercase types (#240)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Stop skipping all type tests (#242)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1134] v3 Retries: allow users to bound the number of redirects to follow (#244)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Parameters: Add type inference for BIGINT and TINYINT types (#246)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Stop skipping some non-type tests (#247)

* Stop skipping TableDDLTest and permanent skip HasIndexTest

We're now in the territory of features that aren't required for sqla2
compat as of pysql==3.0.0 but we may consider adding this in the future.

In this case, table comment reflection needs to be manually implemented.
Index reflection would require hooking into the compiler to reflect
the partition strategy.

test_suite.py::HasIndexTest_databricks+databricks::test_has_index[dialect] SKIPPED (Databricks does not support indexes.)
test_suite.py::HasIndexTest_databricks+databricks::test_has_index[inspector] SKIPPED (Databricks does not support indexes.)
test_suite.py::HasIndexTest_databricks+databricks::test_has_index_schema[dialect] SKIPPED (Databricks does not support indexes.)
test_suite.py::HasIndexTest_databricks+databricks::test_has_index_schema[inspector] SKIPPED (Databricks does not support indexes.)
test_suite.py::TableDDLTest_databricks+databricks::test_add_table_comment SKIPPED (Comment reflection is possible but not implemented in this dialect.)
test_suite.py::TableDDLTest_databricks+databricks::test_create_index_if_not_exists SKIPPED (Databricks does not support indexes.)
test_suite.py::TableDDLTest_databricks+databricks::test_create_table PASSED
test_suite.py::TableDDLTest_databricks+databricks::test_create_table_if_not_exists PASSED
test_suite.py::TableDDLTest_databricks+databricks::test_create_table_schema PASSED
test_suite.py::TableDDLTest_databricks+databricks::test_drop_index_if_exists SKIPPED (Databricks does not support indexes.)
test_suite.py::TableDDLTest_databricks+databricks::test_drop_table PASSED
test_suite.py::TableDDLTest_databricks+databricks::test_drop_table_comment SKIPPED (Comment reflection is possible but not implemented in this dialect.)
test_suite.py::TableDDLTest_databricks+databricks::test_drop_table_if_exists PASSED
test_suite.py::TableDDLTest_databricks+databricks::test_underscore_names PASSED

Signed-off-by: Jesse Whitehouse <[email protected]>

* Permanently skip QuotedNameArgumentTest with comments

The fixes to DESCRIBE TABLE and visit_xxx were necessary to get to the
point where I could even determine that these tests wouldn't pass.

But those changes are not currently tested in the dialect. If, in the
course of reviewing the remaining tests in the compliance suite, I find
that these visit_xxxx methods are not tested anywhere else then we should
extend test_suite.py with our own tests to confirm the behaviour for
ourselves.

Signed-off-by: Jesse Whitehouse <[email protected]>

* Move files from base.py to _ddl.py

The presence of this pytest.ini file is _required_ to establish pytest's
root_path

https://docs.pytest.org/en/7.1.x/reference/customize.html#finding-the-rootdir

Without it, the custom pytest plugin from SQLAlchemy can't read the contents
of setup.cfg which makes none of the tests runnable.

Signed-off-by: Jesse Whitehouse <[email protected]>

* Emit a warning for certain constructs

Signed-off-by: Jesse Whitehouse <[email protected]>

* Stop skipping RowFetchTest

Date type work fixed this test failure

Signed-off-by: Jesse Whitehouse <[email protected]>

* Revise infer_types logic to never infer a TINYINT

This allows these SQLAlchemy tests to pass:

test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_bound_limit PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_bound_limit_offset PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_expr_limit_simple_offset PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_simple_limit PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_simple_limit_expr_offset PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_simple_limit_offset[cases0] PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_simple_limit_offset[cases1] PASSED
test_suite.py::FetchLimitOffsetTest_databricks+databricks::test_simple_limit_offset[cases2] PASSED

This partially reverts the change introduced in #246

Signed-off-by: Jesse Whitehouse <[email protected]>

* Stop skipping FetchLimitOffsetTest

I implemented our custom DatabricksStatementCompiler so we can override
the default rendering of unbounded LIMIT clauses from `LIMIT -1` to
`LIMIT ALL`

We also explicitly skip the FETCH clause tests since Databricks doesn't
support this syntax.

Blacked all source code here too.

Signed-off-by: Jesse Whitehouse <[email protected]>

* Stop skipping FutureTableDDLTest

Add meaningful skip markers for table comment reflection and indexes

Signed-off-by: Jesse Whitehouse <[email protected]>

* Stop skipping Identity column tests

This closes https://github.com/databricks/databricks-sql-python/issues/175

Signed-off-by: Jesse Whitehouse <[email protected]>

* Stop skipping HasTableTest

Adding the @reflection.cache decorator to has_table is necessary to pass
test_has_table_cache

Caching calls to has_table improves the efficiency of the connector

Signed-off-by: Jesse Whitehouse <[email protected]>

* Permanently skip LongNameBlowoutTest

Databricks constraint names are limited to 255 characters

Signed-off-by: Jesse Whitehouse <[email protected]>

* Stop skipping ExceptionTest

Black test_suite.py

Signed-off-by: Jesse Whitehouse <[email protected]>

* Permanently skip LastrowidTest

Signed-off-by: Jesse Whitehouse <[email protected]>

* Implement PRIMARY KEY and FOREIGN KEY reflection and enable tests

Signed-off-by: Jesse Whitehouse <[email protected]>

* Skip all IdentityColumnTest tests

Turns out that none of these can pass for the same reason that the
first two seemed un-runnable in db6f52bb329f3f43a9215b5cd46b03c3459a302a

Signed-off-by: Jesse Whitehouse <[email protected]>

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: implement and refactor schema reflection methods (#249)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add GovCloud domain into AWS domains (#252)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Refactor __init__.py into base.py (#250)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Finish implementing all of ComponentReflectionTest (#251)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Finish marking all tests in the suite (#253)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Finish organising compliance test suite (#256)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Fix failing mypy checks from development (#257)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Enable cloud fetch by default (#258)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1137] Reintroduce protocol checking to Python test fw (#248)

* Put in some unit tests, will add e2e

Signed-off-by: nithinkdb <[email protected]>

* Added e2e test

Signed-off-by: nithinkdb <[email protected]>

* Linted

Signed-off-by: nithinkdb <[email protected]>

* re-bumped thrift files

Signed-off-by: nithinkdb <[email protected]>

* Changed structure to store protocol version as feature of connection

Signed-off-by: nithinkdb <[email protected]>

* Fixed parameters test

Signed-off-by: nithinkdb <[email protected]>

* Fixed comments

Signed-off-by: nithinkdb <[email protected]>

* Update src/databricks/sql/client.py

Co-authored-by: Jesse <[email protected]>
Signed-off-by: nithinkdb <[email protected]>

* Fixed comments

Signed-off-by: nithinkdb <[email protected]>

* Removed extra indent

Signed-off-by: nithinkdb <[email protected]>

---------

Signed-off-by: nithinkdb <[email protected]>
Co-authored-by: Jesse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* sqla2 clean-up: make sqlalchemy optional and don't mangle the user-agent (#264)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Add support for TINYINT (#265)

Closes #123

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add OAuth M2M example (#266)

* Add OAuth M2M example

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Native Parameters: reintroduce INLINE approach with tests (#267)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Document behaviour of executemany (#213)

Signed-off-by: Martin Rueckl <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy 2: Expose TIMESTAMP and TIMESTAMP_NTZ types to users (#268)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Drop Python 3.7 as a supported version (#270)

Signed-off-by: Jesse Whitehouse <[email protected]>
(cherry picked from commit 8d85fa8b33a70331141c0c6556196f641d1b8ed5)
Signed-off-by: Sai Shree Pradhan <[email protected]>

* GH Workflows: remove Python 3.7 from the matrix for _all_ workflows (#274)

Remove Python 3.7 from the matrix for _all_ workflows

This was missed in #270

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add README and updated example for SQLAlchemy usage (#273)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Rewrite native parameter implementation with docs and tests (#281)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Enable v3 retries by default (#282)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* security: bump pyarrow dependency to 14.0.1 (#284)

pyarrow is currently compatible with Python 3.8 → Python 3.11

I also removed specifiers for when Python is 3.7 since this no longer
applies.

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump package version to 3.0.0 (#285)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix docstring about default parameter approach (#287)

Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1286] Add tests for complex types in query results (#293)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* sqlalchemy: fix deprecation warning for dbapi classmethod (#294)

Rename `dbapi` classmethod to `import_dbapi` as required by SQLAlchemy 2

Closes #289

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1297] sqlalchemy: fix: can't read columns for tables containing a TIMESTAMP_NTZ column (#296)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepared 3.0.1 release (#297)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Make contents of `__init__.py` equal across projects (#304)

---------

Signed-off-by: Pieter Noordhuis <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix URI construction in ThriftBackend (#303)

Signed-off-by: Jessica <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [sqlalchemy] Add table and column comment support (#329)

Signed-off-by: Christophe Bornet <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Pin pandas and urllib3 versions to fix runtime issues in dbt-databricks (#330)

Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* SQLAlchemy: TINYINT types didn't reflect properly (#315)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1435] Restore `tests.py` to the test suite (#331)

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 3.0.2 (#335)

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update some outdated OAuth comments (#339)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Redact the URL query parameters from the urllib3.connectionpool logs (#341)

* Redact the URL query parameters from the urllib3.connectionpool logs

Signed-off-by: Mubashir Kazia <[email protected]>

* Fix code formatting

Signed-off-by: Mubashir Kazia <[email protected]>

* Add str check for the log record message arg dict values

Signed-off-by: Mubashir Kazia <[email protected]>

---------

Signed-off-by: Mubashir Kazia <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 3.0.3 (#344)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1411] Support Databricks OAuth on GCP (#338)

* [PECO-1411] Support OAuth InHouse on GCP

Signed-off-by: Jacky Hu <[email protected]>

* Update changelog

Signed-off-by: Jesse Whitehouse <[email protected]>

---------

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Jesse Whitehouse <[email protected]>
Co-authored-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1414] Support Databricks native OAuth in Azure (#351)

* [PECO-1414] Support Databricks InHouse OAuth in Azure

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prep for Test Automation (#352)

Getting ready for test automation

Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update code owners (#345)

* update owners

Signed-off-by: yunbodeng-db <[email protected]>

* update owners

Signed-off-by: yunbodeng-db <[email protected]>

* update owners

Signed-off-by: yunbodeng-db <[email protected]>

---------

Signed-off-by: yunbodeng-db <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Reverting retry behavior on 429s/503s to how it worked in 2.9.3 (#349)

Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to version 3.1.0 (#358)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1440] Expose current query id on cursor object (#364)

* [PECO-1440] Expose current query id on cursor object

Signed-off-by: Levko Kravets <[email protected]>

* Clear `active_op_handle` when closing the cursor

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add a default for retry after (#371)

* Add a default for retry after
Signed-off-by: Ben Cassell <[email protected]>

* Applied black formatter
Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix boolean literals (#357)

Set supports_native_boolean to True

Signed-off-by: Alex Holyoke <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Don't retry network requests that fail with code 403 (#373)

* Don't retry requests that fail with 404

Signed-off-by: Jesse Whitehouse <[email protected]>

* Fix lint error

Signed-off-by: Jesse Whitehouse <[email protected]>

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bump to 3.1.1 (#374)

* bump to 3.1.1

Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix cookie setting (#379)

* fix cookie setting

Signed-off-by: Ben Cassell <[email protected]>

* Removing cookie code

Signed-off-by: Ben Cassell <[email protected]>

---------

Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fixing a couple type problems: how I would address most of #381 (#382)

* Create py.typed

Signed-off-by: wyattscarpenter <[email protected]>

* add -> Connection annotation

Signed-off-by: wyattscarpenter <[email protected]>

* massage the code to appease the particular version of the project's mypy deps

Signed-off-by: wyattscarpenter <[email protected]>

* fix circular import problem

Signed-off-by: wyattscarpenter <[email protected]>

---------

Signed-off-by: wyattscarpenter <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* fix the return types of the classes' __enter__ functions (#384)

fix the return types of the classes' __enter__ functions so that the type information is preserved in context managers eg with-as blocks

Signed-off-by: wyattscarpenter <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add Kravets Levko to codeowners (#386)

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepare for 3.1.2 (#387)

Signed-off-by: Ben Cassell <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update the proxy authentication (#354)

changed authentication for proxy

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix failing tests (#392)

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Relax `pyarrow` pin (#389)

* Relax `pyarrow` pin

Signed-off-by: Dave Hirschfeld <[email protected]>

* Allow `pyarrow` 16

Signed-off-by: Dave Hirschfeld <[email protected]>

* Update `poetry.lock`

Signed-off-by: Dave Hirschfeld <[email protected]>

---------

Signed-off-by: Dave Hirschfeld <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix log error in oauth.py (#269)

* Duplicate of applicable change from #93

Signed-off-by: Jesse Whitehouse <[email protected]>

* Update changelog

Signed-off-by: Jesse Whitehouse <[email protected]>

* Fix after merge

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Jesse Whitehouse <[email protected]>
Signed-off-by: Levko Kravets <[email protected]>
Co-authored-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Enable `delta.feature.allowColumnDefaults` for all tables (#343)

* Enable `delta.feature.allowColumnDefaults` for all tables

* Code style

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Co-authored-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix SQLAlchemy tests (#393)

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Add more debug logging for CloudFetch (#395)

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Update Thrift package (#397)

Signed-off-by: Milan Lukac <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepare release 3.2.0 (#396)

* Prepare release 3.2.0

Signed-off-by: Levko Kravets <[email protected]>

* Update changelog

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* move py.typed to correct places (#403)

* move py.typed to correct places

https://peps.python.org/pep-0561/ says 'For namespace packages (see PEP 420), the py.typed file should be in the submodules of the namespace, to avoid conflicts and for clarity.'. Previously, when I added the py.typed file to this project, https://github.com/databricks/databricks-sql-python/pull/382 , I was unaware this was a namespace package (although, curiously, it seems I had done it right initially and then changed to the wrong way). As PEP 561 warns us, this does create conflicts; other libraries in the databricks namespace package (such as, in my case, databricks-vectorsearch) are then treated as though they are typed, which they are not. This commit moves the py.typed file to the correct places, the submodule folders, fixing that problem.
Signed-off-by: wyattscarpenter <[email protected]>

* change target of mypy to src/databricks instead of src.

I think this might fix the CI code-quality checks failure, but unfortunately I can't replicate that failure locally and the error message is unhelpful

Signed-off-by: wyattscarpenter <[email protected]>

* Possible workaround for bad error message 'error: --install-types failed (no mypy cache directory)'; see https://github.com/python/mypy/issues/10768#issuecomment-2178450153

Signed-off-by: wyattscarpenter <[email protected]>

* fix invalid yaml syntax

Signed-off-by: wyattscarpenter <[email protected]>

* Best fix (#3)

Fixes the problem by cding and supplying a flag to mypy (that mypy needs this flag is seemingly fixed/changed in later versions of mypy; but that's another pr altogether...). Also fixes a type error that was somehow in the arguments of the program (?!) (I guess this is because you guys are still using implicit optional)

---------

Signed-off-by: wyattscarpenter <[email protected]>

* return the old result_links default (#5)

Return the old result_links default, make the type optional, & I'm pretty sure the original problem is that add_file_links can't take a None, so these statements should be in the body of the if-statement that ensures it is not None

Signed-off-by: wyattscarpenter <[email protected]>

* Update src/databricks/sql/utils.py

"self.download_manager is unconditionally used later, so must be created. Looks this part of code is totally not covered with tests 🤔"

Co-authored-by: Levko Kravets <[email protected]>
Signed-off-by: wyattscarpenter <[email protected]>

---------

Signed-off-by: wyattscarpenter <[email protected]>
Co-authored-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Upgrade mypy (#406)

* Upgrade mypy

This commit removes the flag (and cd step) from https://github.com/databricks/databricks-sql-python/commit/f53aa37a34dc37026d430e71b5e0d1b871bc5ac1 which we added to get mypy to treat namespaces correctly. This was apparently a bug in mypy, or behavior they decided to change. To get the new behavior, we must upgrade mypy. (This also allows us to remove a couple `# type: ignore` comment that are no longer needed.)

This commit runs changes the version of mypy and runs `poetry lock`. It also conforms the whitespace of files in this project to the expectations of various tools and standard (namely: removing trailing whitespace as expected by git and enforcing the existence of one and only one newline at the end of a file as expected by unix and github.) It also uses https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade codebase due to a change in mypy behavior. For a similar reason, it also fixes a new type (or otherwise) errors:

* "Return type 'Retry' of 'new' incompatible with return type 'DatabricksRetryPolicy' in supertype 'Retry'"
* databricks/sql/auth/retry.py:225: error: object has no attribute update  [attr-defined]
* /test_param_escaper.py:31: DeprecationWarning: invalid escape sequence \) [as it happens, I think it was also wrong for the string not to be raw, because I'm pretty sure it wants all of its backslashed single-quotes to appear literally with the backslashes, which wasn't happening until now]
* ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject [this is like a numpy version thing, which I fixed by being stricter about numpy version]

---------

Signed-off-by: wyattscarpenter <[email protected]>

* Incorporate suggestion.

I decided the most expedient way of dealing with this type error was just adding the type ignore comment back in, but with a  `[attr-defined]` specifier this time. I mean, otherwise I would have to restructure the code or figure out the proper types for a TypedDict for the dict and I don't think that's worth it at the moment.

Signed-off-by: wyattscarpenter <[email protected]>

---------

Signed-off-by: wyattscarpenter <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Do not retry failing requests with status code 401 (#408)

- Raises NonRecoverableNetworkError when request results in 401 status code

Signed-off-by: Tor Hødnebø <[email protected]>
Signed-off-by: Tor Hødnebø <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1715] Remove username/password (BasicAuth) auth option (#409)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1751] Refactor CloudFetch downloader: handle files sequentially (#405)

* [PECO-1751] Refactor CloudFetch downloader: handle files sequentially; utilize Futures

Signed-off-by: Levko Kravets <[email protected]>

* Retry failed CloudFetch downloads

Signed-off-by: Levko Kravets <[email protected]>

* Update tests

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix CloudFetch retry policy to be compatible with all `urllib3` versions we support (#412)

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Disable SSL verification for CloudFetch links (#414)

* Disable SSL verification for CloudFetch links

Signed-off-by: Levko Kravets <[email protected]>

* Use existing `_tls_no_verify` option in CloudFetch downloader

Signed-off-by: Levko Kravets <[email protected]>

* Update tests

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepare relese 3.3.0 (#415)

* Prepare relese 3.3.0

Signed-off-by: Levko Kravets <[email protected]>

* Remove @arikfr from CODEOWNERS

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix pandas 2.2.2 support (#416)

* Support pandas 2.2.2

See release note numpy 2.2.2:
https://pandas.pydata.org/docs/dev/whatsnew/v2.2.0.html#to-numpy-for-numpy-nullable-and-arrow-types-converts-to-suitable-numpy-dtype

* Allow pandas 2.2.2 in pyproject.toml

* Update poetry.lock, poetry lock --no-update

* Code style

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Co-authored-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1801] Make OAuth as the default authenticator if no authentication setting is provided (#419)

* [PECO-1801] Make OAuth as the default authenticator if no authentication setting is provided

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1857] Use SSL options with HTTPS connection pool (#425)

* [PECO-1857] Use SSL options with HTTPS connection pool

Signed-off-by: Levko Kravets <[email protected]>

* Some cleanup

Signed-off-by: Levko Kravets <[email protected]>

* Resolve circular dependencies

Signed-off-by: Levko Kravets <[email protected]>

* Update existing tests

Signed-off-by: Levko Kravets <[email protected]>

* Fix MyPy issues

Signed-off-by: Levko Kravets <[email protected]>

* Fix `_tls_no_verify` handling

Signed-off-by: Levko Kravets <[email protected]>

* Add tests

Signed-off-by: Levko Kravets <[email protected]>

---------

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepare release v3.4.0 (#430)

Prepare release 3.4.0

Signed-off-by: Levko Kravets <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1926] Create a non pyarrow flow to handle small results for the column set (#440)

* Implemented the columnar flow for non arrow users

* Minor fixes

* Introduced the Column Table structure

* Added test for the new column table

* Minor fix

* Removed unnecessory fikes

Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-1961] On non-retryable error, ensure PySQL includes useful information in error (#447)

* added error info on non-retryable error

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Reformatted all the files using black (#448)

Reformatted the files using black

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepare release v3.5.0 (#457)

Prepare release 3.5.0

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [PECO-2051] Add custom auth headers into cloud fetch request (#460)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* Prepare release 3.6.0 (#461)

Signed-off-by: Jacky Hu <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>

* [ PECO - 1768 ] PySQL: adjust HTTP retry logic to align with Go and Nodejs drivers (#467)

* Added the exponential backoff code

* Added the exponential backoff algorithm and refractored the code

* Added jitter and added unit tests

* Reformatted

* Fixed the test_retry_exponential_backoff integration test

Signed-off-by: Sai Shree Pradhan <[email protected]>

* [ PECO-2065 ] Create the async execution flow for the PySQL Connector (#463)

* Built the basic flow for the async pipeline - testing is remaining

* Implemented the flow for the get_execution_result, but the problem of invalid operation handle still persists

* Missed adding some files in previous commit

* Working prototype of execute_async, get_query_state and get_execution_result

* Added integration tests for execute_async

* add docs for functions

* Refractored the async code

* Fixed java doc

* Reformatted

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Fix for check_types github action failing (#472)

Fixed the chekc_types failing

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Remove upper caps on dependencies (#452)

* Remove upper caps on numpy and pyarrow versions

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Updated the doc to specify native parameters in PUT operation is not supported from >=3.x connector (#477)

Added doc update

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Incorrect rows in inline fetch result (#479)

* Raised error when incorrect Row offset it returned

* Changed error type

* grammar fix

* Added unit tests and modified the code

* Updated error message

* Updated the non retying to only inline case

* Updated fix

* Changed the flow

* Minor update

* Updated the retryable condition

* Minor test fix

* Added extra space

Signed-off-by: Sai Shree Pradhan <[email protected]>

* Bumped up to version 3.7.0 (#482)

* bumped up version

* Updated to version 3.7.0

* Grammar fix

* Minor fix

Signed-off-by: Sai Shree Pradhan <[email protected]>

* PySQL Connector split into connector and sqlalchemy (#444)

* Modified the gitignore file to not have .idea file

* [PECO-1803] Splitting the PySql connector into the core and the non core part (#417)

…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants