pd.core.arrays.DatetimeArray Issue

[sssaha]

What language are you using?

Python

What version are you using?

0.4.2.

What database are you using?

SQLITE

What dataframe are you using?

Pandas

Can you describe your bug?

Not compatible with Latest pandas 2.2.3 - in the SQL database has datetime

What is the error?

df = cx.read_sql(conn, query)

File "C:\ADO\mdct\venvp313\Lib\site-packages\connectorx_init_.py", line 394, in read_sql
df = reconstruct_pandas(result)
File "C:\ADO\mdct\venvp313\Lib\site-packages\connectorx_init_.py", line 478, in reconstruct_pandas
pd.core.arrays.DatetimeArray(block_data), placement=binfo.cids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "C:\ADO\mdct\venvp313\Lib\site-packages\pandas\core\arrays\datetimelike.py", line 1950, in init
warnings.warn(
~~~~~~~~~~~~~^
# GH#55623
^^^^^^^^^^
...<3 lines>...
stacklevel=find_stack_level(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
FutureWarning: DatetimeArray.init is deprecated and will be removed in a future version. Use pd.array instead.

[pangjunrong]

Thanks for the issue @sssaha, your "error" seems to be FutureWarning rather than a bug. I have tried it with pandas 2.2.3 and don't seem to have encountered any hurdles with datetime data. Could you kindly provide a MRE so I can look into this further?

This Pandas FutureWarning can be easily resolved by following their suggestion in refactoring the following line:
pd.core.arrays.DatetimeArray(block_data), placement=binfo.cids to pd.array(block_data, dtype='datetime64[ns]'), placement=binfo.cids but I suspect we will take a performance hit when moving away from the internals.

I do also see additional DeprecationWarning related to the same reconstruct_pandas method, which would potentially make the above change obsolete:

connectorx-znXkt-OL-py3.11/lib/python3.11/site-packages/pandas/core/frame.py:717: DeprecationWarning: Passing a BlockManager to DataFrame is deprecated and will raise in a future version. Use public APIs instead

My understanding is that most pandas internals are marked with either FutureWarning or DeprecationWarning because they would like to ween off downstream libraries from using their non-public APIs, see discussion here: pandas-dev/pandas#56422.

While looking for a suitable alternative to specifically address your FutureWarning, I came across pandas-dev/pandas#56775, which suggests the use of _from_sequence() — it is another method available in the internals. I benchmarked its performance against the original implementation and found the following:

Datetime Array Creation Benchmarks

Size	Method	Time (µs)
Small (1K items)
	DatetimeArray	2300
	pd.array	2730
	_from_sequence	1770
Medium (10K items)
	DatetimeArray	2909
	pd.array	4418
	_from_sequence	1503
Large (100K items)
	DatetimeArray	1913
	pd.array	6478
	_from_sequence	1224

It would appear that using _from_sequence is consistently faster for the instantiation of pandas arrays during the making process and avoids the FutureWarning as a bonus. If @wangxiaoying is agreeable to this change, I'm happy to explore this optimization which would kill two birds with one stone. 🐦🐦 <- 🪨

The same applies to IntegerArray & BooleanArray, I've benchmarked the latter inclusive of masking operations:

[wangxiaoying]

Hi @pangjunrong , thanks for the investigation!

The performance of the array construction using _from_sequence looks pretty impressive! I think we could switch to it for all the ExtensionArrays.

As for make_block, we probably can wait and see how pyarrow deal with this.

[sssaha]

Any speed gain is always preferable - that is the reason we are using connectorx than panda read SQL. I can wait to upgrade my python version (It seems like Connectorx 0.3.3 is not compatible with Python 3.13)

[AliRaza5959]

Hi! I am on v0.4.3 and still seeing this. Is this released?

5/27/2025, 8:34:53.136 PM
/app/.venv/lib/python3.12/site-packages/connectorx/__init__.py:478: FutureWarning: DatetimeArray.__init__ is deprecated and will be removed in a future version. Use pd.array instead.
5/27/2025, 8:34:53.136 PM
pd.core.arrays.DatetimeArray(block_data), placement=binfo.cids

Thanks

[pangjunrong]

Hi! I am on v0.4.3 and still seeing this. Is this released?

5/27/2025, 8:34:53.136 PM
/app/.venv/lib/python3.12/site-packages/connectorx/__init__.py:478: FutureWarning: DatetimeArray.__init__ is deprecated and will be removed in a future version. Use pd.array instead.
5/27/2025, 8:34:53.136 PM
pd.core.arrays.DatetimeArray(block_data), placement=binfo.cids

Thanks

Hey @AliRaza5959, this change has not yet been released. It'll be in the next one!