Skip to content

Idea: arrow_record_batches cursor method #375

Open
@unj1m

Description

@unj1m

When you call fetchmany_arrow(batchsize) and specify a batch size, you get a table that has multiple record batches.

In my experience, the record batches are much smaller than the batch size I specify. I think the SQL connector has to do record keeping to align the batch size I give and the record batches it gets from the server (IIUC).

When I call fetchmany_arrow, I end up with nested loops. The outer loops loops over fetchmany_arrow calls and the inner loops over the batches returned.

I suspect it would be less bother for everyone if there was an API (e.g. arrow_record_batches()) that returned a record-batch iterator.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions