Skip to content

DuckDB to import columns as strings as a fallback #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jan 13, 2023
Merged

DuckDB to import columns as strings as a fallback #341

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1bdf086
Add insertUntypedCSV function to duckdb
libbey-observable Jan 12, 2023
5cf6f4f
Try untyped insertion as a fallback when loading DuckDBClient
libbey-observable Jan 12, 2023
1650d9b
simplify
libbey-observable Jan 12, 2023
501fbc8
Check error message
libbey-observable Jan 12, 2023
8465454
Use symbol and existing options
libbey-observable Jan 12, 2023
b5a4e7f
Undo changes to table.js
libbey-observable Jan 12, 2023
c27516a
Catch earlier in process, no need for config
libbey-observable Jan 12, 2023
1a60e21
Remove unnecessary variable
libbey-observable Jan 12, 2023
175f5da
Update comment
libbey-observable Jan 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion src/duckdb.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,12 +172,19 @@ async function insertFile(database, name, file, options) {
try {
switch (file.mimeType) {
case "text/csv":
case "text/tab-separated-values":
case "text/tab-separated-values": {
return await connection.insertCSVFromPath(file.name, {
name,
schema: "main",
...options
}).catch(async (error) => {
// If initial attempt to insert CSV resulted in a conversion
// error, try again, this time treating all columns as strings.
if (error.toString().includes("Could not convert")) {
return await insertUntypedCSV(connection, file, name);
}
});
}
case "application/json":
return await connection.insertJSONFromPath(file.name, {
name,
Expand Down Expand Up @@ -205,6 +212,13 @@ async function insertFile(database, name, file, options) {
}
}

async function insertUntypedCSV(connection, file, name) {
const statement = await connection.prepare(
`CREATE TABLE '${name}' AS SELECT * FROM read_csv_auto(?, ALL_VARCHAR=TRUE)`
);
return await statement.send(file.name);
}

async function insertArrowTable(database, name, table, options) {
const connection = await database.connect();
try {
Expand Down