PDO_ODBC truncating CHAR field Db2i driver

[fleadram]

Description

The following code:

<?php
user = {myUser};
$password = {myPassword};
$rdb = {myDsn};
$dsn = "DSN={{$rdb}};NAM=1;ALWAYSCALCLEN=1;CCSID=1208";

$sql = "
SELECT long_char_field
 FROM table
 WHERE id = '1'
";

$attrOpions = [
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
    PDO::ATTR_PERSISTENT => false,
    PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
];
$db = new PDO("odbc:$dsn", $user, $password, $attrOpions);

$stmt = $db->prepare($sql);
$stmt->execute();
while (($row = $stmt->fetch())) {
    var_dump($row["LONG_CHAR_FIELD"]);
}

$db = odbc_connect($dsn, $user, $password);

$stmt = odbc_prepare($db, $sql);
odbc_execute($stmt);
while (($row = odbc_fetch_array($stmt))) {
    var_dump($row["LONG_CHAR_FIELD"]);
}

Resulted in this output:

string(256) "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
string(560) "00000000016500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

But I expected this output instead:

string(560) "00000000016500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"  
string(560) "00000000016500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

I've seen a lot of discussion around pdo_odbc and long field fetching, but nothing that exactly matched what I'm seeing here. Similar though. We have a field that's defined in ddl as CHAR(560) on db2 for i (using up to date drivers from here). After upgrading our php from 7.0.1 to 8.3.20 we're having the issue where this column is only returning 256 chars, and the data returned is not positionally correct (i.e. the chunks seem to get overlaid on one another).

The problematic logic seems to be the chunk approach for retrieving "long" columns starting here

I was able to debug, and patched in @NattyNarwhal 's printfs from here https://gist.github.com/NattyNarwhal/278aedf8e3c48bb8dcb4ae14cde0b283#file-debugging-printf-patch

What I see in both cases is that the driver is fetching a length of 256 at first, but the logic to continue the loop is (correctly) triggered 2x

if (rc==SQL_SUCCESS_WITH_INFO || (rc==SQL_SUCCESS && C->fetched_len > 255)) 

What I see happening as I step through is that though the original_fetched_len is 256, the status is still SQL_SUCCESS_WITH_INFO causing it to drop into this logic block that overlays the result with the new data.

				str = zend_string_realloc(str, used + 256, 0);
				memcpy(ZSTR_VAL(str) + used, buf2, 256);
				used = used + 255;

Interestingly, if we patch a rollback of this commit, eliminating the recalculation of used, things go back to working.

I also suspect that this PR would fix it too, but I don't know if that's still planned to go in.

Here's the logging output from the patch above.

!!! Get col stmt 0x7ffffe68e000 col 0
*** Is long string? 1
*** After fetching 256, fetched len 256 data "00000000016500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
*** Actually long
@@@ fixed_used 0 = orig_fetched_len 256 - C->fetched_len 256
@@@ (and used = 255)
@@@ strange fetch_len 256, buf "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
@@@ fixed_used 206 = orig_fetched_len 256 - C->fetched_len 50
@@@ (and used = 255)
@@@ normal fetch_len 50, buf "00000000000000000000000000000000000000000000000000"
@@@ fixed_used 256 = orig_fetched_len 256 - C->fetched_len 0
@@@ (and used = 256)
*** Unicode? 0
*** OK, got string "0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"

PHP Version

PHP 8.3.20 (cli) (built: Apr 8 2025 20:21:18) (NTS gcc x86_64)
Copyright (c) The PHP Group
Zend Engine v4.3.20, Copyright (c) Zend Technologies
with Zend OPcache v8.3.20, Copyright (c), by Zend Technologies
with Xdebug v3.4.2, Copyright (c) 2002-2025, by Derick Rethans

Operating System

Red Hat Enterprise Linux release 9.5 (Plow)

[NattyNarwhal]

Yeah, this codepath in PDO_ODBC isn't ideal and we worry about regressions between different drivers a lot. That said, I haven't heard users I'm supporting having many issues recently outside of dealing with binaries (but they're also using the Db2i on IBM i, so the driver may behave differently there).

I'm still interested in GH-10809 - if it works for you, it'd be a good idea to try to get it merged. I was a bit hesitant due to performance and compatibility concerns (as Db2i is the driver I'm most familiar with, I would like to know if impacts other drivers).

Of course, there's also the possibility of a driver bug. I'd recommend checking both the unixODBC trace and the Db2i driver trace as well.

[fleadram]

Excellent, thanks for the quick response. I'll grab those traces in a bit, appreciate the direction.

[fleadram]

ok, here's the traces https://gist.github.com/fleadram/a3a07397f647413ba25e8b4b7fb95340

from my untrained reading, I don't see anything too suspicious in the odbc trace, but I see several

[IBM][System i Access ODBC Driver]Column 1: CWB0111 - A buffer passed to a system call is too small to hold return data dsn: dsn sys: dsn row: 4294967295 col: 1

in the iaccess trace.

Also, tomorrow I'll try patching in the above PR and test that.

[kadler]

FYI, the duplicate CWB0111 are just a consequence of trying to convert from CCSID 37 to 1208 (UTF-8). The way the driver does this is by first converting from 37 to UTF-16 (13488), then from UTF-16 to UTF-8 so each step gives a buffer too small error.

The ODBC trace is more baffling, since the string length output looks all kinds of wrong:

Strlen Or Ind = 0x7ffffe690250 -> 256 (64 bits)
Strlen Or Ind = 0x7ffffe690250 -> 256 (64 bits)
Strlen Or Ind = 0x7ffffe690250 -> 50 (64 bits)

This length should be the total length required, but it returns 256 on the first run. That should have returned 255 bytes plus null terminator, so that would indicate that it returned all the data. The second run returns the same, so another 255 bytes. Finally the last one returns 50, so there should be 49 bytes plus null terminator. This totals 559 bytes, one less than the actual length of 560. This definitely seems a problem with our driver. I know I made changes in the latest update to fix this behavior for LOB locators, but I guess I need to do the same for inline conversions.

The output of the last SQLGetData call also seems to not be null-terminated:

2025-04-17;21:02:29.843;ODBC;36;36;10008;  Target: 3030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030

The previous outputs show the hexdump ends with 00, but here it's all 30 till the end. Either the previous two fetches dumped the data incorrectly or this one is. This might explain why the last one is 50 instead of 51, which accounts for the off-by-one issue.

@NattyNarwhal Does pdo_odbc use these lengths to construct the length of the PHP variable it creates?

[fleadram]

@kadler thanks for chiming in, while I'll defer to @NattyNarwhal for sure on the guts of the php, I think I would agree with your statement that This length should be the total length required, but it returns 256 on the first run. That's what I'm seeing. From my reading of other issues it seems that PHP trusts that the driver will return the full column length on this call (i.e. that &C->fetched_len will contain that value) but our iaccess db2 is returning 256 regardless.

` /* fetch it into C->data, which is allocated with a length
* of 256 bytes; if there is more to be had, we then allocate
* bigger buffer for the caller to free */

	rc = SQLGetData(S->stmt, colno+1, C->is_unicode ? SQL_C_BINARY : SQL_C_CHAR, C->data,
		256, &C->fetched_len);
	orig_fetched_len = C->fetched_len;`

[NattyNarwhal]

It's been a while since I looked at this part of PDO_ODBC, but I believe it should return the full length left on the initial fetch of 256, yes. The driver does accommodate for this situation by calling in a loop until NO_DATA (as MS recommends), but the length calculation could be pretty fiddly due to things like calculations of length between the encoding it's in vs. requested. I know that PR I authored a while back was motivated by such a case (especially due to shift states in DBCS encodings).