New 'UNLIST' function #8418

ChudaykinAlex · 2025-01-29T11:46:44Z

Purpose

The function parses the input string using the specified delimiter (comma "," is implied by default) and returns the identified substrings as discrete records containing a single field. Additionally, the desired type of the returned field can be specified. If the specified data type conversion is impossible, an error is raised at runtime.

See also: #8005

Syntax and rules

<table value function> ::= UNLIST(<input> [, <separator>] [, <data type conversion>]) [AS] <correlation name> [(<derived column name>)]
<input> ::= <value>
<separator> ::= <value>
<data type conversion> ::= RETURNING <data type>

)Arguments:

<input>: any value expression that returns a string/blob of characters (or may be converted to a string), including string literals, table columns, constants, variables, expressions, etc. This parameter is mandatory.
<separator>: optional value expression that returns a string which is used as a delimiter (i.e. it separates one value from another inside the input string). It may also be a BLOB TEXT value, limited to 32KB. If an empty string is specified, the output will be just one record containing the input string. If omitted, the comma character is used as a delimiter.
<data type>: target data type to convert the output values into. Alternatively, a domain can be specified as the returned type. If omitted, VARCHAR(32) is implied. Feel free to suggest any better alternative default.
<correlation name>: alias of the record set returned by the UNLIST function. It is a mandatory parameter (per SQL standard).
<derived column name>: optional alias of the column returned by the UNLIST function. If omitted, UNLIST is used as an alias.

Changes in the code

New rules are added to the parser (parse.y).
New BLR code is added to support table-value functions with a sub-code for the UNLIST function.
New class TableValueFunctionSourceNode is added, inherited from RecordSourceNode. This class is a generic implementation of table-value functions inside the engine (to be reused later by other table-value functions defined in the SQL specification). Its descendant classes are responsible for implementing particular functions (UNLIST is just the first one in this list, more to follow).
Extend csb_repeat with jrd_tab_func* pointing to a new structure defining the compile-time info about the function (name, fields, format).
New class TableValueFunctionScan is added, inherited from RecordStream. It generates records for the function stream. Class UnlistFunctionScan is inherited from TableValueFunctionScan to implement the record generator specific to the UNLIST function.

Examples

SELECT * FROM UNLIST('1,2,3,4,5') AS U;

UNLIST                           
================================ 
1                                
2                                
3                                
4                                
5    


SELECT * FROM UNLIST('100:200:300:400:500', ':' RETURNING INT) AS U;

      UNLIST 
============ 
         100 
         200 
         300 
         400 
         500 


SELECT U.* FROM UNLIST('text1,text2,text3') AS U;

UNLIST                           
================================ 
text1                            
text2                            
text3     


SELECT C0 FROM UNLIST('text1,text2,text3') AS U(C0);

C0                               
================================ 
text1                            
text2                            
text3 


SELECT U.C0 FROM UNLIST('text1,text2,text3') AS U(C0);

C0                               
================================ 
text1                            
text2                            
text3 


SET TERM ^ ;
RECREATE PROCEDURE TEST_PROC RETURNS (PROC_RETURN_INT INT)
AS
DECLARE VARIABLE text VARCHAR(11);
BEGIN
	text = '123:123:123';
	FOR SELECT * FROM UNLIST( :text, ':' RETURNING INT) AS A INTO :PROC_RETURN_INT DO
	SUSPEND;
END^
SET TERM ; ^

SELECT * FROM TEST_PROC;
PROC_RETURN_INT 
=============== 
            123 
            123 
            123 


CREATE DOMAIN D1 AS INT;
SELECT TEST_DOMAIN FROM UNLIST('1,2,3,4' RETURNING D1) AS A(TEST_DOMAIN);

 TEST_DOMAIN 
============ 
           1 
           2 
           3 
           4 
           

CREATE VIEW TEST_VIEW AS SELECT * FROM UNLIST('1,2,3,4') AS A(B);
SELECT B FROM TEST_VIEW;

B                                
================================ 
1                                
2                                
3                                
4      


SELECT UNLIST FROM UNLIST('UNLIST,A,S,A') AS A;

Statement failed, SQLSTATE = 42S22
Dynamic SQL Error
-SQL error code = -206
-Column unknown
-UNLIST
-At line 2, column 9
At line 24 in file ...

sim1984 · 2025-01-29T12:04:17Z

What about interaction with the IN predicate? In the initial discussion, the idea was put forward that it would be nice to be able to perform queries like this

SELECT * FROM table WHERE id IN UNLIST('1,2,3' RETURNING INT)

That is, without additional wrapping of the UNLIST results in a derived table.

ChudaykinAlex · 2025-01-29T12:22:27Z

I didn't do anything specifically for IN.
I may not understand the specific task at hand. Here is an example of how I understood your question.

recreate table test1 (s01 INT);
insert into test1(s01)values('1');
insert into test1(s01)values('2');
insert into test1(s01)values('3');
insert into test1(s01)values('4');
insert into test1(s01)values('5');
insert into test1(s01)values('6');
insert into test1(s01)values('7');
insert into test1(s01)values('8');

SELECT * FROM test1 WHERE s01 IN (SELECT * FROM UNLIST('1,8' RETURNING INT) AS U);

         S01 
============ 
           1 
           8

sim1984 · 2025-01-29T12:37:27Z

In the discussion of #8005 it was suggested to use the shortened syntax for IN UNLIST for convenience, but it is not necessary.

One more question.

SELECT *
FROM UNLIST(?, ',' RETURNING INT)

What type of input parameter will be returned to the client after preparing the query?

BLOB SUB_TYPE TEXT or VARCHAR(N). If the second, what will be N?

For the output parameter, unless otherwise specified, it will be VARCHAR(32) judging by the description above.

asfernandes · 2025-01-29T23:48:00Z

doc/sql.extensions/README.unlist

+
+G)
+	CREATE DOMAIN D1 AS INT;
+	SELECT TEST_DOMAIN FROM UNLIST('1,2,3,4' RETURNING D1) AS A(TEST_DOMAIN);


Is it TYPE OF [COLUMN] allowed?

Check. Granted. Added example in readme

asfernandes · 2025-01-29T23:49:58Z

src/dsql/ExprNodes.cpp

@@ -6446,9 +6458,28 @@ dsql_fld* FieldNode::resolveContext(DsqlCompilerScratch* dsqlScratch, const Meta
 		return nullptr;
 	}

+	const TEXT* dsqlName = NULL;


NULL -> nullptr

asfernandes · 2025-01-29T23:52:49Z

src/dsql/parse.y

+table_value_function
+	: table_value_function_clause table_value_function_correlation_name table_value_function_columns_name
+		{
+			auto *node = nodeAs<TableValueFunctionSourceNode>($1);


Suggested change

auto *node = nodeAs<TableValueFunctionSourceNode>($1);

auto node = nodeAs<TableValueFunctionSourceNode>($1);

Сorrected.

asfernandes · 2025-01-29T23:56:59Z

src/dsql/parse.y

+
+
+%type <metaNameArray> table_value_function_columns_name
+table_value_function_columns_name


Just use derived_column_list instead of create duplicate specific rules.

Apparently I didn't notice this rule.

asfernandes · 2025-01-30T00:00:03Z

src/jrd/RecordSourceNodes.cpp

@@ -3651,6 +3657,331 @@ RseNode* SelectExprNode::dsqlPass(DsqlCompilerScratch* dsqlScratch)
 	return PASS1_derived_table(dsqlScratch, this, NULL);
 }

+TableValueFunctionSourceNode*


This is not a Firebird code convention. Please start the method name in the same line as the return type and break later as necessary.

I used the clang-format utility to do the formatting. That's what he decided. I'll be more careful

asfernandes · 2025-01-30T00:11:26Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+	const auto valueDesc = EVL_expr(tdbb, request, valueItem);
+	if (valueDesc == nullptr)
+	{
+		rpb->rpb_number.setValid(true);


Why true?

asfernandes · 2025-01-30T00:12:05Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+
+	const auto textType = toDesc->getTextType();
+
+	auto setStringToRecord = [&] (string str, USHORT length = 0)


Whould not be a string reference?

I'd even say it should be a const string reference.

asfernandes · 2025-01-30T00:13:47Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+			if (length == 0)
+				continue;
+
+			string valueStr(buffer, length);


Please do not copy the buffer to another string.

Are you telling me to read the data directly into a string?

I think it's better to use something lightweight like string_view here, so as not to allocate memory and not do unnecessary copying.

I'm going to try an experiment with string_view

Reworked for string_view. tests pass

asfernandes · 2025-01-30T00:15:34Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+			auto end = AbstractString::npos;
+			do
+			{
+				auto size = end = valueStr.find(separatorStr);


Is this code compatible with multi-character separator and the list string split in different blob segments?

No, it's not. To be improved, I guess. While LIST would never split the separator, the input may be provided from the different source as well and we cannot predict the layout.

@asfernandes, do we generally consider OK to read/process the blob as a whole? If so, this would be the easiest solution (and IIRC it worked this way in our original implementation) and we do the same in some built-in functions. Or should it be considered a bad practice to be revisited/fixed and thus the new code should be able to process blobs in chunks?

At the very least, processing BLOBs in parts will save a lot on reading huge BLOBs if not all results are needed UNLIST

SELECT * FROM UNLIST(?, ',') ROWS 10

It requires to move the processing from open() to getRecord(). Currently implemented differently, but it could be worth the effort.

Shouldn't it depend on OPTIMIZE FOR {FIRST | ALL} ROWS? I.e. process/return records in chunks for ALL ROWS (not necessarily read/cache the whole blob, maybe just some sliding buffer) but process/return records one by one for FIRST ROWS?

Shouldn't it depend on OPTIMIZE FOR {FIRST | ALL} ROWS? I.e. process/return records in chunks for ALL ROWS (not necessarily read/cache the whole blob, maybe just some sliding buffer) but process/return records one by one for FIRST ROWS?

I do not see a clear relation of OPTIMIZE FOR and UNNEST. OPTIMIZE FOR is top level query feature, while UNNEST may be in inner queries.

I mean the mechanics, not the clause itself. For example, a subquery with a FIRST clause implies FOR FIRST ROWS when compiling/optimizing its own RSE.

The current implementation has two drawbacks:

Increased memory consumption due to result buffering

Slowing down queries with the FIRST ROWS strategy

In general, it is not very clear why TableValueFunction is a buffered data source. It seems to be related to primary, and all our primary sources are are pipelined. If you need to buffer the result, then the optimizer of the higher level does it.

These issues are being addressed.

asfernandes · 2025-01-30T00:16:42Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+		return;
+	}
+
+	if (valueDesc->isBlob())


Others Firebird functions compare strings using collation rules.
Here this is disregard. Should it be?

Good question to be discussed. Theoretically, one may ask to split the text into chunks by e.g. both 'A' and 'a' used as a case-insensitive separator 'a'. But honestly, I cannot imagine that being used in practice. And collation-aware compares would add some extra overhead. So maybe it's worth comparing byte-wise (converting both value and separator to the common data type, if necessary). Or maybe we should consider two code branches depending on whether byte-wise compare is possible or not.

dyemanov · 2025-01-30T09:26:42Z

In the discussion of #8005 it was suggested to use the shortened syntax for IN UNLIST for convenience, but it is not necessary.

One more question.
SELECT *
FROM UNLIST(?, ',' RETURNING INT)
What type of input parameter will be returned to the client after preparing the query?

BLOB SUB_TYPE TEXT or VARCHAR(N). If the second, what will be N?

For the output parameter, unless otherwise specified, it will be VARCHAR(32) judging by the description above.

Good question to be resolved together. Something like VARCHAR(1024) looks OK at the first glance, but it would lead to a runtime error if overflowed by the user (coercion will fail). CLOB looks safer but has a noticeable overhead if the input string is actually a VARCHAR.

aafemt · 2025-01-30T10:00:37Z

it would lead to a runtime error if overflowed by the user (coercion will fail).

In #8145 I already pass user-provided buffers to looper. One step further and functions can accept user input directly as descriptor avoiding coercion and thus overflow.

AlexPeshkoff · 2025-01-30T10:35:34Z

On 1/30/25 13:01, Dimitry Sibiryakov wrote: it would lead to a runtime error if overflowed by the user (coercion will fail). In #8145 <#8145> I already pass user-provided buffers to looper. One step further and functions can accept user input directly as descriptor avoiding coercion and thus overflow.

This looks like solution.

… parameter type.

dyemanov · 2025-03-10T06:10:46Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+	{
+		rpb->rpb_number.setValid(false);
+		return false;
+	}


rpb->rpb_number.setValid(true);

The processing has been moved to the internalGetRecord method. If “RecordBuffer” is empty, it is tried to be filled in the “nextBuffer” method. Filling will be done in portions. A small refactoring.

asfernandes · 2025-03-11T13:55:46Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+			continue;
+		}
+		return true;
+	} while (1);


Suggested change

} while (1);

} while (true);

asfernandes · 2025-03-11T13:59:53Z

src/jrd/recsrc/TableValueFunctionScan.cpp

+
+bool TableValueFunctionScan::nextBuffer(thread_db* /*tdbb*/) const
+{
+	return false;


Should not TableValueFunctionScan be an abstract class instead?

asfernandes · 2025-03-11T14:00:43Z

src/jrd/recsrc/RecordSource.h

+		struct Impure : public RecordSource::Impure
+		{
+			RecordBuffer* m_recordBuffer;
+			blb* m_blob;


Aren't you mixing specifics of UnlistFunctionScan's impure in the base TableValueFunctionScan?

…ted remarks on abstract class.

pavel-zotov · 2025-03-26T13:41:39Z

run cmd.exe, then do: chcp 65001
run following sql:

set bail on;
set list on;
set echo on;
set names utf8; --------------- [ ! ]
shell del c:\temp\tmp4test.fdb 2>nul;
create database 'localhost:c:\temp\tmp4test.fdb' user 'sysdba' password 'masterkey';
commit;
SELECT * FROM UNLIST('ABCDEFGHIJKLMNOPQRSTUVWXYZ,ABCDEFGHIJKLMNOPQRSTUVWXYZ', ',') AS U;

Output will be:

ISQL Version: WI-T6.0.0.693 Firebird 6.0 Initial
Server version:
Firebird/Windows/AMD/Intel/x64 (access method), version "WI-T6.0.0.693 Firebird 6.0 Initial"
Firebird/Windows/AMD/Intel/x64 (remote server), version "WI-T6.0.0.693 Firebird 6.0 Initial/tcp (PZ)/P20:C"
Firebird/Windows/AMD/Intel/x64 (remote interface), version "WI-T6.0.0.693 Firebird 6.0 Initial/tcp (PZ)/P20:C"
on disk structure version 14.0
select * from unlist('abcdefghijklmnopqrstuvwxyz,abcdefghijklmnopqrstuvwxyz', ',') as u;

Statement failed, SQLSTATE = 22001
arithmetic exception, numeric overflow, or string truncation
-string right truncation
-expected length 8, actual 32

Expected output (will be also if we comment line marked as " [ ! ] "):

UNLIST                          abcdefghijklmnopqrstuvwxyz
UNLIST                          abcdefghijklmnopqrstuvwxyz

pavel-zotov · 2025-03-26T14:09:03Z

PS.
Same for
select * from unlist('abcdefghi,abcdefghi', ',') as u;
(but not for 'abcdefgh' - for this case function works OK)

sim1984 · 2025-03-26T18:32:36Z

This is not surprising considering that the default return type is This is not surprising considering that the default return type is varchar(32) character set none. It is better to always specify returning type.

pavel-zotov · 2025-03-26T18:38:52Z

IMO, varchar(32) is too small. Why can't we use 8190 ?

dyemanov · 2025-03-26T18:55:32Z

Resulting type should be VARCHAR(32) of the input character set, but it appears it's described as 32 bytes, not characters. To be fixed, AFAIU.

As for the length, maybe 32 is a short limit, but this function is to be commonly used with numbers and this limit is enough for that usage case. If you want to deal with strings, better specify the length yourself (nobody knows how many characters gonna fit).

pavel-zotov · 2025-03-27T13:11:08Z

QA note: deferred waiting for fix of resulting type (expect 32 characters rather than bytes)

ChudaykinAlex · 2025-04-09T11:01:58Z

Good day. Thank you for your comments. I have looked into the previous example. I found an error in my code. I made a patch, now locally checking the build and tests. And will prepare for publishing.
I have executed your script from the example.

pavel-zotov · 2025-04-09T11:04:38Z

0xFF.
IMO, it is inconveniently that one need to repeat some parts of code (e.g. "unicode_char(0x2114)").
Why this function could not be used as other functions, like this:

with
d as (
    select
         blob_fld
        ,unicode_char(0x2114) as separator
    from t_longblob
)
, e as (
    select unlist(d.blob_fld, d.separator returning blob character set utf8) as x
    from d
)
select e.x
from e
where e.x containing d.separator
;

-- ?

dyemanov · 2025-04-09T12:03:53Z

We have no other table-valued functions yet, so your example is syntactically incorrect. It should look something like:

, e as (
    select * from d, unlist(d.blob_fld, d.separator returning blob character set utf8) as u (x)
)

pavel-zotov · 2025-04-09T14:00:32Z

It should look something like

Yes, it works in such way. Thanks!

pavel-zotov · 2025-04-10T10:56:53Z

It seems that ASCII_CHAR(0) can not be used as separator (at least currently):

echo set list on; set count on; select * from unlist('1', ascii_char(1)) as u(x); | isql /:employee
X                               1
Records affected: 1
(OK, expected)

echo set list on; set count on; select * from unlist('1', ascii_char(0)) as u(x); | isql /:employee
-- NO OUTPUT. FB hangs with 100% load of one CPU core.
-- Ctrl-Creak in ISQL does not help: FB process continues its activity

Checked on Windows.
This is URL to FB dump, stack trace, snapshot 6.0.0.725-a2b05f4-x64.7z and other info:
https://drive.google.com/drive/folders/1BgTSsRqvyD-cbmnvyE43xlG0vpxuOwk5?usp=sharing

aafemt · 2025-04-10T11:00:13Z

ascii_char(0) must not be used anywhere outside of OCTETS. Perhaps this function should explicitly prohibit zero value.

But the infinite loop is definitely a bug.

pavel-zotov · 2025-04-10T11:40:55Z

One more example:

connect '/:employee';
set echo on;
set list on;
set count on;

select '1' || ascii_char(26) || '2' as chr_26_lst from rdb$database;

select /* check ascii_char(26) */ * from unlist( '1' || ascii_char(26) || '2', ascii_char(26)) as u(x);

select /* check ascii_char(26) */ * from unlist('1→2', '→') as u(x);

Output will be:

set list on;
set count on;
select '1' || ascii_char(26) || '2' as chr_26_lst from rdb$database;
CHR_26_LST                      1→2
Records affected: 1

select /* check ascii_char(26) */ * from unlist( '1' || ascii_char(26) || '2', ascii_char(26)) as u(x);
X                               1
X                               2
Records affected: 2

select /* check ascii_char(26) */ * from unlist('1
Expected end of statement, encountered EOF

aafemt · 2025-04-10T11:52:51Z

This is a separate ISQL problem, not UNLIST one.

pavel-zotov · 2025-04-10T11:53:00Z

Perhaps this function should explicitly prohibit zero value.

maybe chr(0) must also be prohibited in LIST ():

echo select list('1', ascii_char(0)) from (select 1 x from rdb$types rows 5); | isql /:employee
LIST:
11111

echo select list(ascii_char(0), ':') from (select 1 x from rdb$types rows 5); | isql /:employee
LIST:
::::

aafemt · 2025-04-10T11:55:24Z

"This function" I meant ascii_char().

pavel-zotov · 2025-04-10T11:57:36Z

"This function" I meant ascii_char().

Why to decline the whole function usage ? IMO, it is enough to restrict only some values that it returns, at least chr(0).

aafemt · 2025-04-10T12:00:36Z

IMO, it is enough to restrict only some values that it returns, at least chr(0).

Yes, this is exactly what I wrote: ascii_char would better to refuse zero,

pavel-zotov · 2025-04-10T12:02:41Z

This is a separate ISQL problem, not UNLIST one.

#8512

ChudaykinAlex · 2025-04-10T12:04:19Z

It seems that ASCII_CHAR(0) can not be used as separator (at least currently):
echo set list on; set count on; select * from unlist('1', ascii_char(1)) as u(x); | isql /:employee
X                               1
Records affected: 1
(OK, expected)

echo set list on; set count on; select * from unlist('1', ascii_char(0)) as u(x); | isql /:employee
-- NO OUTPUT. FB hangs with 100% load of one CPU core.
-- Ctrl-Creak in ISQL does not help: FB process continues its activity
Checked on Windows. This is URL to FB dump, stack trace, snapshot 6.0.0.725-a2b05f4-x64.7z and other info: https://drive.google.com/drive/folders/1BgTSsRqvyD-cbmnvyE43xlG0vpxuOwk5?usp=sharing

I found the problem with the infinite loop, I will fix it.
As for the ban ascii_char(0), I won't give you an answer yet.

aafemt · 2025-04-10T12:07:37Z

As for the ban ascii_char(0), I won't give you an answer yet.

Perhaps you can answer an other question: is UNLIST supposed to work with (VAR)BINARY datatype (charset OCTETS) and binary BLOBs?

pavel-zotov · 2025-04-10T14:55:21Z

It seems that semicolon character ( ';' = ascii_char(59)) has some effect on time of PREPARE statement when source string has lot of such separators.
Consider attached .zip file, it contains four queries with different characters used as separator: ':', ';', '<' and '='.
In each case extremely long source string is used for parsing:
q'#t;y;;;<skipped>;;;#' -- where <skipped> is sequence of ~32K similar characters.

Queries look like this:

select /* check ascii_char(58) */ * from unlist(q'#t:y::: ... :::#')', ':') ;
select /* check ascii_char(59) */ * from unlist(q'#t;y;;; ... ;;;#')', ';') ;
select /* check ascii_char(60) */ * from unlist(q'#t<y<<< ... <<<#')', '<') ;
select /* check ascii_char(61) */ * from unlist(q'#t=y=== ... ===#')', '=') ;

Then run trace with config:

    log_connections = true
    log_transactions = true
    log_statement_prepare = true
    log_statement_start = true
    log_statement_finish = true
    log_statement_free = true
    print_perf = true

(its log also is in attached .zip)

For 1st, 4rd and 4th statements trace will show similar parts which executes almost instantly:

2025-04-10T17:26:13.2550 (232:0000000001FE24C0) START_TRANSACTION
...
2025-04-10T17:26:13.2550 (232:0000000001FE24C0) EXECUTE_STATEMENT_FINISH
...
SET TRANSACTION

2025-04-10T17:26:13.2550 (232:0000000001FE24C0) FREE_STATEMENT
...
2025-04-10T17:26:13.2550 (232:0000000001FE24C0) START_TRANSACTION
...
2025-04-10T17:26:13.2650 (232:0000000001FE24C0) PREPARE_STATEMENT
...
Statement 59:
-------------------------------------------------------------------------------
select /* check ascii_char(58) */ * from unlist(q'#t:y:::::::::::::::::::::::::::::::::
      3 ms

2025-04-10T17:26:13.2660 (232:0000000001FE24C0) EXECUTE_STATEMENT_START
...
select /* check ascii_char(58) */ * from unlist(q'#t:y:::::::::::::::::::::::::::::::::

2025-04-10T17:26:13.3730 (232:0000000001FE24C0) EXECUTE_STATEMENT_FINISH
...

But for SECOND statement (when ; is used as delimiter) trace will show that there is weird "gap" in timestamps after START_TRANSACTION and PREPARE_STATEMENT.
Size of this "gap" is about 8 seconds.

Appropriate lines are marked as [ 1 ] and [ 2 ], their stamps: 2025-04-10T17:26:23.0550 and 2025-04-10T17:26:31.1140:

2025-04-10T17:26:23.0550 (232:0000000001FE24C0) START_TRANSACTION
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040
       (TRA_276, CONCURRENCY | WAIT | READ_WRITE)

2025-04-10T17:26:23.0550 (232:0000000001FE24C0) EXECUTE_STATEMENT_FINISH
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040
       (TRA_276, CONCURRENCY | WAIT | READ_WRITE)

-------------------------------------------------------------------------------
SET TRANSACTION
0 records fetched
      0 ms, 1 write(s), 1 fetch(es), 1 mark(s)

2025-04-10T17:26:23.0550 (232:0000000001FE24C0) FREE_STATEMENT
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040

-------------------------------------------------------------------------------
SET TRANSACTION

2025-04-10T17:26:23.0550 (232:0000000001FE24C0) START_TRANSACTION ------------------------ [  1  ]
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040
       (TRA_277, READ_COMMITTED | NO_REC_VERSION | WAIT | READ_WRITE)

2025-04-10T17:26:31.1140 (232:0000000001FE24C0) PREPARE_STATEMENT ------------------------ [  2  ]
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040
       (TRA_277, READ_COMMITTED | NO_REC_VERSION | WAIT | READ_WRITE)

Statement 59:
-------------------------------------------------------------------------------
select /* check ascii_char(59) */ * from unlist(q'#t;y;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
      2 ms

2025-04-10T17:26:31.1140 (232:0000000001FE24C0) EXECUTE_STATEMENT_START
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040
       (TRA_276, CONCURRENCY | WAIT | READ_WRITE)

Statement 59:
-------------------------------------------------------------------------------
select /* check ascii_char(59) */ * from unlist(q'#t;y;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

2025-04-10T17:26:31.2350 (232:0000000001FE24C0) EXECUTE_STATEMENT_FINISH
   employee (ATT_245, SYSDBA:NONE, NONE, TCPv6:fe80::8974:2b63:cd85:3f68%17/56873)
   C:\FB\60SS\isql.exe:10040
       (TRA_276, CONCURRENCY | WAIT | READ_WRITE)

unlist-long-duplicated-separators-at-end.zip

mrotteveel · 2025-04-10T15:24:25Z

That thing with ; is probably the new ISQL feature to automatically recognize when a statement is complete.

dyemanov · 2025-04-11T05:57:44Z

As for the ban ascii_char(0), I won't give you an answer yet.

Perhaps you can answer an other question: is UNLIST supposed to work with (VAR)BINARY datatype (charset OCTETS) and binary BLOBs?

Yes, it's expected to work.

pavel-zotov · 2025-04-11T15:48:55Z

I have a Q about how string that looks like floating-point number is interpreted in UNLIST.
If we check these statements:

echo set heading off; select cast(2.225073858507202e-308 as double precision) from rdb$database; | isql /:employee
echo set heading off; select cast(2.225073858507201e-308 as double precision) from rdb$database; | isql /:employee

-- then output will be:

2.225073858507202e-308
0.000000000000000

So, minimal value that is distinguish from zero is first of above mentioned.
Now lets check this:

echo select * from unlist('2.225073858507202e-307' returning double precision) as a(unlist_double_01); | isql /:employee

Output:

Statement failed, SQLSTATE = 22003
arithmetic exception, numeric overflow, or string truncation
-numeric value is out of range

Why ?

PS.
Minimum value that does not raise this exception in UNLIST (when it must return double) is: 9.9e-307 (9.899999999999e-307 - raises such error).

dyemanov · 2025-04-11T16:56:49Z

It's not about UNLIST at all, you're comparing different cases. See:

SQL> select cast('2.225073858507202e-308' as double precision) from rdb$database;

                   CAST 
======================= 
Statement failed, SQLSTATE = 22003
arithmetic exception, numeric overflow, or string truncation
-numeric value is out of range
SQL>

pavel-zotov · 2025-04-11T17:10:45Z

0xFF. Is this a bug or no ? (i mean: cast('2.225073858507202e-308' as double precision) ==> numeric value is out of range)

dyemanov · 2025-04-11T17:16:30Z

IMO: if the numeric literal fits the double precision range, CAST from string should work. But maybe the underlying conversion is trickier and may cause undesired side effects.

pavel-zotov · 2025-04-16T17:54:53Z

QA note: deferred because of
#8418 (comment)
#8418 (comment)
(waiting for fix / resolution)

… based on reported QA issues

dyemanov · 2025-04-19T17:39:11Z

Please retest with the new snapshot.

pavel-zotov · 2025-04-20T19:06:29Z

Checked on 6.0.0.744-e883a89: all fine now. New test for UNLIST will be added soon.

pavel-zotov · 2025-05-21T20:48:56Z

::: QA note :::
See functional/intfunc/unlist/ folder with several tests related to this function.

ChudaykinAlex added 3 commits January 29, 2025 11:16

Implementation of the new internal function UNLIST

0167f8f

Added raedme file

0653686

Fix typo

c7c6429

asfernandes reviewed Jan 30, 2025

View reviewed changes

ChudaykinAlex added 4 commits February 3, 2025 11:33

Correction of comments from the review. Added initialization of input…

be29efd

… parameter type.

Improved names, added error output when blr code is incorrect

dae3606

Conversion to string_view

4e4420c

Fix windows build

c4928d5

dyemanov linked an issue Feb 22, 2025 that may be closed by this pull request

UNLIST table-valued function #8005

Closed

dyemanov reviewed Mar 10, 2025

View reviewed changes

ChudaykinAlex added 2 commits March 11, 2025 10:35

Changed processing of BLOB as input parameter.

f0a8d10

The processing has been moved to the internalGetRecord method. If “RecordBuffer” is empty, it is tried to be filled in the “nextBuffer” method. Filling will be done in portions. A small refactoring.

Merge branch 'master' into master_unlist_new_function

b55d5b2

asfernandes reviewed Mar 11, 2025

View reviewed changes

Corrected remarks about variables belonging to the base class. correc…

42bbb27

…ted remarks on abstract class.

dyemanov assigned ChudaykinAlex Mar 19, 2025

dyemanov added the fix-version: 6.0 Alpha 1 label Mar 19, 2025

dyemanov merged commit f00a24c into FirebirdSQL:master Mar 24, 2025
24 checks passed

pavel-zotov added the qa: deferred label Mar 27, 2025

pavel-zotov mentioned this pull request Apr 10, 2025

ISQL can not prepare query containing ascii_char(26), issuing "Expected end of statement, encountered EOF" #8512

Open

dyemanov added a commit that referenced this pull request Apr 19, 2025

Another postfixes for PR #8418 (UNLIST function) by Alexey Chudaykin,…

33ad7e6

… based on reported QA issues

pavel-zotov added qa: covered by another tests and removed qa: deferred labels May 21, 2025

	auto *node = nodeAs<TableValueFunctionSourceNode>($1);
	auto node = nodeAs<TableValueFunctionSourceNode>($1);



		%type <metaNameArray> table_value_function_columns_name
		table_value_function_columns_name


		const auto textType = toDesc->getTextType();

		auto setStringToRecord = [&] (string str, USHORT length = 0)

Uh oh!

New 'UNLIST' function #8418

New 'UNLIST' function #8418

Uh oh!

Conversation

ChudaykinAlex commented Jan 29, 2025

Uh oh!

sim1984 commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChudaykinAlex commented Jan 29, 2025

Uh oh!

sim1984 commented Jan 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dyemanov Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sim1984 Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sim1984 commented Jan 29, 2025 •

edited

Loading

dyemanov Feb 20, 2025 •

edited

Loading

sim1984 Feb 21, 2025 •

edited

Loading

pavel-zotov commented Mar 26, 2025 •

edited

Loading

sim1984 commented Mar 26, 2025 •

edited

Loading