StreamedJsonResponse does not stream

[macropay-solutions]

Hi,

We implemented a StreamedJsonResponse for large sets of data and we noticed it crashes with memory issues because the json_encode can't be used for streaming data.

What alternatives exist to this issue?

If it can't stream data, why is it called Streamed?

Thank you.

Originally posted by @macropay-solutions in #60252

[mtarld]

Hey @macropay-solutions,

AFAIK, the StreamedJsonResponse does stream data. But to do so, it needs to iterates over an iterable.
This means that if your data is a first-level iterable, it should be streamed properly.
However, if it is nested in an object, for example, the stream will only consist of one chunk (and therefore won't be streamed).

If you're in that second case, you might want to take a look at the upcoming and experimental JsonStreamer component, which will be released in Symfony 7.3

[macropay-solutions]

@mtarld we tried giving it directly the iterator (LazyCollection), not an array with an iterator in it. In both cases we received memory error on 3.7 mil rows for example.

Update

Description of the behaviour:
laravel/framework#55509 (comment)

Update:

With the LazyColection as first param the result looks like this:

[
    {
        "id": 2000,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",
        "updated_at": null,
        "primary_key_identifier": "2000"
    },...
]

[mtarld]

I think you're right, it might be something here 🙂
IMO, the flush is missing (same for StreamedResponse::setChunks()

[macropay-solutions]

@mtarld Nope. We tried also with flush and ob_flush. If you read through the related issue from laravel, you will see why.

[macropay-solutions]

@mtarld

If you're in that second case, you might want to take a look at the upcoming and experimental JsonStreamer component, which will be released in Symfony 7.3

Is JS capable of decoding partial json strings if the json is sent like:

[
    {
        "id": 2000,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",
        "updated_at": null,
        "primary_key_identifier": "2000"
    },

then

    {
        "id": 2001,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",
        "updated_at": null,
        "primary_key_identifier": "2001"
    },

then

    {
        "id": 2002,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",
        "updated_at": null,
        "primary_key_identifier": "2002"
    }
]

?

[macropay-solutions]

But on our implementation that echoes a json on a new row, and after each echo, we ob_flush and flush, in 20k rows about 300-400 rowa are split while sent, so, the above question about js becomes:

Is JS capable of decoding partial json strings if the json is sent like:

[
    {
        "id": 2000,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",
        "updated_at": null,
        "primary_key_identifier": "2000"

then

    },
    {
        "id": 2001,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",

then

        "updated_at": null,
        "primary_key_identifier": "2001"
    },
    {
        "id": 2002,
        "parent_id": null,
        "client_id": 49507,
        "currency": "EUR",
        "value": "23.00",
        "created_at": "2024-01-17 10:05:07",
        "updated_at": null,
        "primary_key_identifier": "2002"
    }
]

?

[mtarld]

I'll try to dig it out soon, can you in the meantime create and share a reproducer so we are sure we are talking about the same thing?

[macropay-solutions]

We have little experience with symfony, this is a laravel example:

return new \Symfony\Component\HttpFoundation\StreamedJsonResponse(
    Operation::query()->with('client')->lazyByIdDesc(1000, 'id'),
     200,
     [],
     JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES
);

Can be seen in action here https://laravel-crud-wizard.com/laravel-10/laravel-lumen-crud-wizard#operations

Put in textarea

limit=-1

and submit => a prompt to download the json file will appear with a delay (until it builds it in memory). The default condition is id < 20000 to not give memory error.

To compare with less rows in response use in textarea:

limit=-1&created_at[from]=2020-04-23 00:00:00&updated_at[o]=isNotNull

This will be almost instant returning

[
    {
        "id": 9,
        "parent_id": 3,
        "client_id": 177601,
        "currency": "EUR",
        "value": "75.00",
        "created_at": "2024-01-17 10:05:04",
        "updated_at": "2025-04-17 09:06:48",
        "primary_key_identifier": "9"
    },
    {
        "id": 8,
        "parent_id": null,
        "client_id": 45015,
        "currency": "EUR",
        "value": "43.00",
        "created_at": "2024-01-17 10:05:04",
        "updated_at": "2025-04-17 09:15:20",
        "primary_key_identifier": "8"
    },
    {
        "id": 7,
        "parent_id": 3,
        "client_id": 126362,
        "currency": "EUR",
        "value": "95.00",
        "created_at": "2024-01-17 10:05:04",
        "updated_at": "2025-04-16 16:39:25",
        "primary_key_identifier": "7"
    },
    {
        "id": 6,
        "parent_id": 2,
        "client_id": 84224,
        "currency": "EUR",
        "value": "97.00",
        "created_at": "2024-01-17 10:05:04",
        "updated_at": "2025-04-16 13:47:33",
        "primary_key_identifier": "6"
    },
    {
        "id": 5,
        "parent_id": 2,
        "client_id": 97396,
        "currency": "EUR",
        "value": "79.00",
        "created_at": "2024-01-17 10:05:04",
        "updated_at": "2025-04-16 15:32:53",
        "primary_key_identifier": "5"
    },
    {
        "id": 4,
        "parent_id": 2,
        "client_id": 165915,
        "currency": "EUR",
        "value": "89.00",
        "created_at": "2024-01-17 10:05:04",
        "updated_at": "2025-04-16 13:33:50",
        "primary_key_identifier": "4"
    },
    {
        "id": 1,
        "parent_id": null,
        "client_id": 1,
        "currency": "EUR",
        "value": "10.00",
        "created_at": "2024-01-03 19:39:02",
        "updated_at": "2025-04-16 13:51:10",
        "primary_key_identifier": "1"
    }
]

For normal json response (and executed queries) for the above last situation use:

limit=0&created_at[from]=2020-04-23 00:00:00&updated_at[o]=isNotNull&id[to]=20000

to see only the count

or

limit=10&created_at[from]=2020-04-23 00:00:00&updated_at[o]=isNotNull&id[to]=20000

to see all in normal json response.

[mtarld]

As stated here, flushing is your responsibility. Therefore, have you tried something like:

$iterateAndFlush = static function (iterable $iterable): iterable
{
    $i = 0;
    foreach ($iterable as $item) {
        yield $item;

        if ($i && $i % 200 === 0) {
            ob_flush();
            flush();
        }

        ++$i;
    }
}

return new \Symfony\Component\HttpFoundation\StreamedJsonResponse(
    $iterateAndFlush(Operation::query()->with('client')->lazyByIdDesc(1000, 'id')),
     200,
     [],
     JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES
)

?

[macropay-solutions]

@mtarld #60257 (comment)

@mtarld Nope. We tried also with flush and ob_flush. If you read through the related issue from laravel, you will see why.

[mtarld]

Can you share me that piece of code? Maybe something going on here