Infinite waiting for CURLOPT_WRITEFUNCTION in case of no response

[accountnujen]

Description

To reach an address that responds every 5 seconds, I use CURLOPT_WRITEFUNCTION. Since I want to maintain an infinite connection, I have CURLOPT_TIMEOUT => 0.

function writer($ch, $data) {
  handler($data);
  return strlen($data);
}

$ch = curl_init("$urlcam/cgi-bin/eventManager.cgi?action=attach&codes=[NewFile]&heartbeat=5");
curl_setopt_array($ch, [
        CURLOPT_CONNECTTIMEOUT => 20,
        CURLOPT_TIMEOUT => 0,
        CURLOPT_RETURNTRANSFER => 1,
        CURLOPT_WRITEFUNCTION => 'writer'
    ]);

$result = curl_exec($ch);
curl_close($ch);

The response looks like this:

--myboundary
Content-Type: text/plain
Content-Length: 9
Heartbeat
--myboundary
Content-Type: text/plain
Content-Length: 9
Heartbeat
...

Everything works as long as there is a response from the url. However, if the url stops responding, nothing happens. No error or warning. The PHP script will stay inside the "writer" function and never leave it.

For example, if you implement this functionality using fopen:

$resource = fopen("http://".$host.$uri, 'r', false, $context);
while (!feof($resource)) {
    print_r(date("d-m-Y H:i:s"));
    $line = fgets($resource);
    if (str_contains($line,"--myboundary")) {
        handler($response);
        $response = '';
    } else {
        $response.=$line;
    }
}

Then the print_r function will be called every minute. Yes, if the url is restored, the fopen connection will not be restored, but at least we can handle the error/state to restart the script. In the case of CURL, this cannot be done as nothing happens.

We need to add functionality to leave the function specified in CURLOPT_WRITEFUNCTION. I suppose you could take CURLOPT_CONNECTTIMEOUT and display an error if the WRITEFUNCTION has not received a response within that time.

PHP Version

8.1.23

Operating System

Ubuntu 20.04

[damianwadley]

What do you mean, it "stays inside the writer function"? If the endpoint isn't writing anything then cURL will wait forever (since you disabled the timeout) for it to start writing data again - or until the connection otherwise terminates. If that's not the behavior you want then you have to write different code.

If you need to support the endpoint potentially not sending data for a while, and the code needing to "interrupt" itself for a moment before waiting for data again, then the standard solution is a select on the socket with some appropriate timeout; if it receives data then great, if not then react accordingly.
Except the whole point of there being a heartbeat mechanism is so that you don't have to deal with that: if the endpoint doesn't give you a ping in an appropriate amount of time, that means its "heart" has stopped and you should consider the connection dead.

Point is, I do see some problems here, but the WRITEFUNCTION isn't one of them.

[accountnujen]

If the endpoint isn't writing anything then cURL will wait forever

Incorrect. If the service is restored, the function has not continued. The URL belongs to me and I can stop the service. If I stop the service and then start it, the "writer" will not continue. It will stay still the same. Roughly speaking, if writer will have this form:

function writer($ch, $data) {
  echo "start ".date("H:i:s");
  handler($data);
  echo "end";
  return strlen($data);
}
$ch = curl_init(....
$result = curl_exec($ch);
echo "Finish";

then I will have nothing happening at all in the logs after disabling and enabling the service on the URL. For example, if I stop the URL service at 16:00:10 and start it at 16:30:00, the logs will show this:

start 16:00:05
end
start 16:00:10
end

That's it. To continue working I will have to restart the script. I will never see echo "Finish" at all.

Except the whole point of there being a heartbeat mechanism is so that you don't have to deal with that: if the endpoint doesn't give you a ping in an appropriate amount of time, that means its "heart" has stopped and you should consider the connection dead.

That's the thing, if the heart stops, nothing happens at all. WRITEFUNCTION does not provide for the possibility of a heart stopping. Even though the URL becomes unavailable, I don't have a connection terminated that doesn't already exist. So some sort of limit needs to be set. For example in fopen the default timeout is set to 60 seconds. I can set a different timeout with this command stream_set_timeout($resource,30);. I think the same should be true for CURL.

[lucasnetau]

curl_exec is a blocking function and will only return once the connection is complete. Since you have disabled timeouts then it will wait indefiently if the connection is open but does not transmit.
Try using CURLOPT_XFERINFOFUNCTION which will be called every second even during stalls and abort the connection if no data is seen for X amount of time.

Your other option is to wrap the curl inside curl_multi and handle the connection in an async loop.

[accountnujen]

Try using CURLOPT_XFERINFOFUNCTION which will be called every second even during stalls and abort the connection if no data is seen for X amount of time.

Either I haven't figured out CURLOPT_XFERINFOFUNCTION, or it refuses to work together with CURLOPT_WRITEFUNCTION. Could you please give me an example of an implementation?

Your other option is to wrap the curl inside curl_multi and handle the connection in an async loop.

I risk losing the event if it occurred between queries, so this option doesn't work well for me.

And the most important thing here:

curl_exec is a blocking function and will only return once the connection is complete. Since you have disabled timeouts then it will wait indefiently if the connection is open but does not transmit.

I disabled because, in my case, I want to wait for the heartbeat forever. And what I'm driving at is that PHP Curl needs another timeout type that will be responsible for the situation in which the URL has stopped responding. And yes, this is my personal situation, however it should work for the rest of us.

Let me give you an example. We are downloading data from the server using CURLOPT_WRITEFUNCTION. Let's say we know that the file will not exceed 1GB. Given our speed and the speed of the server, we can assume that 300 seconds is enough. We set the TIMEOUT limit to 300 seconds. We start the download and after a second the server shuts down (power problems). We managed to download only a few kilobytes. After 60 seconds, the server has been restarted, but our download will not continue. Moreover, it will not stop. It will wait 5 fu**ing minutes and then say connection time out. If we knew that the download was interrupted, we could restart it, but we don't know that. We don't know anything at all.

[lucasnetau]

CURLOPT_NOPROGRESS also needs to be set to 0. From the docs: "PHP automatically sets this option to [true]" and this disabled the XFERINFOFUNCTION callback per https://everything.curl.dev/transfers/callbacks/progress

https://github.com/lucasnetau/reactphp-http-browser-curl/blob/main/src/Browser.php

curl_setopt($curl, CURLOPT_NOPROGRESS, 0);
curl_setopt($curl, CURLOPT_XFERINFOFUNCTION, function($curl, $dl_total, $dl_xfer, $ul_total, $ul_xfer) {
    //Do your timeout checking here
}

I risk losing the event if it occurred between queries, so this option doesn't work well for me.

Not entirely sure what you mean by this. curl_multi allows async use of curl, you won't miss any data.

Let me give you an example. We are downloading data from the server using CURLOPT_WRITEFUNCTION. Let's say we know that the file will not exceed 1GB. Given our speed and the speed of the server, we can assume that 300 seconds is enough. We set the TIMEOUT limit to 300 seconds. We start the download and after a second the server shuts down (power problems). We managed to download only a few kilobytes. After 60 seconds, the server has been restarted, but our download will not continue. Moreover, it will not stop. It will wait 5 fu**ing minutes and then say connection time out. If we knew that the download was interrupted, we could restart it, but we don't know that. We don't know anything at all.

Welcome to the world of TCP. If the server goes away without closing the TCP connection your operating system may be handling retries in the background and so the socket that curl is using is not closed and the close functions are not called. Connections may not be established after server restart and the server may silently drop retries. There can be all sorts of reasons for the connection not to be closed, cURL timeouts are there for a reason, if you disable them libcURL may block until you kill it

[accountnujen]

Welcome to the world of TCP.

I'd like to decode TCP as "Typical Clinical Psychiatry" then.
00:00-- I'm creating a query
00:05 - url is disabled.
00:30 - url is available again.
00:35 - another user retrieves data from the url.
01:00 - someone else successfully accesses the url.
02:00 - I'm waiting for something
02:30 - other users have successfully accessed url 100 times
03:00 - I'm still waiting
04:00 - waiting...
05:00 - oops, time out error.
What makes you think this is normal? My opinion is that it should not be.

And I'm just giving you the 1GB example. What if we download a large amount of data? 300 seconds won't be enough. We'll set a limit of 10-20-30 minutes. What, we're gonna wait 30 minutes because somebody's power went out? Why is there a timeout for connection, a timeout for all work, but no timeout for problems during work?

It feels like the PHP team just doesn't have the heart to deal with this, so I'm told to either use workaround or happily report that this is the way it should be. No, it's not supposed to be like that.

I will state the problem again: curl does not stop after a connection is broken and continues to run until timeout is triggered. If the timeout is unlimited, it will never terminate even if the url has been restarted. This should not be the case, it is not normal.

[lucasnetau]

cURL documents this https://everything.curl.dev/usingcurl/timeouts, it is normal.

You need to detect the network failure however you can. The CURLOPT_XFERINFOFUNCTION is designed to do just this task.

Some causes and other options discussed here curl/curl#8345 keep-alive can also be used to detect connection failure.