HTTPClient: implement Stream class

[francescolavra]

It is now possible to stream HTTP request data without knowing in advance the data size (using chunked transfer encoding), and to stream HTTP response data (correctly decoded according to the transfer encoding method used by the server) without requiring an output Stream implementation.
(The existing getStream() method in the HTTPClient class returns the underlying WiFiClient stream which is not aware of the HTTP transfer encoding.)

When using chunked encoding in an HTTP request, each call to write() results in the transmission of a separate chunk. An example code snippet is below:

addHeader("Content-Type", "plain/text");
addHeader("Transfer-Encoding", "chunked");
sendHeader("POST");
print("first chunk");
print("second chunk");
endRequest();

A primary use case for using the Stream implementation to receive an HTTP response is when parsing large response data: using the HTTPClient Stream implementation, the parser does not require an in-memory buffer where the entire HTTP response data is stored.

[JAndrassy]

a POST example with print and println could demonstrate the sending part.
(is it possible to send chunked?)

[francescolavra]

No, currently the HTTPClient supports only receiving with chunked encoding, and doesn't implement the sending part; that was the case also without this Stream implementation.

[JAndrassy]

No, currently the HTTPClient supports only receiving with chunked encoding, and doesn't implement the sending part; that was the case also without this Stream implementation.

but with this PR HTTPClient has println and it can't be used, because the payload can be send only at once with get, post functions

[francescolavra]

but now HTTPClient has println and it can't be used, because the payload can be send only at once with get, post functions

Actually it's not true that the payload can be sent only at once: you can start an HTTP request via sendHeader() and then send the payload separately. Before, you could send the payload only by getting the underlying WiFiClient stream, via getStream(); now you can also stream directly to the HTTPClient.
So yes, println can be used (and it could be used also without this Stream implementation).

[JAndrassy]

but now HTTPClient has println and it can't be used, because the payload can be send only at once with get, post functions

Actually it's not true that the payload can be sent only at once: you can start an HTTP request via sendHeader() and then send the payload separately. Before, you could send the payload only by getting the underlying WiFiClient stream, via getStream(); now you can also stream directly to the HTTPClient.
So yes, println can be used (and it could be used also without this Stream implementation).

ok. sorry. I didn't have an opportunity to use the ESP8266HTTPClient. But to send a POST request the Content-length header should be set or chunked transfer encoding should be used. And providing the Stream to a function which composes the request body usually means that we don't know the content-length before.
One could use my StreamLib library's ChunkedPrint, but it would be better if the ESP8266HTTPClient would support chunked POST body.

[francescolavra]

ok. sorry. I didn't have an opportunity to use the ESP8266HTTPClient. But to send a POST request the Content-length header should be set or chunked transfer encoding should be used. And providing the Stream to a function which composes the request body usually means that we don't know the content-length before.
One could use my StreamLib library's ChunkedPrint, but it would be better if the ESP8266HTTPClient would support chunked POST body.

OK, I added chunked encoding support in HTTP request transmission as well.

[d-a-v]

This proposal

• makes the http client a stream class
  • read() is implemented but not read(buf,len)
• allows chunk encoding for both receiving and sending data
• initial sendRequest() is not aware of chunk encoding (maybe ok ?)
• has no example / test / doc

Legacy behaviour seems to be OK.
For a better understanding, can someone tell about a use case, or an implicit use case where this addition is useful ?

[francescolavra]

An example where this addition would be useful is at https://github.com/francescolavra/arduino-iota-client/blob/f6ce76453cfa1984df14b3be9b6de478a44e40b6/src/IotaClient.cpp#L355, where we need to parse a JSON string coming from an HTTP response: the existing code calls the getStream() method of the HTTPClient class, but this method returns the underlying WiFiClient stream which is not aware of the HTTP transfer encoding, so the parsing will fail if the HTTP server uses chunked encoding in its response; with this addition, the above code can be changed to pass _client instead of _client.getStream() as input stream argument to the deserializeJson() method, and it will work for both non-chunked and chunked encoding.

[d-a-v]

LGTM

[d-a-v]

peek() is using available() to manage chunk-encoded streams.
Can read() do the same instead of duplicating code ?

[francescolavra]

Yes, I think something like below would work for read():

while (!available()) {
    if (!connected()) {
        return -1;
    }
    yield();
}
if (_transferEncoding == HTTPC_TE_CHUNKED) {
    _chunkOffset++;
}
return _client->read();

But I'm afraid I don't have time to dig up my development environment and an ESP8266 to test this code.

[astrolemonade]

Hey! What changes should be done to finally merge these?