Allow URL-encoded newlines by replacing CR/LF with spaces in httplib

[tim-janik]

Description:
Currently, the cpp-httplib library rejects requests containing CR, LF, or NUL characters within field values, as per this commit:
975cf0d
Due to this issue:
#1908
Released with cpp-httplib v0.17.1.

This behavior is problematic for use cases where URLs include encoded newlines (e.g., %0A), for instance in the llama-server project (PR#11150) where a multi-line string in a URL is used to pre-fill the prompt text field (newlines are useful and needed in that context).

Steps to Reproduce:

1. Send a request with a URL containing %0A (e.g., http://localhost:8080/?something=%0A).
2. A cpp-httplib server returns a 400 error, indicating an invalid request.

Expected Behavior:
According to RFC 9110, servers should replace CR, LF, and NUL characters with spaces in field values or rejecting the request. In the above case, using spaces would be vastly preferable, instead of not allowing the server logic to handle requests at all.
For instance www.google.com/search?q=cpp-httplib%0A also continues to process HTTP requests in the face of %0A in a URL.

Current Behavior:
The library rejects any request containing CR, LF, or NUL in field values, resulting in a 400 error.

Proposed Solution:
Modify e.g. the parse_header() function in httplib.h to replace CR, LF, and NUL characters with spaces instead of rejecting the request.

[kenballus]

You're getting a couple of things mixed up.

1. The term "field value" in the HTTP RFCs refers specifically to header or trailer field values. If any of these bytes are encountered within a request's URL, it is always an error. Replacing with SP is not permissible in that circumstance.
2. The URL you provided doesn't contain CR, LF, or NUL. It contains only a percent-encoded LF, which is totally allowed within URLs. (This is the whole point of percent encoding!)

In summary, this behavior is a bug, but not for the reason you're suggesting. It seems that the problem is that cpp-httplib checks for CR, LF, and NUL after evaluating percent-encoded octets, but it should apply that check before.

[yhirose]

@tim-janik @kenballus thanks for clarifying this situation. I now agree it's a bug in cpp-httplib. Thanks!

[yhirose]

@tim-janik, @ngxson, I added the following unit test to send "/?something=%0A" to reproduce this problem, but it returns 200 and I don't see any problem in the cpp-httplib server. Could give me the code snippet of your code that can reproduce it. Thanks!

282f2fe

[tim-janik]

@tim-janik, @ngxson, I added the following unit test to send "" to reproduce this problem, but it returns 200 and I don't see any problem in the cpp-httplib server. Could give me the code snippet of your code that can reproduce it. Thanks!

282f2fe

The issue described happens with the interactive WebUI of llama-server, here is how to trigger it:

• Checkout workaround: https://github.com/tim-janik/llama.cpp/tree/textarea-query-string-httplib-good
• Build: `cmake -B build -DGGML_CUDA=ON && nice -n19 cmake --build build --config Release -j
• Run: llama.cpp/build/bin/llama-server -m Phi-3.5-mini-instruct.Q8.gguf
• Testing:

# Navigate to the WebUI with ?m=..., click Send, this will send a POST request and then generate
firefox http://localhost:8080/?m=Hello%0A
# Or, navigate to the WebUI with ?q=..., this will automatically send a POST request and then generate
firefox http://localhost:8080/?q=Hello%0A
> request: GET / 127.0.0.1 200
> request: POST /v1/chat/completions 127.0.0.1 200

• Checkout the %0A issue: https://github.com/tim-janik/llama.cpp/tree/textarea-query-string-httplib-bad
• Build & Run
• Testing:

# Navigate to the WebUI, this will automatically send a POST request and then 400
firefox http://localhost:8080/?q=Hello%0A
> request: GET / 127.0.0.1 200
> request: POST /v1/chat/completions  400

The diff between the two branches is merely:

diff --git a/examples/server/httplib.h b/examples/server/httplib.h
index c2f12dd2..b6b76665 100644
--- a/examples/server/httplib.h
+++ b/examples/server/httplib.h
@@ -4224,7 +4224,7 @@ inline bool parse_header(const char *beg, const char *end, T fn) {
     // value MUST either reject the message or replace each of
     // those characters with SP before further processing or
     // forwarding of that message.
-    static const std::string CR_LF_NUL("\r\n\0", 3);
+    static const std::string CR_LF_NUL("\r\0", 3); // CR_LF_NUL("\r\n\0", 3); // https://github.com/ggerganov/llama.cpp/issues/11335#issuecomment-2614759009
     if (val.find_first_of(CR_LF_NUL) != std::string::npos) { return false; }
 
     fn(key, val);

[yhirose]

@tim-janik I still don't fully understand your situation, yet. This parse_header parses HTTP headers, and this particular check is for a HTTP header value, not a URL. I just wonder if the string http://localhost:8080/?q=Hello%0A is in a HTTP header value. If so, what's the HTTP header key?

[tim-janik]

Opening the network tab before "Send" shows this POST request (using a different Port here and pasting from Firefox):

POST |  
-- | --
scheme | http
host | localhost:1111
filename | /v1/chat/completions
 Address | 127.0.0.1:1111
POST
scheme
	http
host
	localhost:1111
filename
	/v1/chat/completions
Address
	127.0.0.1:1111
Status
200
OK
VersionHTTP/1.1
Transferred39.25 kB (39.07 kB size)
Referrer Policystrict-origin-when-cross-origin
DNS ResolutionSystem	
Access-Control-Allow-Origin
	http://localhost:1111/
Content-Type
	text/event-stream
Keep-Alive
	timeout=5, max=100
Server
	llama.cpp
Transfer-Encoding
	chunked
Accept
	*/*
Accept-Encoding
	gzip, deflate, br, zstd
Accept-Language
	en-US,en;q=0.5
Connection
	keep-alive
Content-Length
	512
Content-Type
	application/json
Cookie
	CSRF-Token-XYZ
DNT
	1
Host
	localhost:1111
Origin
	http://localhost:1111/
Priority
	u=4
Referer
	http://localhost:1111/?q=Hello%0A
Sec-Fetch-Dest
	empty
Sec-Fetch-Mode
	cors
Sec-Fetch-Site
	same-origin
Sec-GPC
	1
User-Agent

There is a %0A in the Referer field, is that possibly tripping up cpp-httplib?

[yhirose]

@tim-janik thanks for the detailed information!

[yhirose]

@tim-janik could you try #2033 please?