Space at end of query string is trimmed

[Laerte]

Faced this issue while writing a spider, basically if we don't percent-encoding the space or only have one parameter the space is trimmed (if we have more parameters but the one with space is at end its also trimmed).

Find below the snippet:

from w3lib.url import safe_url_string
from urllib.parse import parse_qsl, urlparse

url = safe_url_string("https://httpbin.org/anything?keyword=A ")
assert (dict(parse_qsl(urlparse(url).query))["keyword"] == "A ") is False

url = safe_url_string("https://httpbin.org/anything?keyword=A%20")
assert (dict(parse_qsl(urlparse(url).query))["keyword"] == "A ") is True

url = safe_url_string("https://httpbin.org/anything?keyword=A &dummy=value")
assert (dict(parse_qsl(urlparse(url).query))["keyword"] == "A ") is True

url = safe_url_string("https://httpbin.org/anything?keyword=A%20&dummy=value")
assert (dict(parse_qsl(urlparse(url).query))["keyword"] == "A ") is True

url = safe_url_string("https://httpbin.org/anything?dummy=value&keyword=A ")
assert (dict(parse_qsl(urlparse(url).query))["keyword"] == "A ") is False

url = safe_url_string("https://httpbin.org/anything?dummy=value&keyword=A%20")
assert (dict(parse_qsl(urlparse(url).query))["keyword"] == "A ") is True

[Gallaecio]

I think the current behavior is OK for scenarios where you want a behavior consistent with that of a web browser, i.e. with what would happen if you pasted that URL in the address bar of a web browser.

Maybe we should have different functions for the different behaviors. Not sure how to call them, though.

[wRAR]

I've recently found:

    # Only lstrip url as some applications rely on preserving trailing space.
    # (https://url.spec.whatwg.org/#concept-basic-url-parser would strip both)
    url = url.lstrip(_WHATWG_C0_CONTROL_OR_SPACE)
    scheme = scheme.strip(_WHATWG_C0_CONTROL_OR_SPACE)

(from urllib.parse.urlsplit()). Added in python/cpython#102508 which I haven't read.