URL_REGEXP is not accurate

[IlanFrumer]

@andrewaustin @petebacondarwin

This commit broke the URL validation: ffb6b2f

For example this URL (which uses a curly brackets macro syntax and should be a perfectly valid URL):

http://ads.xxxxx.com/xxxxxx/?pid=1234&pos=c&r={TIMESTAMP}

Also check this
http://stackoverflow.com/questions/5984116/regular-expression-for-w3c-compliant-urls

[andrewaustin]

{ and } are not allowed in URIs unless they are URL encoded. You can read RFC 3986 for the full rules about URI characters: https://tools.ietf.org/html/rfc3986#section-2 . They are allowed in IRIs though it seems.

RFC 3987 is even more explicit about URIs vs IRIs and these characters:

Systems accepting IRIs MAY also deal with the printable characters in US-ASCII that are not allowed in URIs, namely "<", ">", '"', space, "{", "}", "|", "", "^", and "`", in step 2 above.

So I guess it's a question of should we be validating URLs as URIs (RFC 3986) or IRIs (RFC 3986 - where { and } are allowed).

[IlanFrumer]

@andrewaustin thanks for the clarification
Since we are validating a user input I think it is more clear to use IRI rather than forcing the user to escape some characters.
Also (as i said) this syntax was valid before this commit, so if we choose URI we must add it to the breaking changes list.

Thanks.

[petebacondarwin]

We will make a decision about this before RC.1

[Narretz]

We've been following Chrome in the past in terms of what urls are allowed (not implementation). Both Chrome and Firefox allow curly brackets, so I think we should too.

[gkalpak]

AFAIK, we've never actually followed Chrome in terms of URL validity.
Before ffb6b2f, there was a bug in URL_REGEXP that allowed all non-whitespace characters in hostname (i.e. it never matched port, path, query, hash - it considered it all to be one, big, valid hostname), so the right half of the RegExp never affected anything.

After ffb6b2f and e4bb838 (which built on top of the former), we try to more closely match the URI spec. Browsers seems to be much more forgiving - maybe we should be too (i.e. be more concerned with validating the structure, than the actual characters).