improve pseudo-html entity conversion

[alexcjohnson]

• Fix Some HTML entities are double-decoded. #2927 - Move entity conversion later in the convertToTspans pipeline, so we don't try to interpret < as < and make a tag out of it
• Support all numbered entities, decimal and hex - including the correct ampersand code!
• Improve performance - instead of running the string through a separate regex replace for every entity, just run it through one regex replace that finds all entities. Note that this would make it feasible, performance-wise, to include ALL named html entities, but the full list is over 2000 items, and there are other options (unicode literals, or numbered entities if you want your code/JSON to be pure ASCII) so I suggest we keep it at just the existing set.
• Make the GL version work the same as the SVG version. Rather than fix the fixEntities function there, which was overly aggressive about finding entities, I just pointed it to the SVG version. It's possible that the simple loop-based function there would be faster, but making it work correctly would have slowed it down and made it more complicated, and anyway the other transformations in html2unicode use regex replacements so I guess this wasn't hyper-optimized anyway.

cc @etpinard

[alexcjohnson]

No, the ampersand is not ...

[etpinard]

Oops. My bad.

[alexcjohnson]

oh haha I assumed I did that 🙈

[etpinard]

To be fair, not just IE doesn't support fromCodePoint, 2015 Chrome and 2014 FF didn't have it either.

[alexcjohnson]

Fair enough - though we don't support such old versions of Chrome and FF, do we?

[etpinard]

do we?

We used to have a blob on https://plot.ly/javascript/ (that I can't find right now) saying plotly.js is supported in all SVG-enabled browsers. But yeah, the fraction of users still using Chrome41 and FF29 must be tiny.

[etpinard]

Very nice here.

Now string_mapping is only required in svg_text_utils.js. Would you mind moving the entityToUnicode lookup table to svg_text_utils.js and 🔪 string_mapping.js?

[alexcjohnson]

good call -> 🔪 string_mapping.js in a943db2

[etpinard]

Support all numbered entities, decimal and hex - including the correct ampersand code!

Brilliant ✨

[etpinard]

Very nice PR. 🥇

I tried to find a case where the new convertEntities logic would contradict the works of #804, but to no avail.

I'll give you a 💃 , but I wouldn't mind waiting a few days before merging it in case we need to patch 1.40.0.

---

Now about,

Note that this would make it feasible, performance-wise, to include ALL named html entities, but the full list is over 2000 items, and there are other options (unicode literals, or numbered entities if you want your code/JSON to be pure ASCII) so I suggest we keep it at just the existing set.

If someone really wants us to decode all named html entities, I suggest we make an optional component (similar to calendars) that would require some third-party module (e.g. he) to handle the conversions.