rustdoc: refactor rustdoc syntax highlighting for a more flexible API

Clients can now use the rustdoc syntax highlighter to classify tokens, then use that info to put together there own HTML (or whatever), rather than just having static HTML output.

Author: nrc