API docs are not being properly indexed for search

[waylan]

The default behavior is for search to use the heading of the relevant section as the title of the search result. However, the code spans within headings of the API docs are being HTML escaped in the results and many of the obvious headings are not being returned as results. I'm assuming the second issue is related to the first in that the correct text is not getting indexed. In other words, if only the plain text content of a heading was being indexed, then that would result in better search results.

Consider the following example. ESCAPED_CHARS is an instance attribute of the Markdown class. One might expect that its relevant section (ESCAPED_CHARS) would be returned in a search for the string ESCAPED_CHARS, but it is not. However, the method markdown.inlinepatterns.EscapeInlineProcessor.handleMatch is in the results because it mentions ESCAPED_CHARS in the body of its documentation. Yet, in the search result, the title is the text <code class="doc-symbol doc-symbol-toc doc-symbol-method"></code>&nbsp;handleMatch not handleMatch or markdown.inlinepatterns.EscapeInlineProcessor.handleMatch as one might expect. Frustratingly, the search term handleMatch does not return that result at all. Yet, it is in the results for the search term doc-symbol-method, which shouldn't even be indexed, as it is an HTML class assigned to the code span, not text.

@pawamoy do you have any insight into this? I have not yet looked at the code and am not sure how the mkdocsstrings extension passes its generated pages to search for indexing.

[pawamoy]

We recently reached our first funding goal, and therefore released the show_symbol_type_heading option. By releasing it, it was activated in Python-Markdown's docs, because the option was already set to true. Previously it had no effect.

While we investigate this issue, you can set show_symbol_type_heading: false in mkdocs.yml 🙂

[waylan]

Okay, so the symbol type code span is being included in the index. I failed to recognize that as the body text of the code span wasn't included in the search result. But now I realize that the body text is added by CSS so it isn't in the HTML. Looks like the code element and the non-breaking space that follows it need to be removed before indexing then.

[pawamoy]

We could also try to somehow unescape the title before adding it to the page. IIUC escaping is done in the format_result function the search plugin's main.js script:

function formatResult (location, title, summary) {
  return '<article><h3><a href="' + joinUrl(base_url, location) + '">'+ escapeHtml(title) + '</a></h3><p>' + escapeHtml(summary) +'</p></article>';
}

That would at least fix the rendering issue in the search page.

[waylan]

While we investigate this issue, you can set show_symbol_type_heading: false in mkdocs.yml

Actually, this setting makes no difference. To be clear, changing the setting does remove the symbol type from the documentation, but it does not remove it from the search results. I expect that should be a clue as to where the problem is.

While testing this, I also noticed that any search result which returns the module (top heading in the page which included the module level docstring) also has the entire header (name of module) in escaped code span (for example, <code>markdown</code> for a search result pointing at the markdown module. There are no symbol types there and in fact this issue existed prior to the release of 1.8.

[pawamoy]

Oh, yes, when we don't separate signatures from the headings, then headings are the signatures, and they're wrapped in code tags. I suppose we could wrap them in spans instead (as you suggested in previous discussions), with the right CSS styling, but they would be escaped as <span ...>Markdown</span> once again?

About symbols still appearing in the search results, try also setting show_symbol_type_toc: false 🤷

[waylan]

About symbols still appearing in the search results, try also setting show_symbol_type_toc: false

That fixes the issue. The show_symbol_type_heading option has no bearing on the issue either way, only show_symbol_type_toc. And that makes sense. MkDocs uses the toc title of a page for the search index. Regardless, these code spans should not be making it into the search index regardless of the show_symbol_type_toc setting.

However, we still have the issue with the module titles. I should point out that all of the headings (for functions, classes, attributes, methods, etc) have code spans in them, which get stripped. But only the module titles do not get their tags stripped. The offending code span here is not the symbol type, but the code span which contains the module name.

[waylan]

I just deployed the docs with the changes in #1434 and search results are better (they are actually usable). However, we still have the issue with the module names not being striped of tags before being indexed. Therefore, this issue is still unresolved.

[waylan]

However, we still have the issue with the module names not being striped of tags before being indexed. Therefore, this issue is still unresolved.

Upon further investigation, it appears that this issue is a symptom of mkdocs/mkdocs#3357. MkDocs' search plugin gets its entries from 2 sources: (1) a page and (2) the sections of a page. The sections are obtained by iterating over the TOC for a page. The TOC already sanitizes section headings and therefore we don't have an issue with them (for the most part). However, until mkdocs/mkdocs#3357 is resolved, there is no guarantee that a page title is sanitized. Unfortunately, the search plugin assumes that it is. I could submit a change to mkdocs which removes that assumption (by explicitly sanitizing the title within the search plugin before indexing), but it would be more sensible to just fix mkdocs/mkdocs#3357 directly.

That said, there is still the issue with symbol types in the TOC. They should not be getting through the TOC sanitation. I wonder if there is a change to mkdocsstrings which could work around the issue entirely. That needs further investigation. It is also possible that the TOC extension itself needs better sanitation. I'm leaving this open until we can get definitive answers on those matters.

[waylan]

It is also possible that the TOC extension itself needs better sanitation.

After I posted that I remembered that the type symbols are added by mkdocsstrings after Markdown is done with the TOC. Therefore, changes to the TOC extension would have no effect. I wonder if perhaps mkdocsstrings needs to take a different approach here so as not to dirty the TOC and page title. And that is what I want to investigate.

[waylan]

we still have the issue with the module names not being striped of tags before being indexed.

After some investigation, it is now clear that the as-yet unresolved problem lies with MkDocs itself. See mkdocs/mkdocs#3357, which is specifically about ensuring page titles do not contain markup. When that is resolved, markup will not be passed to the search index for whole-page entries.

Also of note is mkdocs/mkdocs#3560. If that (or something similar) gets merged, then we can turn show_symbol_type_toc back on without ill effects. I am leaving this open as a reminder about that.