generate sitemap

[SidVal]

Hi.

Is it possible to create a sitemap for the docsify site?

[QingWei-Li]

Impossible. You can create it manually, but I am not sure if the hash router is valid for the search engines.

[SidVal]

This is interesting

JavaScript Crawling and Indexing – Final Results
Let’s start with basic configurations for all the frameworks used for this experiment.

Source: Can Google Properly Crawl and Index JavaScript Frameworks? A JavaScript SEO Experiment

Repo's source: https://github.com/kamilgrymuza/jsseo

---

Crawling

Source: JavaScript vs. Crawl Budget: Ready Player One

---

Final thoughts

Then it is useless to generate a sitemap.
In SEO terms, our website would not have a good impact for search engines. :(

[trusktr]

I want to re-open this I think it'd be valuable to generate a site map, regardless of hash mode. Some people will use the non-hash mode in which case it is useful. Also we have SSR (being fixed in a current PR) and upcoming plans for static site generation, both of which would benefit from a sitemap.

According to this article from 2014, Google can index hash-based routes as separate pages if using a "hash bang" syntax: instead of making your pages have the form example.com/#/some/page they should be of the form example.com/#!/some/page and then Google will consider the hash as part of the URL. Hash-bang is not required anymore since 2018 according to Google.

What's the latest on hash-based routing and SEO?

cc @jhildenbiddle @anikethsaha

EDIT:

According to official words from Google (see links in that article), people are straight up confused (look at the comments). It isn't clear if hash routing works with Google SEO. If you follow and read all the related tweets, you will be confused. In particular, see these two seemingly contradictory tweets:

• https://twitter.com/JohnMu/status/1052471373497552896
• https://twitter.com/JohnMu/status/1052621056488431616

EDIT: According to https://searchengineland.com/google-can-crawl-ajax-just-fine-322254, hashes should be SEO friendly now, and the Google crawler understands hash-based routing (follow hash changes) and indexes content on dynamic page changes (hash changes).

[trusktr]

Based on that last article, I think we should just make sitemaps regardless. If it works with hashes, it works. If it doesn't, it doesn't. But at least for the other cases we'll be covered (especially SSR and static sites).

For static generation, we will need to programmatically assimilate a list of pages (f.e. based on _sidebar.md, _navbar.md, links in pages, etc). This information allows us to know which static pages we need to output. We can also use this information for sitemap output. Static site generation, sitemap generation, or both, would re-use the same code mechanism.

[trusktr]

Ah! This is interesting. I tried to run the Docsify site through Google Search Console's Rich-results test and mobile-friendly test. Here are the results:

• Rich-results test: https://search.google.com/test/rich-results?id=zXJ0x8Kt7ZbTwkp6rT41wA (view the problem details, and view the rendered HTML). It shows that Google understands (waits for) the dynamic hash content.
• Mobile-friendly test: https://search.google.com/test/mobile-friendly?id=2VzK8qyKtlUaGpiJ1Mn5Mg

As you can see in either test, it has issues reading URLs in anchor tags, for example. It has no idea that we will convert them into hash URLs. I think for v5 we should re-consider how we output the anchor tags, so that Google can understand them.

These two tests are basically a window into how the Google Crawler sees and understands web sites (and has no issues loading a page from a hash route).

[trusktr]

By the way, I found these tools while watching the http://web.dev/live conference Day 1 video that was released a few days ago: https://youtu.be/H89hKw06iWs?t=9201 (at 2 hours 33 minutes it goes into the Google Search stuff). The video shows you how to debug SEO problems with it on SPAs and similar. Neat!!

After that the same guy talks about Structured Data, and the main cool feature is that we can place the structures data on the page dynamically any time we change pages, and Google bot reads the information any time we generate it so that it know when/what to index on an SPA. That's a bit off topic from sitemaps though.

I think the bottom line is we can make a sitemap for hash-based SPAs (like Docsify's default mode). It'll be useful regardless, for other modes.

[trusktr]

@waruqi I thought you commented about your xmake sitemap generator (I saw the email). That's neat!