Improve behavior after publishing a new version causes `/latest/` to lose children

[kpreid]

Current behavior:

1. Alice visits https://docs.rs/example/latest/example/really_useful_module/index.html.
2. Alice sees documentation for example::really_useful_module from example version 0.17.0.
3. Alice thinks this is worth revisiting, and bookmarks the page or leaves the tab open.
4. Bob publishes version 0.18.0 of package example, which renames really_useful_module to rum.
5. Alice revisits or reloads.
6. Alice sees a 404 page that says “The requested resource does not exist”, and provides no navigation related to example at all.

This leaves Alice with no easy option to recover other than manual URL editing. It would be useful if the 404 page:

• Acknowledged that the package and version exist, even though the specific documentation file does not.
• Provided links to recover such as to the root of the version docs (https://docs.rs/example/latest/) and to a page where other versions could be browsed (https://docs.rs/crate/example/).
• Extra fancy stretch goal: lookup the most recent previous version which had such a page and link to that.

Please note that I'm proposing that the 404 page contains useful links, not that it become a redirect to any of the above. Redirects are (in my opinion) bad UX in this scenario because they may confuse the user as to where they were and what they were previously looking at, and also, the fact that a new version was published that deleted a module is potentially highly relevant information.

[syphar]

I like the idea.
we probably would just have to define a new error type & page with the necessary information.

Also note another option:
when you switch platforms in a release and the other platform doesn't have the current file, we convert the URL into a search and run a rustdoc search on the other platform. Not sure if this would be a useful other option for this 404 page.

[ibilalkayy]

@kpreid Here is a suggestion.

How to Solve the Issue

1. Design a New 404 Error Page:

   • Information Display: The 404 page should clearly state that the requested resource does not exist but that the package and version are valid.
   • Navigation Options: Provide direct links to:
     • The root documentation page for the current version.
     • A page to browse other versions of the package.
   • Previous Version Lookup (Stretch Goal):
     • Implement a feature to search for the most recent previous version where the resource was available and link to that version.

2. Implementing Navigation Links:

   • Root Documentation Link: This can be achieved by extracting the base URL from the request and appending the version to navigate to the root documentation for that version.
   • Version Browsing Link: Provide a link to the package’s main page where users can select different versions to view.
   • Previous Version Lookup:
     • Implement logic to search through the version history of the package to find the last version where the resource existed. This might involve querying metadata or change logs.

Example 404 Page Design

Here’s a conceptual design for the improved 404 error page:

---

404 Not Found

The requested resource does not exist in the current version of the package.

Package and Version Information:

• Package: example
• Version: 0.18.0

Navigation Options:

• Return to the root documentation for this version
• Browse other versions of the package

(Optional) Previous Version with Resource:

• The requested resource was available in version 0.17.0. [View documentation for this version]