Tracking Issue for Metrics Initiative

[jieyouxu]

The Metrics Initiative

This is a tracking issue for the Metrics Initiative. Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, please:

• Discuss the Metrics Initiative in the zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/Metrics.20initiative, or
• Discuss the Metrics Initiative in the internals thread: https://internals.rust-lang.org/t/no-telemetry-in-the-rust-compiler-metrics-without-betraying-user-privacy/19275, or
• Open a new dedicated issue about the specific matter and label it suitably with A-metrics Area: Metrics , then link back to this tracking issue.

Please file dedicated issues for specific concerns that you wish to register. Discussions or concerns become very hard to track on GitHub issues once they reach more than a couple of comments, and GitHub's UI will then collapse discussions.

Context

• Vision Blogpost: https://estebank.github.io/rustc-metrics.html
• MCP: Have rustc produce a local log of execution metrics compiler-team#679
• Related Zulip Conversations
  • Council: https://rust-lang.zulipchat.com/#narrow/stream/392734-council/topic/Metrics.20Initiative.
  • Compiler: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Metrics.20Initiative
  • Compiler MCP discussion: https://rust-lang.zulipchat.com/#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/Opt-in.20flag.20for.20absolute.20paths.20in.20diagnos.E2.80.A6.20compiler-team.23770
  • Lang: https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/Metrics
• WIP planning document: https://hackmd.io/@yaah/rk7KEwW90

Motivation

Excerpt from the Council Zulip thread as summary (edited by me):

We're envisioning three use cases for the Metrics Initiative:

1. Supporting feature development, e.g. answering specific questions such as when the old and new trait solvers diverge, or helping identify and resolve bugs.
2. Guiding improvements to User Experience, e.g. knowing which compiler errors are causing the most confusion or are hit the most frequently, focusing on improving those first, and verifying that the improvements help.
3. Improving perf feedback loops and insight, e.g. helping identify pathological edge cases, similar to work @nnethercote has done manually in the past

We're focusing initially on the first use case since we see that as the most likely to have a significant impact.
We want to get to the point where other contributors can leverage the metrics to answer their own questions while we continue to build up the supporting infrastructure.

To do that, we'd like to gather specific use cases where people would like to leverage metrics and build the supporting infrastructure around those real-world needs

Guiding Aims

• Trust: Do not violate the trust of our users
  • NO TELEMETRY, NO NETWORK CONNECTIONS
  • Emit metrics locally
  • User information should never leave their machine in an automated manner; sharing their metrics should always be opt-in, clear, and manual.
  • All of this information would only be stored on disk, with some minimal retention policy to avoid wasteful use of users’ hard drives
• Feedback: improving feedback loops to assist with iterative improvement within the project
  • answer questions from real production environments in a privacy-preserving way
  • improve legibility of rare or intermittent issues
  • earlier warnings for ICEs and other major issues on nightly, improving the likelihood that we'd catch them before they hit stable.
    • https://blog.rust-lang.org/2021/05/10/Rust-1.52.1.html
• Performance impact
  • leave no trace (minimize performance impact, particularly for default-enabled metrics)
• Extensible:
  • it should be easy to add new metrics as needed
  • Only add metrics as a way to answer a specific question in mind, with an explicitly documented rationale
• User experience:
  • improving user experience of reporting issues to the project
  • improving the user experience of using the compiler, measuring the impact of changes to user experience

Suggested Use Cases

• Unstable Feature Usage Metrics #129485 - motivation: to help the library team prioritize which features they should stabilize. Suggested by @Amanieu

TODO

• Inform project members of the initiative and gather real-world needs to build initial metrics infrastructure around
• Complete the metrics loop (e.g., design and implement tools to send metrics back to the project for analysis, tools to analyze metrics locally and notify users when issues we want insight into have been encountered, analyzing metrics in crater runs)
• Automated cleanup of metrics to prevent unbounded disk usage

Concerns and Related Issues

• ICE dumps leak user names etc #128594
• Long ty names written to disk can leak username info in paths #129296

Related Labels

• A-metrics Area: Metrics for issues and PRs containing discussions, bugs, implementation work or are otherwise related to metrics.
• -Zmetrics-dir Unstable option: metrics directory for the unstable metrics output directory flag.

Implementation History

• Add -Zmetrics-dir=PATH to save diagnostic metadata to disk #128702

[jieyouxu]

cc @yaahc and @estebank I created a tracking issue for the Metrics Initiative since I couldn't find any existing ones, please edit as you see fit.

[VorpalBlade]

Is there a link to an RFC or similar for this that describes how this will work? I cannot find any (public) overall design doc.

EDIT: For something as contentious as metrics/telemetry it is extra important to make everything related to the proposal clear and public.

[jieyouxu]

Is there a link to an RFC or similar for this that describes how this will work? I cannot find any (public) overall design doc.

EDIT: For something as contentious as metrics/telemetry it is extra important to make everything related to the proposal clear and public.

Thanks, this was fixed by linking to the vision document blog posts, planning document and the MCP