Rustc takes 3.5 GB RSS to check `vtparse`

[jyn514]

I tried this code: wezterm/wezterm@fa4bbbd

I expected to see this happen: The compiler builds the code in a reasonable amount of memory (say, < 800 MB).

Instead, this happened: The compiler uses over 3 GB, causing docs.rs to kill the process: rust-lang/docs.rs#1247

Meta

rustc --version --verbose:

> rustc --version -v
rustc 1.51.0-nightly (a62a76047 2021-01-13)
binary: rustc
commit-hash: a62a76047ea24aad7639f14eb3ce0e620b77bdb7
commit-date: 2021-01-13
host: x86_64-unknown-linux-gnu
release: 1.51.0-nightly

-Z time-passes

> cargo rustc --profile=check -- -Z time-passes
    Checking utf8parse v0.2.0
    Checking vtparse v0.4.0 (/home/joshua/test-rustdoc/wezterm/vtparse)
time: 0.002; rss: 52MB	parse_crate
time: 0.000; rss: 52MB	attributes_injection
time: 0.000; rss: 53MB	incr_comp_prepare_session_directory
time: 0.000; rss: 53MB	incr_comp_garbage_collect_session_directories
time: 0.000; rss: 53MB	recursion_limit
time: 0.000; rss: 53MB	plugin_loading
time: 0.000; rss: 53MB	plugin_registration
time: 0.000; rss: 53MB	pre_AST_expansion_lint_checks
time: 0.000; rss: 53MB	crate_injection
time: 0.000; rss: 56MB	pre_AST_expansion_lint_checks
time: 0.000; rss: 56MB	pre_AST_expansion_lint_checks
time: 0.174; rss: 83MB	expand_crate
time: 0.000; rss: 83MB	check_unused_macros
time: 0.174; rss: 83MB	macro_expand_crate
time: 0.000; rss: 83MB	maybe_building_test_harness
time: 0.002; rss: 83MB	AST_validation
time: 0.000; rss: 83MB	maybe_create_a_macro_crate
time: 0.002; rss: 88MB	complete_gated_feature_checking
time: 0.196; rss: 88MB	configure_and_expand
time: 0.000; rss: 88MB	prepare_outputs
time: 0.000; rss: 88MB	blocked_on_dep_graph_loading
time: 0.013; rss: 97MB	hir_lowering
time: 0.004; rss: 97MB	early_lint_checks
time: 0.000; rss: 98MB	setup_global_ctxt
time: 0.001; rss: 98MB	dep_graph_tcx_init
time: 0.002; rss: 98MB	create_global_ctxt
time: 0.000; rss: 103MB	looking_for_entry_point
time: 0.000; rss: 103MB	looking_for_plugin_registrar
time: 0.000; rss: 103MB	looking_for_derive_registrar
time: 0.009; rss: 108MB	misc_checking_1
time: 0.007; rss: 116MB	type_collecting
time: 0.000; rss: 116MB	impl_wf_inference
time: 0.000; rss: 150MB	unsafety_checking
time: 0.000; rss: 150MB	orphan_checking
time: 0.059; rss: 150MB	coherence_checking
time: 0.016; rss: 156MB	wf_checking
time: 0.572; rss: 221MB	item_types_checking
time: 0.064; rss: 245MB	item_bodies_checking
time: 0.719; rss: 245MB	type_check_crate
time: 0.002; rss: 245MB	match_checking
time: 0.004; rss: 245MB	liveness_and_intrinsic_checking
time: 0.006; rss: 245MB	misc_checking_2
time: 2.499; rss: 1910MB	MIR_borrow_checking
time: 0.761; rss: 3685MB	MIR_effect_checking
time: 0.000; rss: 3685MB	layout_testing
time: 0.002; rss: 3685MB	death_checking
time: 0.001; rss: 3685MB	unused_lib_feature_checking
time: 1.156; rss: 3393MB	crate_lints
time: 0.007; rss: 3393MB	module_lints
time: 1.164; rss: 3393MB	lint_checking
time: 0.006; rss: 3393MB	privacy_checking_modules
time: 1.176; rss: 3393MB	misc_checking_3
time: 0.016; rss: 3393MB	generate_crate_metadata
time: 0.004; rss: 3393MB	codegen_crate
time: 0.000; rss: 3393MB	assert_dep_graph
time: 0.001; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::type_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::generics_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::predicates_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::mir_const_qualif)
time: 0.020; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::mir_for_ctfe)
time: 0.001; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::optimized_mir)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::coverageinfo)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::covered_file_name)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::covered_code_regions)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::promoted_mir)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::unsafety_check_result)
time: 0.002; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::typeck)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::diagnostic_only_typeck)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::used_trait_imports)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::mir_borrowck)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::eval_to_allocation_raw)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::eval_to_const_value_raw)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::check_match)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::symbol_name)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::codegen_fn_attrs)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::codegen_fulfill_obligation)
time: 0.001; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::specialization_graph_of)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::adt_drop_tys)
time: 0.000; rss: 3394MB	encode_query_results_for(rustc_middle::ty::query::queries::unused_generic_params)
time: 0.026; rss: 3394MB	encode_query_results
time: 0.028; rss: 3394MB	incr_comp_serialize_result_cache
time: 0.028; rss: 3394MB	incr_comp_persist_result_cache
time: 0.000; rss: 3394MB	incr_comp_serialize_dep_graph
time: 0.002; rss: 3394MB	incr_comp_encode_serialized_dep_graph
time: 0.003; rss: 3394MB	incr_comp_encode_dep_graph
time: 0.003; rss: 3394MB	incr_comp_persist_dep_graph
time: 0.031; rss: 3394MB	serialize_dep_graph
time: 0.010; rss: 3330MB	free_global_ctxt
time: 0.000; rss: 3330MB	join_worker_thread
time: 0.001; rss: 3330MB	copy_all_cgu_workproducts_to_incr_comp_cache_dir
time: 0.001; rss: 3330MB	finish_ongoing_codegen
time: 0.000; rss: 3330MB	llvm_dump_timing_file
time: 0.000; rss: 3330MB	serialize_work_products
time: 0.000; rss: 3330MB	incr_comp_finalize_session_directory
time: 0.000; rss: 3330MB	link_binary_check_files_are_writeable
time: 0.000; rss: 3330MB	link_binary_remove_temps
time: 0.000; rss: 3330MB	link_binary
time: 0.000; rss: 3330MB	link_crate
time: 0.002; rss: 3330MB	link
time: 5.477; rss: 3330MB		total
    Finished check [unoptimized + debuginfo] target(s) in 6.10s

The main offenders that stand out are

time: 2.499; rss: 1910MB	MIR_borrow_checking
time: 0.761; rss: 3685MB	MIR_effect_checking

which spike up from time: 0.006; rss: 245MB misc_checking_2 before. It would be nice to reduce this somehow.

[Aaron1011]

It looks like the high memory usage is being driven by the MIR for the TRANSITIONS static: https://github.com/wez/wezterm/blob/fa4bbbd077837d4eafe26805079f95a51054c4d7/vtparse/src/transitions.rs#L300-L315

It has 33,826 (!!!) basic blocks.

[Aaron1011]

Running cargo expand shows that the large number of basic blocks is caused by the initializer being enormous: https://gist.github.com/Aaron1011/73c3f35c6aa1b07bf5a5d2240bc71b19

[bugadani]

In this concrete case the memory mostly goes to the dataflow framework. The dataflow framework allocates a vector of bitsets for the entire universe. This bitset vector is number of move paths wide and number of basic blocks long. This particular initializer has 33.826 basic blocks and 124559 move paths, which means a single dataflow pass (MaybeUninitializedPlaces, MaybeInitializedPlaces, EverInitializedPlaces) each allocate around 500 MB of memory.
I'm not sure why this memory is not released, or if it is, it's release is not reflected in the rss numbers.