`--extern mycrate=path/to/my/crate/with/random.suffix` fails with "file name should be lib*.rlib or lib*.so"

[tamird]

macro.rs:

extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro_attribute]
pub fn your_macro(_attr: TokenStream, item: TokenStream) -> TokenStream {
    // Macro implementation
    item
}

main.rs:

extern crate macros;

#[macros::your_macro]
fn main() {
    println!("Hello, world!");
}

This works:

rustc --crate-type proc-macro macros.rs -o libmacros.dylib
rustc main.rs --extern macros=libmacros.dylib

This doesn't:

rustc --crate-type proc-macro macros.rs -o libmacros.so
rustc main.rs --extern macros=libmacros.so

error: extern location for macros is of an unknown type: libmacros.so
 --> main.rs:4:1
  |
4 | extern crate macros;
  | ^^^^^^^^^^^^^^^^^^^^

error: file name should be lib*.rlib or lib*.dylib
 --> main.rs:4:1
  |
4 | extern crate macros;
  | ^^^^^^^^^^^^^^^^^^^^

error[E0463]: can't find crate for `macros`
 --> main.rs:4:1
  |
4 | extern crate macros;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0463`.

This has come up in Rust-For-Linux where we'd like to always use the .so suffix, but this causes the build to fail on macOS. If we change the suffix to .dylib, it works on macOS, but not Linux.

[ojeda]

Thanks @jieyouxu for the label -- to provide some context: some kernel developers want to build Linux under macOS, but the kernel does not support that (the macOS support is maintained out-of-tree). So they may need something like this to minimize the differences with upstream. There may be other possible solutions (e.g. using symlinks, if that works, but it would require knowing the names in general). Perhaps this could help other projects to be ported to macOS too.

[tamird]

The toy example I posted above doesn't exactly capture the kernel case. Running the kernel build with RUSTC_LOG=rustc_metadata=debug make -d LLVM=1 rust/kernel.o I see this rustc invocation:

rustc \
 --edition=2021 \
 -Zbinary_dep_depinfo=y \
 -Astable_features \
 -Dnon_ascii_idents \
 -Dunsafe_op_in_unsafe_fn \
 -Wmissing_docs \
 -Wrust_2018_idioms \
 -Wunreachable_pub \
 -Wclippy::all \
 -Wclippy::ignored_unit_patterns \
 -Wclippy::mut_mut \
 -Wclippy::needless_bitwise_bool \
 -Wclippy::needless_continue \
 -Wclippy::no_mangle_with_rust_abi \
 -Wclippy::undocumented_unsafe_blocks \
 -Wclippy::unnecessary_safety_comment \
 -Wclippy::unnecessary_safety_doc \
 -Wrustdoc::missing_crate_level_docs \
 -Wrustdoc::unescaped_backticks \
 -Cpanic=abort \
 -Cembed-bitcode=n \
 -Clto=n \
 -Cforce-unwind-tables=n \
 -Ccodegen-units=1 \
 -Csymbol-mangling-version=v0 \
 -Crelocation-model=static \
 -Zfunction-sections=n \
 -Wclippy::float_arithmetic \
 --target=aarch64-unknown-none \
 -Ctarget-feature="-neon" \
 -Cforce-unwind-tables=n \
 -Zbranch-protection=bti,pac-ret \
 -Copt-level=2 \
 -Cdebug-assertions=y \
 -Coverflow-checks=y \
 -Cforce-frame-pointers=y \
 -Cdebuginfo=1 @./include/generated/rustc_cfg \
 --extern alloc \
 --extern build_error \
 --extern macros=./rust/libmacros.so \
 --extern bindings \
 --extern uapi \
 --emit=dep-info=rust/.kernel.o.d \
 --emit=obj=rust/kernel.o \
 --emit=metadata=rust/libkernel.rmeta \
 --crate-type rlib \
 -L./rust \
 --crate-name kernel rust/kernel/lib.rs \
 --sysroot=/dev/null

which emits

  INFO rustc_metadata::creader resolving crate `macros`                                                                                                                                                                                                                                                                                              
  INFO rustc_metadata::creader falling back to a load                                                                                                                                                                                                                                                                                                
  INFO rustc_metadata::locator dylib reading metadata from: /Users/tamird/src/linux/rust/libmacros.so                                                                                                                                                                                                                                                
  INFO rustc_metadata::locator Rejecting via proc macro: expected false got true                                                                                                                                                                                                                                                                     
  INFO rustc_metadata::locator metadata mismatch                                                                                                                                                                                                                                                                                                     
  INFO rustc_metadata::creader resolving crate `core`
error[E0463]: can't find crate for `macros`                      
  --> rust/kernel/block/mq/operations.rs:26:3    
   |                                                 
26 | #[macros::vtable]                                           
   |   ^^^^^^ can't find crate   

Creating a symlink from libmacros.dylib to libmacros.so and changing --extern macros=./rust/libmacros.so to --extern macros=./rust/libmacros.dylib makes the build work and changes the rustc_metadata logs to:

  INFO rustc_metadata::creader resolving crate `macros`                       
  INFO rustc_metadata::creader falling back to a load
  INFO rustc_metadata::locator dylib reading metadata from: /Users/tamird/src/linux/rust/libmacros.so
  INFO rustc_metadata::creader register crate `macros` (cnum = 4. private_dep = false)                                                                                    
  INFO rustc_metadata::creader resolving crate `bindings`
  INFO rustc_metadata::creader falling back to a load

[jieyouxu]

I suspect this isn't super easy to change, as it probably affects how deps are looked up in a non-trivial way.

[tamird]

This is the source of the rejection:

rust/compiler/rustc_metadata/src/locator.rs

Lines 728 to 752 in f79fae3

	if file.starts_with("lib") && (file.ends_with(".rlib") || file.ends_with(".rmeta"))
	|| file.starts_with(self.target.dll_prefix.as_ref())
	&& file.ends_with(self.target.dll_suffix.as_ref())
	{
	// Make sure there's at most one rlib and at most one dylib.
	// Note to take care and match against the non-canonicalized name:
	// some systems save build artifacts into content-addressed stores
	// that do not preserve extensions, and then link to them using
	// e.g. symbolic links. If we canonicalize too early, we resolve
	// the symlink, the file type is lost and we might treat rlibs and
	// rmetas as dylibs.
	let loc_canon = loc.canonicalized().clone();
	let loc = loc.original();
	if loc.file_name().unwrap().to_str().unwrap().ends_with(".rlib") {
	rlibs.insert(loc_canon, PathKind::ExternFlag);
	} else if loc.file_name().unwrap().to_str().unwrap().ends_with(".rmeta") {
	rmetas.insert(loc_canon, PathKind::ExternFlag);
	} else {
	dylibs.insert(loc_canon, PathKind::ExternFlag);
	}
	} else {
	self.crate_rejections
	.via_filename
	.push(CrateMismatch { path: loc.original().clone(), got: String::new() });
	}

The problem is that the compiler is trying to guess the type of the library; it isn't enough the the user gave the path because we also need to decide if it's an rlib, rmeta, or dylib, and we do that based on the file name.

Later in the process we'll load the metadata from the file, but the implementations are quite divergent:

rust/compiler/rustc_codegen_ssa/src/back/metadata.rs

Lines 52 to 83 in f79fae3

	impl MetadataLoader for DefaultMetadataLoader {
	fn get_rlib_metadata(&self, target: &Target, path: &Path) -> Result<OwnedSlice, String> {
	load_metadata_with(path, |data| {
	let archive = object::read::archive::ArchiveFile::parse(&*data)
	.map_err(|e| format!("failed to parse rlib '{}': {}", path.display(), e))?;
	for entry_result in archive.members() {
	let entry = entry_result
	.map_err(|e| format!("failed to parse rlib '{}': {}", path.display(), e))?;
	if entry.name() == METADATA_FILENAME.as_bytes() {
	let data = entry
	.data(data)
	.map_err(|e| format!("failed to parse rlib '{}': {}", path.display(), e))?;
	if target.is_like_aix {
	return get_metadata_xcoff(path, data);
	} else {
	return search_for_section(path, data, ".rmeta");
	}
	}
	}
	Err(format!("metadata not found in rlib '{}'", path.display()))
	})
	}
	fn get_dylib_metadata(&self, target: &Target, path: &Path) -> Result<OwnedSlice, String> {
	if target.is_like_aix {
	load_metadata_with(path, |data| get_metadata_xcoff(path, data))
	} else {
	load_metadata_with(path, |data| search_for_section(path, data, ".rustc"))
	}
	}

Solving this would probably require finding a way to interrogate the files without relying on their filenames.