Macro-generated macro hygiene change

[dtolnay]

The following procedural macro works with nightly-2017-12-02 but fails with nightly-2017-12-03. I don't have an intuition for which is the correct behavior but just wanted to make sure this is intentional.

bb42071...f9b0897. #46343 seems relevant and is in the right range so cc @jseyfried and @nrc.

This is relevant to dtolnay/proc-macro-hack#15.

#!/bin/bash

set -x

cargo new mac
echo >>mac/Cargo.toml '
[lib]
name = "mac"
proc-macro = true
'
echo >mac/src/lib.rs '
extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro_derive(Hygiene, attributes(hygiene))]
pub fn hygiene(_: TokenStream) -> TokenStream {
    let macro_rules = "
        macro_rules! hygiene {
            () => { x }
        }
    ";
    macro_rules.parse().unwrap()
}
'

cargo new repro --bin
echo >>repro/Cargo.toml 'mac = { path = "../mac" }'
echo >repro/src/main.rs '
#[macro_use] extern crate mac;
fn main() {
    let x = 0usize;

    #[derive(Hygiene)]
    #[hygiene(print = "x")]
    struct S;

    println!("{:?}", hygiene!());
}
'

cargo run --manifest-path repro/Cargo.toml

[nikomatsakis]

Seems likely that the behavior was intentional, given that the PR is entitled "Fix hygiene bug". But I'd like to know more definitively.

#46489 <-- possible dup?

[nrc]

I would not expect this to work. Creating the macro_rules macro should be fine because derive is unhygienic. However, when hygiene (the macro_rules macro) expands it creates an x which has the hygiene context of that macro's definition, so unrelated to the declared x in main.

[nrc]

I was wrong, I think. Since the macro_rules macro is inside main it should be able to see x. The attribute seems to have nothing to do with this example though.

[nikomatsakis]

@nrc and I were chatting. Let me explain what is happening here for future reference. @dtolnay you can confirm the details. @dtolnay has some kind of horrible hack-y crate wherein it lets you fake a "procedural macro" by translating the definition to some sort of #[derive] that, when executed, winds up generating a struct and macro-rules pair side by side, sort of like this:

fn foo() { 
    let x = 10; 
    struct Bar; 
    macro_rules! foo { () => { x } }; 
    let y = foo!(); 
}

fn main() { }

That example works, but something else must be happening around derive to make this example not work. I'd like to better understand what that something is!

(I'd also like to know @dtolnay how many crates are affected by this regression, directly/indirectly.)

[dtolnay]

You are correct about how proc-macro-hack works (if you strike the word "horrible").

I looked through the crates on crates.io that would be broken by this. I believe they are:

• bstring, which exposes a bformat! macro similar to format!. @murarth
• array-macro. @xfix
• reql. @rushmorem

I think all the other crates that depend on proc-macro-hack are only passing in literals, as in ip!("127.0.0.1").

[jseyfried]

@nikomatsakis

That example works, but something else must be happening around derive to make this example not work. I'd like to better understand what that something is!

The reason why this doesn't work is because macros 1.1 isn't totally unhygienic -- it's just mostly unhygienic the same way macro_rules! is mostly unhygienic.

This example fails for the same reason that the following fails (and has failed for years):

macro_rules! hygiene_outer {
    () => {
        macro_rules! hygiene {
            () => { x }
        }
    }
}

fn main() {
    let x = 0usize;
    hygiene_outer!();
    println!("{:?}", hygiene!());
}

[jseyfried]

That being said, I could make macros 1.1 fully unhygienic to avoid breakage here.

The main reason they have macro_rules!-hygiene is because we want the expanded tokens to show up in diagnostics as from the macro, and using macro_rules! hygiene was the simplest way to implement this.

Once it is possible to have a span that shows up as from a macro yet is fully unhygienic, it would be easy to make macros 1.1 expansions fully unhygienic. Since macros 1.1 can only be used on items, I believe this can only make a difference when expanding into a macro_rules!.

[KamilaBorowska]

@dtolnay array-macro no longer uses proc-macro-hack. An older version did, but the dependency did get removed, and array-macro-internal crate exists only for versions 0.1.0 and 0.1.1, it's no longer used. Not even version 0.1.2 uses it.

Thanks for notifying me about this issue however 👍.

[dtolnay]

I find @jseyfried's example compelling in #46478 (comment). I don't necessarily think we should feel compelled to respect backward compatibility here if the change that broke this is a more correct and consistent design than the old one. Feel free to close this issue.

I disagree. I am new here but shouldn't unless this is a widely known and used feature of rust support it but warn the user. Can't we allow this to be the default but if the user wants the better version force it into there build scripts for instance? @jseyfried you state to want to make it work both ways is it possible to warn users now about it and when it does not break anymore not do so?
Breaking something just because it's better could kill legacy code so I would warn users about it at least. If @jseyfried or others want my help with it that's fine, I will new a little help in terms of answering my questions if any.