Please make verbose XFAIL tracebacks optional

[mgorny]

What's the problem this feature will solve?

Since #11735, -rx causes pytest to output tracebacks for expected failures. This is sometimes undesirable with projects that have a lot of xfail-ing tests. For example, with pandas outputting XFAIL tracebacks takes a few minutes and results in 80M of output.

Describe the solution you'd like

I think the cleanest solution would be to have another option to control whether tracebacks are displayed for XFAILs. This would make it possible to retain XFAIL in summary output (via -rx or -ra), without the extra overhead of printing all the tracebacks.

Alternative Solutions

We could avoid passing -rx, but then we'd lose the list of XFAILs from test summary, and that one is actually desirable.

Additional context

n/a

[nicoddemus]

@mgorny I agree, for projects with a lot of xfail tests it produces too much output.

Do you have a suggestion of what the option would be?

[mgorny]

I'm terrible at naming, so I'd go for --xfail-verbose=.

[nicoddemus]

Oh you mean a separate flag? I would prefer not.

Perhaps we can change the feature itself to require a separate -r letter to show the xfail tracebacks. It would be a "breaking change" from #11735/#11574, but perhaps it would be for the best.

cc @okken @sturmf

[mgorny]

Yeah, sure, that would work for me too.

[okken]

I’d be ok with a separate -r letter.

Also, if a separate option, --xfail-tb with no/short/long, etc, like the existing --tb flag, wouldn’t be terrible. But it might be harder to implement.

[kpfleming]

Yes, please! I just spent 10 minutes trying to track down why my CI job was reporting a failure, and assumed it was because of the XFAIL tracebacks (but it was not, coverage dropped below the required threshold).

[nicoddemus]

I like --xfail-tb. If it is simple to implement no/short/long etc that is good, if not, the first implementation can just enable tracebacks when --xfail-tb is passed, then later we can improve on that to allow the no/short/long option.

Would love to see a PR in this direction -- should be a simple extension of the work done in #11574.

[okken]

I can take a look, probably next week.

[okken]

Initial implementation looking pretty good so far.
I've added --xfail=tb flag, which defaults to auto.

So turning off xfail tracebacks could be done with --xfail-tb=no.
I'll probably regularly run with --xfail-tb=short.

@nicoddemus Should we default --xfail-tb to no?
It seems more natural to me to leave it defaulting to auto, like --tb, but I'm really ok with defaulting off.
That would be a behavior break, though.

[bluetech]

It seems more natural to me to leave it defaulting to auto, like --tb, but I'm really ok with defaulting off.
That would be a behavior break, though.

My opinion is to default to off, because xfails are expected so their traceback is not normally interesting.

[nicoddemus]

I agree with defaulting to no. Even if it is a behavior change, I think it is for the best, I consider it was not a good decision to leave it on by default, after seeing it working in the wild. I think in this case it makes sense because we will be reverting to the previous and expected behavior.

[okken]

Thanks everyone for your rapid feedback on this.

[okken]

Ok. This is more difficult than I imagined at first.
tbstyle and functions that use it are kinda all over the code base.

Perhaps a change of plans.

How about, we leave the styling of tracebacks to --tb and just have --xfail-tb be a boolean, and it just controls if they on or off?

I've modified the changelog entry, tests, and code to treat --xfail-tb as a boolean flag. It's cleaner and less invasive to the code base than trying to have it behave like --tb with independent styles.

[nicoddemus]

How about, we leave the styling of tracebacks to --tb and just have --xfail-tb be a boolean, and it just controls if they on or off?

I agree, that was what I meant by this:

if not, the first implementation can just enable tracebacks when --xfail-tb is passed, then later we can improve on that to allow the no/short/long option.

I see you closed the PR, was that intentional?