Call native operator

[SteveL-MSFT]

Problem Statement

Currently, there are cases where cutting and pasting a native command line fails to run as expected in PowerShell. This may be due to incorrect parsing of quotes meant to be passed to the native command or use of PowerShell syntax that is not meant to be interpreted as PowerShell. PowerShell has a --% special argument when used with native commands treats the rest of the arguments as literals passed to the native command, but has several issues:

1. It is not discoverable, users need to know about this special parameter ahead of time
2. |, &&, and || take precedence, so: wsl --% ls | less would execute wsl ls and pipe the results to less running in PowerShell rather than less running in wsl
3. If you cut and paste a command line, you would need to edit the line to insert --% towards the beginning
4. On Unix systems, the args after --% are passed verbatim w/o globbing where native commands on Unix expect the shell to perform globbing

Proposed technical implementation details

Proposal is to introduce a new --% (Call Native) operator.

Any text content after this operator will call into the "default shell" of the OS to execute. On Windows, this would be cmd.exe and on Unix-based systems this would be /bin/sh. This resolves the globbing issue on Unix-based systems, and also allow %variable% expansion on Windows. Unlike --% switch, this also means that |, &&, and || are treated as part of the native command line.

This means that these two are functionally the same:

wsl --% ls $foo `&`& echo $PWD

--% wsl ls $foo && echo $PWD

where $foo and $PWD is evaluated by the shell within WSL. Note that in the first example, you would have to know to escape && to have it execute within WSL instead of within PowerShell.

To pipe output from such execution back into PowerShell, the user is required to store the results into a variable first:

$out = --% ls *.txt
$out | select-string hello

Note that unlike the current & call operator, you cannot use any PowerShell syntax, so:

--% $commandline

would not resolve $commandline as a variable first by PowerShell, but instead pass $commandline to the default shell to process unresolved.

The cut & paste problem is solved by simply pasting after --% is typed.

The above example for wsl would look like:

--% wsl ls | less

where the intent is to have that whole line execute within the WSL Linux instance.

Discoverability

Users already familiar with --% as a switch may easily transition to using this new operator where it makes sense. For new users, --% is unique so that search engines find it easily related to PowerShell.

Alternate Considerations

&! and &n were proposed as the sigil, but there was push back because &! is a valid operator in some languages making a web search for documentation more difficult. There was also a concern whether visual similar to & call operator would be confusing to users.

There is question about supporting line-continuation when using this new operator. I would suggest that we do not support it initially.

A cmdlet solution instead of an operator solution was proposed. We believe this doesn't solve the "cut & paste" problem as now you need to know to put the pipeline in single quotes and/or escape special characters. We do believe a cmdlet as in Invoke-NativeCommand (noun to be determined) would be useful as an additional option instead of replacing the need for an operator.

Related issues

This should also solve these issues:

#12491
#1761

[essentialexch]

I like the idea and find it useful, but think it would be far more useful (I'm thinking specifically in the case of a DSL) if I can get a single string expansion. So if I've built a cmd.exe compatible string in my PS, I have a way to string verbatim to cmd.exe. Perhaps

&n -command $string

That would potentially also let me specify my interpreter:

&n -shell /bin/sh -command $string
or
&n -shell cmd.exe -command $string

in terms of the name/operator, is "&!" in use? That has similarity to shbang( "#!" ).

[SteveL-MSFT]

$! is a good suggestion!

The shell can be specified by simply specifying it as the command:

&! /bin/sh -c blah

The $var expansion creates a problem in that $ may need to be literal to be passed to the native command/shell. One of the problems this tries to avoid is when people need to figure out how to properly escape everything. If you require variable expansion, you could always do:

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

However, in cases where you want to mix PowerShell with the native command, it's probably best to use the current syntax and just be aware of what needs to be escaped properly as I would consider that advanced usage.

[essentialexch]

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

I can't believe you just suggested iex. :-)

[SteveL-MSFT]

@essentialexch there are a few times where it's appropriate :)

Note that in the example, expectation is that the user properly validated the contents of $mycommand before executing!

[bergmeister]

Great writeup and a solution for problems like the one you described are definitely needed.
I myself didn't know about --% and don't remember ever seeing it unfortunately. Therefore the proposed &n operator might suffer from similar discoverability problems and it doesn't look natural to me to combine a symbol with a character (and it reminds me of %f in C printf statements). Since it's a breaking change anyway Is there a reason why it couldn't be for example &&?
Maybe it would me more intuitive to use something like brackets (or double braces?) to mark the area that you want to execute. Example: & [ wsl ls ] | less

[JustinGrote]

I agree &! is the more intuitive way to go for those coming from other shells, however in super-duper-linter I am usually running native commands by building a parameter array and then splatting it to the command.

$command = 'linter'
$myargs = @(
    '-config'
    'linterconfig.path'
)
if ($Test) {$myargs += 'test'}
& $command @myargs

So this is the best way in case of conditional and whatnot for me

[SteveL-MSFT]

@bergmeister I originally thought of && as visually it's similar to & which many people know today. However, && being a pipeline chain operator may cause confusion about what it is supposed to do. I like the &! suggestion currently.

[SteveL-MSFT]

@JustinGrote, there's no reason to not continue to use that. the &! syntax is really for cases where you just want to cut and paste or have some args that conflict with PowerShell syntax and don't want to or don't know how to escape properly

[oising]

Oh man, I remember having discussions with Bruce and Jason about this a decade ago. My gut feeling is that inventing another operator seems unnecessary here. I know that some people on this thread have not heard about --% but I'll wager than there are more than you think. What's wrong with:

# run ls in wsl, return results to powershell, then pipe to wsl again, to grep.
& wsl ls | wsl grep -i "foo"  

#  the entire pipeline right of wsl will run in wsl. 
& --% wsl ls | grep -i "foo"

Why hasn't anyone suggested this? It's logically consistent with usage of --% elsewhere to say "everything after this is to be passed without special powershell treatment."

[JustinGrote]

@oising well for one, that command doesn't work because you have to preface it with the command you want to run, are you suggesting adding the functionality?

This kinda does what is expected:
function Invoke-LiteralCommand ($command) {Invoke-Expression "& $command --% $args"}

Invoke-LiteralCommand ping -W 200 www.google.com | grep icmp

[oising]

@JustinGrote Yes, I know it doesn't work now :) I am suggesting that instead of &n or &! or whatever else is being talked about. It just makes more sense to me: & is call and --% is suppress powershell parsing; together they are coherent and more discoverable than adding something new. I don't like the idea of having a "call" and a "call native" operator.

I've expanded my example to show the differences.