[sf-lug] vi/ex: %, #, !, and shell file globbing in ex command arguments / Re: entering dates in vim text editor

[Michael Paoli]

> From: alexkleider <alexkleider at protonmail.com>
> Subject: Re: [sf-lug] entering dates in vim text editor
> Date: Wed, 18 Nov 2020 00:51:48 +0000

>> :r! date +'\%c'
>> Mon 16 Nov 2020 03:18:13 PM PST
>>
>> Wow, pain in the tochis, huh? Some days, I really hate having to always
>> worry about weird little shell quoting and escaping problems.

> Thanks, Rick.  That does the trick...
> although I don't understand why the % must be escaped when it appears
> within single quotes.

Because it's ex/vi/vim interpreting the %, per longstanding ex/vi
behavior and POSIX, etc. - NOT THE SHELL that's interpreting the %.
And ex/vi uses \ for escape, it uses neither " nor ' for escape.

Again:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ex.html
"Non- <backslash>-escaped '%' characters appearing in file arguments
to any ex command shall be replaced by the current pathname;
unescaped '#' characters shall be replaced by the alternate pathname."

Let's give yet another bit of example:
$ cd "$(mktemp -d)"
$ echo a > a && echo b > b
$ TZ=GMT0 SHELL=/bin/sh strace -fv -e trace=execve -s2048 -o .strace.out ex *
a: 2 files to edit: unmodified: line 1
:n
b: unmodified: line 1
:e#
a: unmodified: line 1
:!echo first shell command
first shell command
!
:!date +'\%c \%\% bare percent:% hash:# pling:! escaped: \%\% \# \!'
!d
Wed Nov 18 08:09:06 2020 % bare percent:a hash:b pling:echo first  
shell command escaped: % # !
!
:!mv a A && mv b B && ls *
A  B
!
:arg
a [b]
:n! *
A: 2 files to edit: unmodified: line 1
:n
B: unmodified: line 1
:e#
A: unmodified: line 1
:!echo % #
!e
A B
!
:q!
$

So, let's look at what we did in ex(/vi), and see how that compares to
what was (not) given to the shell to execute, so first what the shell
was given to execute, and most of our additional executions, from our
captured strace(1) data:

$ sed -ne 's/^[0-9][0-9]* *//;/^execve("/!d;s/\], .*$/], ...)/p'  
.strace.out | egrep '^execve\("(/usr)?/bin/(sh|date|mv|ls)"'
execve("/bin/sh", ["sh", "-c", "echo first shell command"], ...)
execve("/bin/sh", ["sh", "-c", "date +'%c %% bare percent:a hash:b  
pling:echo first shell command escaped: %% # !'"], ...)
execve("/usr/bin/date", ["date", "+%c %% bare percent:a hash:b  
pling:echo first shell command escaped: %% # !"], ...)
execve("/bin/sh", ["sh", "-c", "mv a A && mv b B && ls *"], ...)
execve("/usr/bin/mv", ["mv", "a", "A"], ...)
execve("/usr/bin/mv", ["mv", "b", "B"], ...)
execve("/usr/bin/ls", ["ls", "A", "B"], ...)
execve("/bin/sh", ["sh", "-c", "echo A B"], ...)
$

The bit of sed(1) and egrep(1) there is to trim out the uninteresting -
notably just to shell executions and other executions of (potential)
interest and also trimming 3rd argument to execve(2) to just
show ... - otherwise it would be the entire environment listing.

So, we did:
:!echo first shell command
and we have:
execve("/bin/sh", ["sh", "-c", "echo first shell command"], ...)
No surprises there, that's typically what system(3) would do on
*nix, generally invoking sh with arg0 of sh, arg1 of -c, and
arg2 being the shell command, and we can see that in the
execve arguments - first it shows us the binary executed: /bin/sh
then the arguments, starting with arg0, shown as an array:
["sh", "-c", "echo first shell command"]
We can see it's mostly arg2 we're interested in - that's the actual
command (argument option to shell's -c option) where the
command of interest is passed,
so in our ex action:
:!echo first shell command
the arg2 part that's passed to shell is, as a single argument/string:
echo first shell command
Again, no surprises, but hopefully that well explains how typical
command is passed from application (e.g. ex/vi/...) to shell.

So, in our editor, we do:
:!date +'\%c \%\% bare percent:% hash:# pling:! escaped: \%\% \# \!'
and we then have:
execve("/bin/sh", ["sh", "-c", "date +'%c %% bare percent:a hash:b  
pling:echo first shell command escaped: %% # !'"], ...)
Note shell's arg2 is:
date +'%c %% bare percent:a hash:b pling:echo first shell command  
escaped: %% # !'
Note how \% from editor got changed to % and % to a, etc.
That's because it's the editor that's handling %, #, and !, before
being passed to shell, and if those don't have \ right before them,
editor interprets and parses and replaces them - if there's \ right
before them, editor strips that \ off and passes the character through
without change to the shell.  That's it - no more, no less.
So, e.g. shell quoting does absolutely noting to prevent the editor from
doing the substitution for % # and ! ... unless of course one happened
to put \ right in front of them, in which case the editor turns that
pair into just that single character % # or ! respectively.
So, e.g., if one does, from the editor in that case,
"%" or '%' - editor doesn't care, % not preceded by \ so editor does
its substitution on % before passing it to shell.  Only if editor
sees \ right before will it then remove that \ and pass the
% (or # or !) to the shell without the editor having done its
substitution on it.

Note also that in date format (+) string, to get a literal % of output,
we need to use %% for date, otherwise date will interpret the % in that
context.
Shell doesn't care about %, but editor does
(and date format strings does), so to get the editor to pass %% to shell
(which passes it to date), we need \%\% in the editor.

In editor, we do:
:!mv a A && mv b B && ls *
That gives us:
execve("/bin/sh", ["sh", "-c", "mv a A && mv b B && ls *"], ...)
So shell's arg2 is string:
mv a A && mv b B && ls *
Note in this case the editor didn't do any substitution for *,
as in this case it's not a file glob for the editor, but just part of
some shell command, so it passes it along, and shell interprets the *
before execing ls:
execve("/usr/bin/ls", ["ls", "A", "B"], ...)

In editor, we do:
:n! *
Note that that is not a shell command!  The ! is a force option to ex
command n[ext].
So, be keenly aware of the difference between '! ' and ' !' in vi/ex.
So, no shell command executed here, the editor does the
globbing/expansion of *

In editor, we do:
:!echo % #
And we get:
execve("/bin/sh", ["sh", "-c", "echo A B"], ...)
So, note again, shell arg2 got string:
echo A B
As editor had already done the substitutions on % and #.

references/excerpts:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ex.html
"Non- <backslash>-escaped '%' characters appearing in file arguments
to any ex command shall be replaced by the current pathname;
unescaped '#' characters shall be replaced by the alternate pathname."

http://www.mpaoli.net/~michael/unix/vi/vi.odp
"Slide" 20, 2nd bullet point:
o in most file and shell commands, % will
    substitute current file name, and # will
    substitute "alternate" file name, and ex will do
    shell metasyntax expansion of file names given

http://linuxmafia.com/pipermail/sf-lug/2020q4/015078.html
http://linuxmafia.com/pipermail/sf-lug/2020q4/015081.html
http://linuxmafia.com/pipermail/sf-lug/2020q4/015083.html