The Subtle Substitute Command

Most of you already know the :substitute command by its shortest abbreviation :s and use it in roughly this form:

to make some change within the line you are on, in the first case, or change every instance in the file in the second. If you use both forms you are already ahead of the game. Too many class instructors and textbook writers try to tell you that the way to change some phrase throughout the file is to type something like:

This is wasteful nonsense. Both forms accomplish exactly the same thing, but the second version involves extra typing for you and an extra run through the file for your computer. It does not matter that not every line in your file will contain a “Smith” to be changed — the :substitute command will execute properly in either version, and quietly if even one line in the file has a “Smith” it can change.

But neither form as it stands is sure to change every “Smith” in the file. The :substitute command is set up to change only the first example of the string it finds on any particular line, so a line in the text that originally read:

will be changed by either version of the command to read:

ex mode has a built-in solution for this problem: place a lower-case letter “g” at the very end of the command, immediately after the last “/” mark, in order to make the change on every such string in each line. So typing this:

will make that text line come out as:

Finer tuning of the instances can be done by a little trickery. Suppose you are working on tables, and want to change only the very last “k37” on each line to “q53”. This command will do it:

If this seems surprising, remember that in a search pattern with a wild card, the editor always extends the match to the greatest length it can. In this case that means the string starting at the beginning of the line and ending with the last “k37” in the line.

You probably already know that you don’t always have to type the search pattern that indicates the text to be replaced by a substitution command. If you want to reuse your very last search pattern, whether that was in a substitution command or not, you can use an empty search pattern string to stand for the last search pattern, so the two commands below are actually identical.

Either command will go to the next line containing “Murphy” and there replace the first “Murphy” with “Thatcher”.

Within a substitution command’s search pattern to find the text to be removed, all the normal search-pattern metacharacters apply. So do two more that are reserved only for substitution commands: the “\(” and “\)” metacharacters.

These two metacharacters don’t match anything themselves, so:

are exactly the same command as far as they go. But the substitution command remembers what it finds to match the text between a pair of “\(” and “\)” metacharacters, for use in the replacement text. Whenever your replacement pattern contains “\1” the editor replaces that metacharacter with whatever matched the characters that were between the first pair of “\(” and “\)” metacharacters. A “\2” in the replacement pattern is removed and replaced by whatever was matched by the characters between the second pair. And so on — you can have up to nine pairs in one substitution command. These metacharacter pairs can even be nested in the to-be-replaced text; the one that starts first will be represented by “\1” and so on. So if you extend that second substitution command above to read:

the substitution command will produce a line that begins:

Or if you type:

then your command will change the first line below to read as the second line, just beneath it:

(I used the colon “:” character to separate the parts of the command, instead of the slash “/” character, solely to make it more readable for you. There is no danger of the editor confusing “/” with “\” or “l” (el) with “1” (one) etcetera.)

As the preceding examples show, the “\(” and “\)” are not too useful with plain-text search patterns; about their only real value there is when you are searching for something that’s difficult to spell correctly, and don’t want to type it into the replacement pattern with possible spelling errors. (Spelling errors aren’t so dangerous in the to-be-replaced text, because they only cause the pattern match to fail.)

These metacharacters save the day, though, when you are dealing with other search metacharacters in searching for text that you will want to put back in. (Often the only way to specify the exact spot you want the replacement done is to include some neighboring text in the search pattern, and tell the editor that after the neighboring text has been taken out it is to be put back in right where it was.) Here are three examples of this kind of substitution:

The first of these simply inserts “No.” in the middle of phrases that are section numbers, throughout the document. But the “\(” and “\)” notation is essential to preserve the section number in each case, and also to leave unchanged the capitalization or noncapitalization of the first letter of “section”.

The second command does the same thing, but only on the very next line that has a section number to change. The surprise here is that I put the “\(” and “\)” in the address pattern to find the correct line. A line address doesn’t use these metacharacters, of course, but it does not object to them, either. It just ignores them in its own line search, but does pass them along when a following substitution command reuses the last search pattern, as happens in this example.

The third example is useful in editing a list of answers to exercises. It stops at each answer to a true-or-false question and capitalizes the entire word “answer”. The innovative aspect of this command is that it finds the letter “T” or “t” or “F” or “f” following the word “answer”, so it will not change the capitalization where an answer is numerical rather than true or false. And yet, the letter indicating whether “true” or “false” is the correct answer is not discarded as a side effect. This is primarily an example of a change that can be done more simply by using other metacharacters in the replacement pattern. Those other metacharacters are described below.

The string of characters you want to put in via a substitution command can use its own list of metacharacters. They’re entirely different from the metacharacters used in searching for a pattern you want to take out of a line.

In a replacement pattern, the “&” stands for the entire text that was to be replaced. Use this when you want to add rather than replace text. For example, to change “kit” to “kit and kaboodle” regardless of whether “kit” is capitalized, use:

If you have magic turned off (:help 'magic), you must backslash the “&” to give it metavalue. With magic on, backslash an “&” to insert it as a regular character. Turning magic off is an antiquated practise and is probably not something you’ll want to do often, if ever. Vim does offer an enhanced magic mode called very magic that attempts to offer a regular expression dialect closer to Perl’s. Use /\v/ to enable that at the start of your patterns. There is no option to permanently enable very magic by default, thankfully.

The “~” character represents the replacement pattern you used in your last substitution command. One good use for this is in correcting various misspellings of a single word:

A “\U” metacharacter makes all letters following it into capitals; to the end of the replacement pattern or until another metacharacter turns it off. Here’s how you’d use it to change a section of your document to all capitals:

A “\L” gives the opposite effect of a “\U”; all following letters become lower case. You might use this to decapitalize acronyms:

Use “\E” to end the effect of a “\U” or “\L” metacharacter. Everything following the “\E” has the same mix of capitalization as originally. For example, to enclose a line of text in curly braces, and also change just the first word to all capitals:

No “\E” is needed when you switch from “\U” to “\L” in the middle of a replacement pattern, or vice versa. When either of those metacharacters appears, it automatically ends the effect of the other. So if you have a list of book titles, one title per line, with only the first letters of the words capitalized, and you want to change those titles to all capitals before the colon in the middle of each title, and all lower case after it, just type:

This metacharacter capitalizes just the single letter that immediately follows it. If the character that immediately follows it is not an alphabet letter, “\u” does nothing.

The same as “\u”, except that “\l” makes the immediately following letter come out as lower case.

One more thing is important to know about reusing patterns in substitution commands: when all or part of a text-to-be-replaced pattern is going to be used as a replacement pattern, or vice versa, the command reuses the result of the original pattern, after all the metacharacters have been evaluated in the original situation. Since the metacharacters in either of those two types of patterns have no meaning in the other type, it could hardly be otherwise.

But when the reuse involves a text-to-be-replaced pattern being used a second time as a text-to-be-replaced pattern, or a replacement pattern being reused as a replacement pattern, the command brings along all the original metacharacters and evaluates them afresh in the new situation. Thus, in either of the cases in this paragraph, the second use is unlikely to produce exactly the same results as the first use did.

Vim has a suite of powerful lookaround metacharacters that allow you to more specifically anchor your searches when doing substitutions.

See :help /\@= for the section of the manual that explains how these metacharacters are used in Vim, along with examples of what each one is suited for — and occasionally not suited for.

As useful as lookaround is for precision searching, it’s cumbersome to specify correctly and quite often overkill when all you’re really after is a way to specify the start and end of where Vim should match inside your larger search area. Grabbing just the developer’s name in the signature line above is one example. Use \zs to tell Vim to rely on what came before it to anchor the match, but not include it in the match. Use \ze to likewise anchor what follows but not include it in the match.

Vim normally treats the replacement part of a :substitute/find/replace/ command as a string possibly containing various special characters that are expanded to alter and augment the replacement. One such special character, \= (which must appear at the very start of the replacement), treats the rest of the string as an expression, giving your substitutions the full power of Vimscript. If you haven’t learnt Vimscript yet, Damian Conway’s excellent introduction article series, Scripting the Vim editor, is a great place to start.

Inside a \= expression, you must use submatch(0), submatch(1), submatch(2), etc for the &, \1, \2, etc that you would usually use in a replacement to refer to captured groups in the match.

Given the following signature in a file:

We’d use the following substitution to update the timestamp:

Now you’re ready for the Bonus Question in Spot Quiz 4.3

Despite the name, the :substitute command doesn’t always take something out of the line and put something else in its place. Here’s an example that adds text at the start of certain lines without removing anything:

so that text which originally looked like this:

now looks like this:

It’s just as practical to pull some text out of lines without putting anything back in its place. Here are two command lines that do just that:

The latter command removes superfluous spaces at the ends of lines. It doesn’t need the final two slashes because there is no suffix to be distinguished from a replacement pattern.

At times you might use both the previous principles, to create :substitute commands that neither subtract nor add any text. Sound pointless? Here’s one such that I sometimes use when I’m well along in writing one of these tutorials:

Already you know enough about Vim to be able to plan some fairly complex edits. Here’s a short introduction to the art of crafting truly complex substitutions and knowing when to abandon them for multiple simpler solutions that when used together solve the problem more clearly.

That’s usually the best way to build a complex editor command or command script. That’s a programmer’s term that means putting all the little details in separately and then pulling them all together into a unified whole, rather than starting with a grand overall plan and forcing the details to fit.

For example, consider how you would use Vim to automatically add HTML paragraph tags to each paragraph of a manuscript. This requires inserting the string “<p>” at the start of the first line of each paragraph, and the string “</p>” at the end of the last line. This manuscript separates one paragraph from another with a completely empty line (not even a non-printing character on it).

This looks pretty easy. All that seems to be needed is to go to each empty line, then move up to the preceding line to insert the end-of-paragraph string and down to the following line to put in the start-of-paragraph string. But there are flaws in the obvious command to do this:

The first problem is that when the editor goes to the empty first line that commonly begins a file, it will be unable to move up a line to do the first substitution. No substitution is needed there, of course, but since the editor doesn’t leave that empty first line, moving down two lines will put it on the second line of the following paragraph — definitely the wrong place for a start-of-paragraph tag. There are several ways to fix this problem:

Have the editor :mark the empty line before leaving it to execute (or attempt to execute) the first substitution. Then it can go to the marked line (which works even if the editor never left it) and drop down one line to perform the second substitution.

Change the address of that second substitution command from “++” to “/./” in order to move forward to the next nonempty line, which will be the first line of the following paragraph whether the search starts from the empty line or the line above it.

Run two separate :global searches, each of which executes one of the two substitution commands.

Problem number two is that there may be several empty lines between two paragraphs, since HTML interpretation is not affected by them. If the editor is on the first of two or more consecutive empty lines, the command I first proposed above will perform its second substitution on the second empty line just below it. When it moves to the second previously-empty line, it will run the first substitution command on the empty line it just left. (Yes, the second line is no longer empty, but it has already been marked by the :global command before any substitutions are done.) That is, a stretch of text that initially looked like this:

and should have been edited to look like this:

actually turns out like this:

It may look as though this hazard can be defeated by modifying the number two solution to the first problem above. That is, the address for both substitutions will be a search pattern that looks for a line that already has some text on it. This works properly when the editor is on the first of two consecutive empty lines. From the second line, though, it runs its substitution commands on lines that have already been given their tags, so the sample text now looks like this:

What’s really needed here is double-conditional execution. That is, substitution commands must run on a given line only if both of these conditions are true:

In this case, Vim can handle it. The :global portion of the command line takes care of the first condition if the substitution commands' addresses move exactly one line in each direction from the empty line. (Of the three proposed solutions to the first difficulty encountered, numbers one and three both do this much.) To satisfy the second condition, make the substitution commands remove one character from the existing line — and then replace it, of course. This ensures that if there is no character to remove because the line is empty, the substitution command will fail on that line and do nothing.

Either the first or third solution can be adapted to satisfy that second condition. I’ve used the third solution in the example commands below, because the technique is easier to follow than it would be with the first solution:

What we’re calling bottom-up techniques here is the practice of explicitly breaking an otherwise complicated edit down into multiple simpler edits. This is generally a good technique to keep the thought-space of a problem and its solution simple even if a single, albeit complicated, solution exists — revisiting or remembering that solution later might be more taxing than the multiple simpler forms combined. Having said that, Vim does have the capacity to solve the given problem with a single expression and it doesn’t look too harrowing either:

This substitution uses the ever useful \zs to anchor our match at the end of a line containing text that is followed by empty lines. Given the relative simplicity of this single substitution command, you may wonder why attention was given to the need for separating the solution out into seemingly more complicated parts. The simple answer is that this multi-part solution was originally devised on vi and not Vim. Vim has brought a lot of new functionality, features, power and expressiveness to the world of vi. I left this example in here for the very good reason that this is still solid advice in the general case. To see this in practice even for this seemingly defeated example, consider the Bonus Question in the Spot Quiz below.

Bottom-up techniques can be continued if there are yet other special needs to be accommodated. The manuscript may have headlines and subheads mixed in with the paragraphs, and may already have appropriate HTML tags at the beginnings and ends of those heads and subheads.

At times a command needs to be supercharged by way of a far out use of substitution — something perfectly legitimate, but never intended by the people who wrote this editor. Here are a few that you may find useful.

Let’s suppose that you have to alter a long document so that all references to “Acme Distributors” are changed to “Barrett and Sons”. A simple substitution command will make most of these changes, but it will miss those instances where “Acme” appears at the end of one line and the next line starts with “Distributors”. A follow-up pair of substitutions, to replace “Acme” wherever it appears at the end of a line and to replace “Distributors” when it starts a line, would wreak havoc — this document also refers to “Acme Supply Co.” and to three other companies whose names end with “Distributors”.

We can handle this problem nicely with the following two command strings:

The first command goes to every line that ends with “Acme” and then moves forward one line—if and only if that next line begins with “Distributors”, it is changed to begin with “and Sons”. The next command reverses the process to change “Acme” to “Barrett”, but only in the right instances. (Note well that the second command searches for “and Sons”, not “Distributors”, because the first command has changed those line-split “Acme Distributors” to “Acme and Sons”.)

We could even just use two simpler substitutions using \n in the second to catch the wrapped case:

Or we can use Vim’s \_s which captures whitespace across newlines:

Often it is a good strategy to start with a change you definitely don’t want in order to wind up with what you do want. Suppose you are a technical writer who has just finished writing a number of lengthy photo captions full of phrases like “the light spot in the upper righthand corner” and “dark areas near the lower lefthand edge”. Along comes the news that the Art Director has decided to flop all the photos: print them in mirror-image form. Suddenly, everything that was on the right is now on the left, and vice versa.

Your captions will be accurate again if you change every “lefthand” to read “righthand” and vice versa. But how to do that without wading through the whole text and making each change individually? The obvious pair of substitutions will not work:

The second command doesn’t just change the original instances of “righthand” to “lefthand”; it also reverses every change your first command made — now everything is described as being on the lefthand side. But the following three substitution commands will do the job nicely.

By making the first command change “lefthand” temporarily to “QQQQ” (or any other string you know will not be found in your document), you keep those changes safe from the effect of your second command. Then, after that second command has finished, the third command changes those Q strings to what you had wanted in the first place.

Can you do it in one go with Vim? Sure — using a substitute expression.

It can even make sense to type in things incorrectly, then change them to what you want via substitution. When I’m writing documents in plain ASCII, to be printed without any formatting, I often use a line across the page to separate major sections of the document. But where others are satisfied with just a string of hyphens, or another single character, I pretty things up with multi-character dividers like:

Not that I have the patience and concentration to type in page-wide lines of alternating characters, especially when I would have to constantly get on and off the shift key, too. No, I just use my repeat key to fill the line with whatever character will begin my eventual multi-character pattern. For those four patterns above, I would have repeat-keyed in these four lines, respectively:

Then I only have to run a simple repeated substitution to get the line I actually want. Here are the commands I would run on the four lines above, respectively:

At times you’ll have to make changes that are so dependent on human judgement that no substitution tricks could possibly do exactly what’s wanted. In those cases there are two ways to have the editor partially automate those changes.

The first is to use a variant form of the substitute command that presents each potential substitution on your screen, and asks you to accept or reject it. All you need to do is put a lower-case “c” at the end of your substitution command, in the same place you would put a “g” to change every instance on each line, like either of these examples:

Vim will then highlight the matches where substitutions are to be made on your screen, one at a time. After highlighting the match on your screen, Vim will prompt you in the command-line for your action, like:

and wait for you to type your response before moving on to the next match. Read :help :s_c to learn about the meaning of the various responses there.

Even this substitution variant may not give you enough control. You may need to see more than one line to verify your judgement, or the text to be put in may vary from one place to another. In those cases, you can use one of the qualities of the :global command. This is a simplified version of the technique our programmer friend Hal (in the first installment of this tutorial) used to work over the problem points that Lint found in the code he was updating.

If you are editing in screen mode, as usual, you must start by typing a capital “Q” to go into ex mode. Vim also provides a slightly friendlier version of ex mode with the “gQ” command. From ex mode’s colon prompt, give a command like the following (if you want to make the same substitution as in our previous examples):

This command will bring you in turn to each line in the file that contains the string “something” and put you in screen-editing mode there. After you’ve looked around, and made a substitution if you think it justified, typing a capital “Q” takes you out of screen-editing mode and returns you to the global command, which takes you to the next marked line and resumes screen editing.

There’s one problem with this usage pattern of the ex mode global+visual command: suppose we found, after the 10th XXX of 100, that we do not want to view the remaining 90 occurrences. It works as long as we don’t need to type “Q” again — we can stay in full-screen mode until the editing session is finished. But if we want to use the ex mode global+visual command again, we’re taken to the 11th occurrence of XXX and prevented access to the ex mode : prompt. How do we cancel the currently running ex mode global+visual command?

The best way out of this predicament starts with writing your changes to the file. Then, instead of typing “Q” type an :edit command. This brings up a fresh copy of the file you are editing, but since you’ve just done a write, the fresh copy is identical to the one you’ve been working on. Because you haven’t left the editor, most of the state is saved — contents of the named buffers, any maps and/or abbreviations, values of the set command options. You do wipe out a few small items like the contents of the unnamed buffer — and, of course, that suspended global command.

Now you can use the “Q” command to go back into ex mode to run that second global command.

In this tutorial to date, you’ve undoubtedly seen some aspects of the editor that you wish had been designed differently. The good news is that many of these features are yours to change at will—without hacking up the source code and recompiling.

In Part 5 of this tutorial, I’ll elucidate the editor’s built-in facilities for setting up your own editing environment, and the many factors you can modify this way.