Bash pipe tutorial

There is always time for pipes. The white rabbit can wait.

Pipes (or pipelines) are one of those things that you learn to use intuitively through the idiomatic use cases that we know and love but never come around to fully understand. Lucky enough, today is a good day to dive into the depth of pipes, don’t you think?

Heads up, in writing this article, I got better at pipes. Hopefully, you do too.

What are pipes?

A pipe is an enclosed medium that allows flow from one end to another. In the real-world pipes are used to convey matter, mostly liquid such as water or gas such as smoke but sometimes convey a mixture of liquid and solids. In a Linux environment, a pipe is a special file that connects the output of one process to the input of another process. In bash, a pipe is the | character with or without the & character. With the power of both characters combined we have the control operators for pipelines, | and |&.

As you could imagine, stringing commands together in bash using file I/O is no pipe dream. It is quite easy if you know your pipes.

So, before you start killing it with pipes in bash, see how pipelines can help you get more done shell script with less code. Read on.

Pipelines

According to the bash manual section on pipelines (3.2.2 Pipelines), A pipeline is a sequence of one or more commands separated by one of the control operators ‘|’ or ‘|&’. That means every command is a pipeline whether or not you use its pipeline control operators.

When we strip away all of the options in the format for a pipeline:

We get:

What do you know? We’ve been using pipelines in bash all this time without knowing. Well, now you know. Anyways, let’s see how we can start to use pipelines for real with time –p ! and | or &|.

Facts about pipes

• Pipeline time
  A pipeline may begin with time, which reports runtime statistics after completion of the pipeline
• Pipeline portable time
  time accepts the option -p for improved portability of runtime statistics, replacing tab with single space and converting time to seconds with no unit, the output format specified by POSIX
• Pipeline operators and implicit redirection
  By default, only standard output of commands on the left-hand side of the operator | is connect to commands on the other side. To have standard error connected as well the &| operator may be used. However, it is simply shorthand for 2>&1|, which redirects standard error to standard error before the pipeline operator.
• List precedence in pipelines
  If the command on the left-hand side of the pipeline operator is a list ({ command1; command2; …} or (command1;command2;…)), the pipeline waits for the list to complete
• Pipeline behavior under lastpipe
  Commands in a pipeline are executed in subshells unless the lastpipe shopt is enabled. If lastpipe is enabled, the command on the far-right side is executed as a command belonging to the current shell. See Test lastpipe in Tests.
• Custom time format
  time output may be customized using the bash variable TIMEFORMAT. See Test time format in Tests.
• Pipeline behavior under pipefail
  By default, all commands in the pipeline are executed without regard of the exit status of commands on to the left and the exit status of the right-most command is return. However, if pipefail is enabled, the pipeline will terminate abruptly if any of its commands returns a non-zero exit status. Also, the pipeline exit status will be that of the last command exited with a non-zero exit status.

How to use pipes by example

As mentioned in What are pipes, bash has two control operators for pipelines, namely | and |&. That is the groundwork. Let’ go into how to use pipes.

Using | pipes

This is the standard pipeline that most bash programmers have touched sometime or another. It only passes standard output right, down the pipeline.

Source: test-pipeline-standard.sh

Commands

Output

Using |& pipes

This is the non-standard pipeline that most bash programmers seldom touch. It implicitly redirects standard error to standard output and proceeds as in the standard pipeline.#!/bin/bash
## test-pipeline-time2
## version 0.0.1 – initial
##################################################
func() { read -t ${t} input
time -p {
echo ${input-1} 1>&2
sleep 1
echo $(( ${input-1} + 1 ))
}
}
test-pipeline-time2() {
t=0 ; time echo 1 | func | func | func
t=1 ; time echo 1 | func | func | func
t=2 ; time echo 1 | func | func | func
t=3 ; time echo 1 | func | func | func
t=4 ; time echo 1 | func | func | func
}
##################################################
if [ ${#} -eq 0 ]
then
true
else
exit 1 # wrong args
fi
##################################################
test-pipeline-time2
##################################################
## generated by create-stub2.sh v0.1.2
## on Tue, 23 Jul 2019 22:13:53 +0900
## see <https://github.com/temptemp3/sh2>

Source: test-pipeline-nonstandard.sh

Commands

Output

Using pipes with time

Timing pipelines can be tricky at times especially when commands on the right-hand side don’t depend on input from the left-hand side. In this case, commands are executed in parallel. In the following example pipeline timing is affected timing parameters.

Source: test-pipeline-time2.sh

Output:

Using pipes with !

Pipelines can be leveraged to implement certain control logic if an expected behavior is known. Such is the case pipelines with commands that fail and pipefail set on. In the following example we show how to exit a loop if all commands succeed.

Source: test-pipelines-mixed.sh

Output:

Using mixed pipes

In practice, pipelines are often mixed up.  In the following example, we mix it up handling non-standard pipeline errors, producing a nice banner, and finish up with a list of all the errors that came up.

Output

Tests

It is good practice to write tests to ensure that your code is going to behave the way it was intended. Here we have a list of tests which you are welcome to run yourself.

• Test lastpipe – compare pipelines with and without lastpipe enabled
• Test negation – negate the exit status of pipelines
• Test time – time pipeline
• Test time format – customize pipeline runtime statistics
• Test pipefail – run pipelines with pipefail enabled

Test lastpipe

Here is a simple test showing how enabling lastpipe affects the expected behavior of pipelines in bash. That is, you may opt to allow the last command in the pipeline to be executed in the current shell using lastpipe.

Source: test-pipelines-lastpipe.sh

Output

Note that in the case that lastpipe is enabled, changes made in the last command of the pipeline may persist. That is if we update a variable, its value will accessible in the current shell outside of the pipeline.

Test negation

Here is yet another test showing how negation works on pipelines in bash. Note that every time func is called we append a ‘1’ to the variable x. The return status always 1. However, we can change it to 0 using negation.

Source: test-pipeline-negation.sh

Output:

Test time

Here we want to show how to time a pipeline. In the example below, we time a function that takes 1-2 seconds to complete and negate its exit status the second time calling it.

Source: test-pipeline-time.sh

Output:

Test time format

Here we show how to customize pipeline time output. In the example below, in addition to showing the default and portable behavior, we create a custom TIMEFORMAT, which removes precision and ads CPU usage.

Source: test-time-format.sh

Output:

Test pipefail

Here we show how lastpipe affects the exit status returned by a pipeline. In the example below, the exit status of a pipe is 0 if none of the commands returns a non-zero exit status. Otherwise, all pipelines return a non-zero exit status between 1 and 5.

Source: test-pipefail.sh

Output