Linux Lexicon — Input And Output With Pipes And Redirection In Linux

First, we’ll cover pipes as they’re the most often used of the redirection toolset. Pipes work just like they do in Super Mario. To demonstrate this we’ll use the standard commands echo and wc. The echo command simply displays the strings that are fed to it, the wc command counts lines, words, and characters.

Recommended — Complete Linux Lexicon Series

We can see that the name Tux is printed to the terminal just as expected. Now, let’s pipe Tux around and see what information we can get from him.

Here, wc tells us that it received one line of input, one word, and four characters. Tux only has five characters, but wc is counting the line-feed character ( n ) that doesn’t print a visible character, it just forces a new line. So, we’ll move away from a more contrived example of what pipes are and onto something equally contrived, but much more realistic and commonplace in IT. Below here we have an excerpt of an SSH log:

As you can imagine, a log that’s been appended for weeks, or months, can take a lot of time to read through, but we don’t have the time for that. So, using a pipe and another command called grep we can search through a large log in very little time.

With a single command line utilizing a pipe, we can see that someone is unsuccessfully trying to get into Tux’ account over SSH.

Next to pipes in the redirection toolset is what’s simply known as ‘redirection’, no special name for this one, but very useful nonetheless. The redirection tools come in two variants, input and output, and additionally, output itself can be further divided into two more tools.

Pipes are a form of output redirection, but pipes are a very particular case, they redirect the output to the input of another program. The more generic output direction allows the output of a program to be fed into a file. This is very useful when running scheduled scripts as it lets you capture the output that you weren’t there to witness. Let’s say you’d like to capture the last time the files in a given directory were accessed because you think someone might be trying to steal your notes. You could schedule the following (in whichever method you prefer) to snapshot the access time (atime) of the directory and write the output to a file.

And a simple as that, the output of your command will be captured in the script.log file. There is one thing that’s very important to note, here, and that’s that this command will overwrite the existing file every time it’s executed. If the desired effect is to append the log file, then you should use the following:

The difference being >> instead of >.

Now it’s time to address the three tools of the output redirection, we’ve seen >, which, itself, is a combination of two of the tools. There are two components of the output of any program in the shell, there are the stdout and stderr streams, both of which are displayed by the shell, often with no way to distinguish between them. The stdout stream is the standard output, or normal output, of the program, this is where our echo command was printing to. The stderr stream is the standard error output, this is typically where errors are printed. The benefit in splitting output into two streams is that they can be redirected to different places. We can see how this works below.

Imagine a directory filled with files, some of which belong to you, some don’t. When attempting to delete them, you will successfully delete those that belong to you, but the ones that don’t will produce an error.

You can see that we use 1> and 2>, they are stdout and stderr respectively, and the files following them are where the output is redirected. Again, we can user >> instead to ensure that we don’t overwrite the file if it already exists, which is very useful in the case of repeated logging.

Lastly, we have input redirection. Input redirection is providing a file to use as the source of input instead of the user manually providing input or redirecting the output of a previous command to use as input. We’ll use the same excerpt from our SSHD log from above.

All three commands produce the same output, which while redundant, is exactly what we want to see. But, why use input redirection when you can just use a pipe? When using a pipe to push output to another program, you’re involving another program to open the file and read it, and then finally pass it along the pipe. This means that there’s an additional executable being run, creating overhead of its own, and then beyond that, the reading of the file is buffered, and so is the writing of the file onto the pipe. This is trivial when dealing with small files, but it can slow down processes that are passed larger files, so it’s beneficial to simply let BASH (or whichever shell you use) to open the file for your program to read from.

How do you make use of input and output redirection in your daily tasks? Let us know in the comments below.

Recommended — Complete Linux Lexicon Series