Pipeline (Unix)

from Wikipedia, the free encyclopedia
A pipeline running on a text terminal consisting of three programs with stdin , stdout and stderr

A pipeline in Unix-like operating systems consists of programs that are linked to one another via their standard data streams . A program "receives" the standard output (stdout) of the previous program in the chain via the standard input (stdin) . The individual programs are called up one after the other by the previous one.

In the Shell - Syntax is a vertical bar ( |in Unix - jargon called "pipe character" "Pipe" or) used as a pipe character of each program:

Programm1  |  Programm2  |  Programm3

The operating system here uses the principle of " unnamed pipe ", the oldest and the first technique of interprocess communication .

example

  1. Use to ls -dl *key*create a list of all files in the current directory with the string "key" in their name.
  2. Reduce the output list with grep -v "failed"by those lines that contain the string "failed".
  3. Output the resulting list for paging with less :
ls -dl *key*  |  grep -v "failed"  |  less

Standard error

By default , the pipeline collects the standard error outputs (stderr) of its individual child processes and forwards them to the command line at the end . The alternative notation " |&" of the C shell , however, merges the standard output and the standard error output of the respective child processes and passes them on to the next process as standard output . In addition, the pipeline passes in the Bourne shell since Bash 4.0 , the standard error with the notation 2>&1in a file.

history

Douglas McIlroy discovered during his time at Bell Labs (the birthplace of Unix ) that very often users would use the output file of one program as the input file for another. He therefore introduced the concept of the pipe, which Ken Thompson expanded in 1973 in version 3 Unix through the system call pipe()and for the shell through the pipe.

Ken Thompson also introduced the above notation, which greatly simplified the description of the pipe syntax in version 4 Unix .

Many operating systems such as DOS , OS / 2 , Microsoft Windows , and BeOS later implemented the pipeline with identical notation and Tony Hoare further developed the concept of the pipeline with CSP .

The computer program Automator from Apple also concatenates repetitive commands with pipelines. The robot - Icon therefore contributes as a tribute to the original Unix concept a pipe (pipe) in his hands.

See also

  • Everything is a file - cornerstone of the Unix philosophy, pipelines process files in the Unix sense
  • GStreamer - pipeline-based multimedia framework
  • GNU parallel - parallel processing of several shells, also with data from pipes
  • Tee (Unix) - Unix command for tapping data from a pipeline
  • XML pipeline - processing XML files
  • xargs - Unix command to convert standard outputs into other Unix commands via a pipe
  • netcat , socat Connection to TCP / IP Internet socket using pipes.

Web links

Individual evidence

  1. An Introduction to Linux I / O Redirection. DigitalOcean, accessed on September 14, 2018 (Pipes section).
  2. Jürgen Wolf, Klaus-Jürgen Wolf: Linux-Unix-Programming. The comprehensive manual. 2nd edition, Rheinwerk Verlag: Bonn, 2006, chapter 9.1.1.
  3. C-Shell Quick Reference. (PDF) Hamilton Laboratories, accessed September 14, 2018 .
  4. Bash release notes. Case Western Reserve University , accessed September 14, 2018 (Bash-3.2 release: 1dd).
  5. ^ Michael S. Mahoney : The Unix Oral History Project: Release.0, The Beginning. Retrieved September 14, 2018 .
  6. ^ Douglas McIlroy ( Bell Labs ): A Research Unix reader: annotated excerpts from the Programmer's Manual. (PDF) In: CSTR, number 139, 1971-1986. Retrieved September 14, 2018 .
  7. ^ Pipes: A Brief Introduction. The Linux Information Project (LINFO), accessed September 14, 2018 .
  8. Russ Cox: Bell Labs and CSP Threads. Retrieved September 14, 2018 .