hn-classics/_stories/1997/12787660.md

Source

Introduction of Pipes

Draft for Comment
** On the Early History and Impact of Unix
Tools to Build the Tools for a New Millenium
** by Ronda Hauben
rh120@columbia.edu

(Part 2 of 5)

Unix Is Born and the Introduction of Pipes

When AT&T; made the decision to pull out of the Multics collaboration, they took the research operating system off their GE 645 computer and put up the GECOS operating system. Though GECOS was adequate for applications, it was "nowhere near as satisfactory if you were trying to do things that were technically difficult and imperfectly defined," explained Vyssotsky, "which is the main task of research."(23)

For the pioneering work of Bell Labs research programmers like Ken Thompson and the research purposes of the Labs, an operating system more like what Multics had promised was needed. Along with the advantages of immediate feedback which time-sharing provided, the Bell Labs researchers wanted to continue the ability to work collaboratively which time-sharing had made possible.

"What we wanted to preserve," one of the creators of Unix, Dennis Ritchie writes, "was not just a good programming environment in which to do programming, but a system around which a fellowship could form. We knew from experience that the essence of communal computing, as supplied by remote-access, time-shared machines, is not just to type programs into a terminal instead of a keypunch, but to encourage close communication."(24)

Ritchie describes how an informal group led by Thompson had begun investigating alternatives to Multics before the GE-645 Multics machine had been removed from the Labs.(25) Thompson and Ritchie presented Bell Labs with proposals to buy them a computer so they could build their own interactive, time-sharing operating system. Their proposals weren't acted on. Eventually, Thompson found a little used PDP-7 computer. According to Vyssotsky, the orphaned PDP-7 computer was a machine, "more in the class of a Commodore 64 than the class of a PC-AT."(26)

Ritchie, Thompson, and Rudd Canady, who had been part of the Multics project, applied the lessons they had learned to the design of a file system for an experimental operating system. Writing on a chalk board, they created a file system design based on "the particularly simple way of viewing files that was suggested by the Multics I/O system."(27)

"Soon," Ritchie recounts, "Thompson began implementing the paper file system (perhaps `chalk file system' would be more accurate) that had been designed earlier." Thompson was eager to get a working model so he could test it out. He proceeded to create the other aspects of an operating system. "A file system without a way to exercise it was a sterile proposition," notes Ritchie, "so he [Thompson] proceeded to flesh it out with the other requirements for a working operating system, in particular the notion of processes."(28)

Describing the primitive conditions that Thompson faced, Ritchie writes, "At the start, Thompson did not even program on the PDP itself, but instead used a set of macros for the GEMAP assembler on a GE-635 machine."(29) A paper tape was generated on the GE 635 and then tested on the PDP-7 until, according to Ritchie, "a primitive Unix kernel, an editor, an assembler, a simple shell (command interpreter), and a few utilities (like the Unix rm, cat, cp commands) were completed. At this point, the operating system was self-supporting, programs could be written and tested without resort to paper tape, and development continued on the PDP-7 itself."(30)

Ritchie describes how Thompson's PDP-7 assembler was a model of simplicity. "There were no libraries, no loader or link editor," he writes, "the entire source of a program was presented to the assembler, and the output file -- with a fixed name -- that emerged was directly executable."(31) Once the assembler was completed, "the system was able to support itself. And thus the operating system we now call UNIX was born," notes Ritchie.(32)

Among the other active contributors during this period were Bell Labs researchers Rudd Canady and Joe Ossanna. The researchers were anxious to continue their work on a more advanced computer than the PDP-7. However, their efforts to get AT&T; to buy them a more advanced computer for their time-sharing research hadn't succeeded. With the help of Ossanna and another Bell Labs researcher Lee McMahon, they were finally able to convince management to buy them a new PDP-11 computer. To obtain this agreement, though, the researchers promised to create a text processing system. Doug McIlroy explains that "typists everywhere were seen as potential customers of the promised document- preparation program. Only later did Ossanna spot the patent department as a ripe candidate. By then there had already been clients in the telephone-operations business."(33) By Spring 1971, the Bell Labs Unix pioneers had a text formatter, fulfilling their promise. They had translated one they had created for the PDP-7 from a program McIlroy had written in the BCPL language while working on Multics. The text formatter was an assembler program for the PDP-11.(34)

One of the important developments in Unix was the introduction of pipes. Pipes had been suggested by McIlroy during the early days of creating Unix. Ritchie explains how "the idea, explained one afternoon on a blackboard, intrigued us but failed to ignite any immediate action. There were several objections to the idea as put....What a failure of imagination," he admits.(35) McIlroy concurs, describing how the initial effort to add pipes to Unix occurred about the same time in 1969 that Ritchie, Thompson and Canaday were outlining ideas for a file system. "That was when," he writes, "the simple pipeline as a way to combine programs, with data notationally propagating along a chain of (not necessarily concurrent) filters was articulated."(36) However, pipes weren't implemented in Unix until 1972. "Thanks to McIlroy's persistence," Ritchie writes, "pipes were finally installed in the operating system (a relatively simple job), and a new notation was introduced."(37) Several of the old commands had to be changed to make them usable with pipes, notes Ritchie. Summarizing how pipes found their way into Unix, Vyssotsky notes that Thompson put them in, but "it was McIlroy who said, `look you ought to do it.' Pipes, like most things in Unix were not a radically new idea."(38) He describes how similar ideas had appeared in other languages like SIMULA as early as 1967.

Dick Haight, a Unix pioneer who helped develop the Programmer's Workbench of Unix tools, was present the day that pipes were introduced into Unix. He describes what happened:

"I happened to have been visiting the research crew the day they implemented pipes. It was clear to everyone practically minutes after the system came up with pipes working that it was a wonderful thing. Nobody would ever go back and give that up if they could help it."(39)

Also describing the day pipes were introduced, McIlroy writes:

"Open Systems! Our Systems! How well those who were there remember the pipe-festooned garret where Unix took form. The excitement of creation drew people to work there amidst the whine of the computer's cool- ing fans, even though almost the same computer ac- cess, could be had from one's office or from home. Those raw quarters saw a procession of memorable events. The advent of software pipes precipitated a day-long orgy of one-liners....As people reveled in the power of functional composition in the large, which is even today unavailable to users of other systems."(40)

The Software Tool

Pipes had been created by the time the Version 3 Unix Manual appeared in February 1973. The date listed for the creation of pipes is January 15, 1973(41). Not only were pipes a significant addition to Unix, but according to McIlroy, pipes made possible a subsequent important discovery. "In another memorable event," he writes, "the unarticulated notion of software tools, which had been bolstered by pipes, was finally brought home by the liberation of the pattern matching program grep from within the editor."

McIlroy describes how he asked Thompson to create a program to help him with some work that he was trying to do. This program resulted in the invention of the software tool grep.' Following is McIlroy's account of how grep' was taken out from the editor, leading to a clearer understanding of the notion of a software tool. He writes:

"Grep was invented for me. I was making a program to read text aloud through a voice synthesizer. As I invented phonetic rules I would check Webster's dictionary for words on which they might fail. For example, how do you cope with the digraph ui', which is pronounced many different ways: fruit', guile', guilty', anguish', intuit', `beguine'? I would break the dictionary up into pieces that fit in ed's limited buffer and use a global command to select a list. I would whittle this list down by repeated scannings with ed to see how each proposed rule worked."

"The process was tedious, and terribly wasteful, since the dictionary had to be split (one couldn't afford to leave a split copy on line). Then ed copied each part into /tmp, scanned it twice to accomplish the g command, and finally threw it away, which takes time too."

"One afternoon I asked Ken Thompson if he could lift the regular expression recognizer out of the editor and make a one-pass program to do it. He said yes. The next morning I found a note in my mail announcing a program named grep. It worked like a charm. When asked what that funny name meant, Ken said it was obvious. It stood for the editor command that it simulated, g/re/p (global regular expression print)."

"Progress on my talking program accelerated dramatically. From that special-purpose beginning, grep soon became a household word. (Something I had to stop myself from writing in the first paragraph above shows how firmly naturalized the idea now is: I used ed to grep out words from the dictionary.') More than any other single program, grep focused the viewpoint that Kernighan and Plauger christened and formalized in Software Tools': make programs that do one thing and do it well, with as few preconceptions about input syntax as possible."(42)

Grep is listed in the Manual for Version 4 Unix which is dated November, 1973. The date given for the creation of grep is March 3, 1973, following the creation of pipes.(43) The creation of grep, McIlroy explains, was followed by the invention of other special purpose software programs that could be used as tools. He writes:

"A while later a demand arose for another special-purpose program, gres, for substitution: g/re/s. Lee McMahon undertook to write it, and soon foresaw that there would be no end to the family: g/re/d, g/re/a, etc. As his concept developed it became sed, a tool of remarkable utility that is largely unappreciated today, because it capitalizes on the perfect familiarity with ed that was universal ten years ago, but no more. Sed covers a lot of needs. For example, we have never seen fit to adopt the widespread head' utility because sed 10q' does just as well."(44)

What McIlroy refers to as "the unarticulated notion of software tools...brought home by the liberation of...grep from within the editor" has become one of the significant achievements of the Bell Labs research work on Unix. By making it possible to use `grep' to search for a data pattern in a file, without having to use an editor to go inside the file, the Bell Labs researchers discovered that they could create a plethora of software tools to be used in varying combinations, thus facilitating the customized application by a user, a goal sought by those doing research in programming.(45)

McIlroy explains that the notion of "software tool" only became articulated among the Bell Labs researchers with the publication of the book "Software Tools" by Kernighan and Plaugher. "Still unnamed in our circles," McIlroy notes, "until Kernighan and Plauger wrote their book, the idea nevertheless became a guiding principle."(46) McIlroy adds that "We were definitely building tools before K&P;, though we lacked the suggestive word."(47)

Describing how the notion of software tools helps to create an effective programming environment, Brian W. Kernighan and Rob Pike, authors of The Unix Programming Environment, explain that each tool is designed to be used with other tools and achieving this end is more important than how each is designed internally.(48) The most important aspect considered with each tool is the function that the tool is to have. New tools are designed once a new function is intended.

"The guiding principle," they write, "for making the choice should be that each program does one thing. Options are appropriately added to a program that already has the right functionality. If there is no such program then a new program is called for. In that case the usual criteria for program design should be used: the program should be as general as possible, its default behavior should match the most common usage and it should cooperate with other programs."(49)

Thus Unix, according to Kernighan and Pike, created "a new style of computing, a new way of thinking of how to attack a problem with a computer. This style," they explain, "was based on the use of tools: using programs separately or in combination to get a job done, rather than doing it by hand, by monolithic self-sufficient subsystems, or by special-purpose, one-time programs."(50)

The philosophy of using software tools that developed from research in Unix is outlined in the "Foreward" to the special issue of The Bell System Technical Journal published in 1978, on "The Unix Time-sharing System." Describing the principles which they have found to be an important component of the Unix philosophy of software design, the researchers write:

"UNIX utilities are usually thought of as tools -- sharply honed programs that help with generic data processing tasks. Tools were often invented to help with the development of UNIX programs and were continually improved by much trial, error, discussion, and redesign, as was the operating system itself. Tools may be used in combination to perform or construct specific applications."(51)

They explain that a distinctive style evolved as part of Unix research. "Unix software works smoothly together; elaborate computing tasks are typically composed from loosely coupled small parts, often software tools taken off the shelf."(52)

"Sophisticated tools to make tools have evolved," they observe. (53) Software development tools such as "nroff" and "troff" were created. Not only was it important to create tools, but soon tools to create tools, like "yacc" and "lex", were developed. "Yacc" and "lex" were used to create numerous little languages and applications like eqn and awk that greatly enhanced the popularity of Unix.

The evolution of the tool "diff" created by McIlroy, a commonly used software tool, is an example of how tools were continually improved, based on the experience gained using them. McIlroy reports how he based his work on algorithms created by others and then he tried three different algorithms before settling on the one finally used.(54) Other Unix tools were created through a similar process. Each program was created to fulfill some simple capability and was called a tool. The programs were designed to be fun to use and to be helpful to programmers. Among the principles guiding the toolbuilders were:

1. "Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.
2. "Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
3. "Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
4. "Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them."(55)

"Our goals throughout the effort," write Ritchie and Thompson, in describing the research objectives of their work on Unix, "when articulated at all, have always been to build a comfortable relationship with the machine and to explain ideas and inventions in operating systems and other software." (56)

Frederick P. Brooks Jr is another computer pioneer who recognized the importance of pipes and combining single function programs into what he calls "a unified programming environment" to make the work of programmers more productive. Brooks describes Unix as one of "the first integrated environments to come into widespread use," which "improved productivity by integral factors."(57)

In explaining how such an environment functions, he writes:

"They attack the accidental difficulties that result from using individual programs together, by providing integrated libraries, unified file formats, and pipes and filters. As a result, conceptual structures that in principle could always call, feed, and use one another can easily do so in practice." (58)

Following the development of single function software tools which could be used together via pipes in a programming environment, came the development of whole toolchests and workbenches of tools to rationalize the work of programmers.

to be continued