[Source](http://www.catb.org/esr/writings/taoup/html/ "Permalink to The Art of Unix Programming")
# The Art of Unix Programming
| ----- |
| The Art of Unix Programming |
| | | [Next][1] |
* * *
# The Art of Unix Programming
### Eric Steven Raymond
[ Thyrsus Enterprises][2]
`<[esr@thyrsus.com][3]>`
Copyright 2003 Eric S. Raymond
This book and its on-line version are distributed under the terms of the Creative Commons Attribution-NoDerivs 1.0 license, with the additional proviso that the right to publish it on paper for sale or other for-profit use is reserved to Pearson Education, Inc. A reference copy of this license may be found at <http://creativecommons.org/licenses/by-nd/1.0/legalcode>.
AIX, AS/400, DB/2, OS/2, System/360, MVS, VM/CMS, and IBM PC are trademarks of IBM. Alpha, DEC, VAX, HP-UX, PDP, TOPS-10, TOPS-20, VMS, and VT-100 are trademarks of Compaq. Amiga and AmigaOS are trademarks of Amiga, Inc. Apple, Macintosh, MacOS, Newton, OpenDoc, and OpenStep are trademarks of Apple Computers, Inc. ClearCase is a trademark of Rational Software, Inc. Ethernet is a trademark of 3COM, Inc. Excel, MS-DOS, Microsoft Windows and PowerPoint are trademarks of Microsoft, Inc. Java. J2EE, JavaScript, NeWS, and Solaris are trademarks of Sun Microsystems. SPARC is a trademark of SPARC international. Informix is a trademark of Informix software. Itanium is a trademark of Intel. Linux is a trademark of Linus Torvalds. Netscape is a trademark of AOL. PDF and PostScript are trademarks of Adobe, Inc. UNIX is a trademark of The Open Group.
The photograph of Ken and Dennis in [Chapter2][4] appears courtesy of Bell Labs/Lucent Technologies.
The epigraph on the Portability chapter is from the Bell System Technical Journal, v57 #6 part 2 (July-Aug. 1978) pp. 2021-2048 and is reproduced with the permission of Bell Labs/Lucent Technologies.
| ----- |
| **Revision History** |
| Revision 1.0 | 19 September 2003 | esr |
| This is the content that went to Addison-Wesley's printers. | | |
| Revision 0.4 | 5 February 2003 | esr |
| Release for public review. | | |
| Revision 0.3 | 22 January 2003 | esr |
| First eighteen-chapter draft. Manuscript walkthrough at Chapter 12. Limited release for early reviewers. | | |
| Revision 0.2 | 2 January 2003 | esr |
| First manuscript walkthrough at Chapter 7. Released to Dmitry Kirsanov at AW production. | | |
| Revision 0.1 | 16 November 2002 | esr |
| First DocBook draft, fifteen chapters. Languages rewritten to incorporate lots of feedback. Transparency, Modularity, Multiprogramming, Configuration, Interfaces, Documentation, and Open Source chapters released. Shipped to Mark Taub at AW. | | |
| Revision 0.0 | 1999 | esr |
| Public HTML draft, first four chapters only. | | |
* * *
## Dedication
To Ken Thompson and Dennis Ritchie, because you inspired me.
**Table of Contents**
[Preface][1]
:
[Who Should Read This Book][5]
[How to Use This Book][6]
[Related References][7]
[Conventions Used in This Book][8]
[Our Case Studies][9]
[Author's Acknowledgements][10]
I. [Context][11]
:
1\. [Philosophy][12]
:
[Culture? What Culture?][13]
[The Durability of Unix][14]
[The Case against Learning Unix Culture][15]
[What Unix Gets Wrong][16]
[What Unix Gets Right][17]
:
[Open-Source Software][18]
[Cross-Platform Portability and Open Standards][19]
[The Internet and the World Wide Web][20]
[The Open-Source Community][21]
[Flexibility All the Way Down][22]
[Unix Is Fun to Hack][23]
[The Lessons of Unix Can Be Applied Elsewhere][24]
[Basics of the Unix Philosophy][25]
:
[Rule of Modularity: Write simple parts connected by clean interfaces.][26]
[Rule of Clarity: Clarity is better than cleverness.][27]
[Rule of Composition: Design programs to be connected with other programs.][28]
[Rule of Separation: Separate policy from mechanism; separate interfaces from engines.][29]
[Rule of Simplicity: Design for simplicity; add complexity only where you must.][30]
[Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.][31]
[Rule of Transparency: Design for visibility to make inspection and debugging easier.][32]
[Rule of Robustness: Robustness is the child oftransparency and simplicity.][33]
[Rule of Representation: Fold knowledge into data, soprogram logic can be stupid and robust.][34]
[Rule of Least Surprise: In interface design, always do theleast surprising thing.][35]
[Rule of Silence: When a program has nothing surprising to say, it should say nothing.][36]
[Rule of Repair: Repair what you can — but when you must fail, fail noisily and as soon as possible.][37]
[Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.][38]
[Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.][39]
[Rule of Optimization: Prototype before polishing. Get it working before you optimize it.][40]
[Rule of Diversity: Distrust all claims for one true way.][41]
[Rule of Extensibility: Design for the future, because it will be here sooner than you think.][42]
[The Unix Philosophy in One Lesson][43]
[Applying the Unix Philosophy][44]
[Attitude Matters Too][45]
2\. [History][46]
:
[Origins and History of Unix, 1969-1995][47]
:
[Genesis: 1969–1971][48]
[Exodus: 1971–1980][49]
[TCP/IP and the Unix Wars: 1980-1990][50]
[Blows against the Empire: 1991-1995][51]
[Origins and History of the Hackers, 1961-1995][52]
:
[At Play in the Groves of Academe: 1961-1980][53]
[Internet Fusion and the Free Software Movement: 1981-1991][54]
[Linux and the Pragmatist Reaction: 1991-1998][55]
[The Open-Source Movement: 1998 and Onward][56]
[The Lessons of Unix History][57]
3\. [Contrasts][58]
:
[The Elements of Operating-System Style][59]
:
[What Is the Operating System's Unifying Idea?][60]
[Multitasking Capability][61]
[Cooperating Processes][62]
[Internal Boundaries][63]
[File Attributes and Record Structures][64]
[Binary File Formats][65]
[Preferred User Interface Style][66]
[Intended Audience][67]
[Entry Barriers to Development][68]
[Operating-System Comparisons][69]
:
[VMS][70]
[MacOS][71]
[OS/2][72]
[Windows NT][73]
[BeOS][74]
[MVS][75]
[VM/CMS][76]
[Linux][77]
[What Goes Around, Comes Around][78]
II. [Design][79]
:
4\. [Modularity][80]
:
[Encapsulation and Optimal Module Size][81]
[Compactness and Orthogonality][82]
:
[Compactness][83]
[Orthogonality][84]
[The SPOT Rule][85]
[Compactness and the Strong Single Center][86]
[The Value of Detachment][87]
[Software Is a Many-Layered Thing][88]
:
[Top-Down versus Bottom-Up][89]
[Glue Layers][90]
[Case Study: C Considered as Thin Glue][91]
[Libraries][92]
:
[Case Study: GIMP Plugins][93]
[Unix and Object-Oriented Languages][94]
[Coding for Modularity][95]
5\. [Textuality][96]
:
[The Importance of Being Textual][97]
:
[Case Study: Unix Password File Format][98]
[Case Study: .newsrc Format][99]
[Case Study: The PNG Graphics File Format][100]
[Data File Metaformats][101]
:
[DSV Style][102]
[RFC 822 Format][103]
[Cookie-Jar Format][104]
[Record-Jar Format][105]
[XML][106]
[Windows INI Format][107]
[Unix Textual File Format Conventions][108]
[The Pros and Cons of File Compression][109]
[Application Protocol Design][110]
:
[Case Study: SMTP, the Simple Mail Transfer Protocol][111]
[Case Study: POP3, the Post Office Protocol][112]
[Case Study: IMAP, the Internet Message Access Protocol][113]
[Application Protocol Metaformats][114]
:
[The Classical Internet Application Metaprotocol][115]
[HTTP as a Universal Application Protocol][116]
[BEEP: Blocks Extensible Exchange Protocol][117]
[XML-RPC, SOAP, and Jabber][118]
6\. [Transparency][119]
:
[Studying Cases][120]
:
[Case Study: audacity][121]
[Case Study: fetchmail's -v option][122]
[Case Study: GCC][123]
[Case Study: kmail][124]
[Case Study: SNG][125]
[Case Study: The Terminfo Database][126]
[Case Study: Freeciv Data Files][127]
[Designing for Transparency and Discoverability][128]
:
[The Zen of Transparency][129]
[Coding for Transparency and Discoverability][130]
[Transparency and Avoiding Overprotectiveness][131]
[Transparency and Editable Representations][132]
[Transparency, Fault Diagnosis, and Fault Recovery][133]
[Designing for Maintainability][134]
7\. [Multiprogramming][135]
:
[Separating Complexity Control from Performance Tuning][136]
[Taxonomy of Unix IPC Methods][137]
:
[Handing off Tasks to Specialist Programs][138]
[Pipes, Redirection, and Filters][139]
[Wrappers][140]
[Security Wrappers and Bernstein Chaining][141]
[Slave Processes][142]
[Peer-to-Peer Inter-Process Communication][143]
[Problems and Methods to Avoid][144]
:
[Obsolescent Unix IPC Methods][145]
[Remote Procedure Calls][146]
[Threads — Threat or Menace?][147]
[Process Partitioning at the Design Level][148]
8\. [Minilanguages][149]
:
[Understanding the Taxonomy of Languages][150]
[Applying Minilanguages][151]
:
[Case Study: sng][152]
[Case Study: Regular Expressions][153]
[Case Study: Glade][154]
[Case Study: m4][155]
[Case Study: XSLT][156]
[Case Study: The Documenter's Workbench Tools][157]
[Case Study: fetchmail Run-Control Syntax][158]
[Case Study: awk][159]
[Case Study: PostScript][160]
[Case Study: bc and dc][161]
[Case Study: Emacs Lisp][162]
[Case Study: JavaScript][163]
[Designing Minilanguages][164]
:
[Choosing the Right Complexity Level][165]
[Extending and Embedding Languages][166]
[Writing a Custom Grammar][167]
[Macros — Beware!][168]
[Language or Application Protocol?][169]
9\. [Generation][170]
:
[Data-Driven Programming][171]
:
[Case Study: ascii][172]
[Case Study: Statistical Spam Filtering][173]
[Case Study: Metaclass Hacking in fetchmailconf][174]
[Ad-hoc Code Generation][175]
:
[Case Study: Generating Code for the ascii Displays][176]
[Case Study: Generating HTML Code for a Tabular List][177]
10\. [Configuration][178]
:
[What Should Be Configurable?][179]
[Where Configurations Live][180]
[Run-Control Files][181]
:
[Case Study: The .netrc File][182]
[Portability to Other Operating Systems][183]
[Environment Variables][184]
:
[System Environment Variables][185]
[User Environment Variables][186]
[When to Use Environment Variables][187]
[Portability to Other Operating Systems][188]
[Command-Line Options][189]
:
[The -a to -z of Command-Line Options][190]
[Portability to Other Operating Systems][191]
[How to Choose among the Methods][192]
:
[Case Study: fetchmail][193]
[Case Study: The XFree86 Server][194]
[On Breaking These Rules][195]
11\. [Interfaces][196]
:
[Applying the Rule of Least Surprise][197]
[History of Interface Design on Unix][198]
[Evaluating Interface Designs][199]
[Tradeoffs between CLI and Visual Interfaces][200]
:
[Case Study: Two Ways to Write a Calculator Program][201]
[Transparency, Expressiveness, and Configurability][202]
[Unix Interface Design Patterns][203]
:
[The Filter Pattern][204]
[The Cantrip Pattern][205]
[The Source Pattern][206]
[The Sink Pattern][207]
[The Compiler Pattern][208]
[The ed pattern][209]
[The Roguelike Pattern][210]
[The ‘Separated Engine and Interface’ Pattern][211]
[The CLI Server Pattern][212]
[Language-Based Interface Patterns][213]
[Applying Unix Interface-Design Patterns][214]
:
[The Polyvalent-Program Pattern][215]
[The Web Browser as a Universal Front End][216]
[Silence Is Golden][217]
12\. [Optimization][218]
:
[Don't Just Do Something, Stand There!][219]
[Measure before Optimizing][220]
[Nonlocality Considered Harmful][221]
[Throughput vs. Latency][222]
:
[Batching Operations][223]
[Overlapping Operations][224]
[Caching Operation Results][225]
13\. [Complexity][226]
:
[Speaking of Complexity][227]
:
[The Three Sources of Complexity][228]
[Tradeoffs between Interface and Implementation Complexity][229]
[Essential, Optional, and Accidental Complexity][230]
[Mapping Complexity][231]
[When Simplicity Is Not Enough][232]
[A Tale of Five Editors][233]
:
[ed][234]
[vi][235]
[Sam][236]
[Emacs][237]
[Wily][238]
[The Right Size for an Editor][239]
:
[Identifying the Complexity Problems][240]
[Compromise Doesn't Work][241]
[Is Emacs an Argument against the Unix Tradition?][242]
[The Right Size of Software][243]
III. [Implementation][244]
:
14\. [Languages][245]
:
[Unix's Cornucopia of Languages][246]
[Why Not C?][247]
[Interpreted Languages and Mixed Strategies][248]
[Language Evaluations][249]
:
[C][250]
[C++][251]
[Shell][252]
[Perl][253]
[Tcl][254]
[Python][255]
[Java][256]
[Emacs Lisp][257]
[Trends for the Future][258]
[Choosing an X Toolkit][259]
15\. [Tools][260]
:
[A Developer-Friendly Operating System][261]
[Choosing an Editor][262]
:
[Useful Things to Know about vi][263]
[Useful Things to Know about Emacs][264]
[The Antireligious Choice: Using Both][265]
[Special-Purpose Code Generators][266]
:
[yacc and lex][267]
[Case Study: Glade][268]
[make: Automating Your Recipes][269]
:
[Basic Theory of make][270]
[make in Non-C/C++ Development][271]
[Utility Productions][272]
[Generating Makefiles][273]
[Version-Control Systems][274]
:
[Why Version Control?][275]
[Version Control by Hand][276]
[Automated Version Control][277]
[Unix Tools for Version Control][278]
[Runtime Debugging][279]
[Profiling][280]
[Combining Tools with Emacs][281]
:
[Emacs and make][282]
[Emacs and Runtime Debugging][283]
[Emacs and Version Control][284]
[Emacs and Profiling][285]
[Like an IDE, Only Better][286]
16\. [Reuse][287]
:
[The Tale of J. Random Newbie][288]
[Transparency as the Key to Reuse][289]
[From Reuse to Open Source][290]
[The Best Things in Life Are Open][291]
[Where to Look?][292]
[Issues in Using Open-Source Software][293]
[Licensing Issues][294]
:
[What Qualifies as Open Source][295]
[Standard Open-Source Licenses][296]
[When You Need a Lawyer][297]
IV. [Community][298]
:
17\. [Portability][299]
:
[Evolution of C][300]
:
[Early History of C][301]
[C Standards][302]
[Unix Standards][303]
:
[Standards and the Unix Wars][304]
[The Ghost at the Victory Banquet][305]
[Unix Standards in the Open-Source World][306]
[IETF and the RFC Standards Process][307]
[Specifications as DNA, Code as RNA][308]
[Programming for Portability][309]
:
[Portability and Choice of Language][310]
[Avoiding System Dependencies][311]
[Tools for Portability][312]
[Internationalization][313]
[Portability, Open Standards, and Open Source][314]
18\. [Documentation][315]
:
[Documentation Concepts][316]
[The Unix Style][317]
:
[The Large-Document Bias][318]
[Cultural Style][319]
[The Zoo of Unix Documentation Formats][320]
:
[troff and the Documenter's Workbench Tools][321]
[TeX][322]
[Texinfo][323]
[POD][324]
[HTML][325]
[DocBook][326]
[The Present Chaos and a Possible Way Out][327]
[DocBook][328]
:
[Document Type Definitions][329]
[Other DTDs][330]
[The DocBook Toolchain][331]
[Migration Tools][332]
[Editing Tools][333]
[Related Standards and Practices][334]
[SGML][335]
[XML-DocBook References][336]
[Best Practices for Writing Unix Documentation][337]
19\. [Open Source][338]
:
[Unix and Open Source][339]
[Best Practices for Working with Open-Source Developers][340]
:
[Good Patching Practice][341]
[Good Project- and Archive-Naming Practice][342]
[Good Development Practice][343]
[Good Distribution-Making Practice][344]
[Good Communication Practice][345]
[The Logic of Licenses: How to Pick One][346]
[Why You Should Use a Standard License][347]
[Varieties of Open-Source Licensing][348]
:
[MIT or X Consortium License][349]
[BSD Classic License][350]
[Artistic License][351]
[General Public License][352]
[Mozilla Public License][353]
20\. [Futures][354]
:
[Essence and Accident in Unix Tradition][355]
[Plan 9: The Way the Future Was][356]
[Problems in the Design of Unix][357]
:
[A Unix File Is Just a Big Bag of Bytes][358]
[Unix Support for GUIs Is Weak][359]
[File Deletion Is Forever][360]
[Unix Assumes a Static File System][361]
[The Design of Job Control Was Badly Botched][362]
[The Unix API Doesn't Use Exceptions][363]
[ioctl2 and fcntl2 Are an Embarrassment][364]
[The Unix Security Model May Be Too Primitive][365]
[Unix Has Too Many Different Kinds of Names][366]
[File Systems Might Be Considered Harmful][367]
[Towards a Global Internet Address Space][368]
[Problems in the Environment of Unix][369]
[Problems in the Culture of Unix][370]
[Reasons to Believe][371]
A. [Glossary of Abbreviations][372]
B. [References][373]
C. [Contributors][374]
D. [Rootless Root][375]
:
[Editor's Introduction][376]
[Master Foo and the Ten Thousand Lines][377]
[Master Foo and the Script Kiddie][378]
[Master Foo Discourses on the Two Paths][379]
[Master Foo and the Methodologist][380]
[Master Foo Discourses on the Graphical User Interface][381]
[Master Foo and the Unix Zealot][382]
[Master Foo Discourses on the Unix-Nature][383]
[Master Foo and the End User][384]
**List of Figures**
2.1. [The PDP-7.][385]
3.1. [Schematic history of timesharing.][386]
4.1. [Qualitative plot of defect count and density vs. module size.][387]
4.2. [Caller/callee relationships in GIMP with a plugin loaded.][388]
6.1. [Screen shot of audacity.][389]
6.2. [Screen shot of kmail.][390]
6.3. [Main window of a Freeciv game.][391]
8.1. [Taxonomy of languages.][392]
11.1. [The xcalc GUI.][393]
11.2. [Screen shot of the original Rogue game.][394]
11.3. [The Xcdroast GUI.][395]
11.4. [Caller/callee relationships in a polyvalent program.][396]
13.1. [Sources and kinds of complexity.][397]
18.1. [Processing structural documents.][398]
18.2. [Present-day XML-DocBook toolchain.][399]
18.3. [Future XML-DocBook toolchain with FOP.][400]
**List of Tables**
8.1. [Regular-expression examples.][401]
8.2. [Introduction to regular-expression operations.][402]
14.1. [Language choices.][403]
14.2. [Summary of X Toolkits.][404]
**List of Examples**
5.1. [Password file example.][405]
5.2. [A .newsrc example.][406]
5.3. [A fortune file example.][407]
5.4. [Basic data for three planets in a record-jar format.][408]
5.5. [An XML example.][409]
5.6. [A .INI file example.][410]
5.7. [An SMTP session example.][411]
5.8. [A POP3 example session.][412]
5.9. [An IMAP session example.][413]
6.1. [An example fetchmail -v transcript.][414]
6.2. [An SNG Example.][415]
7.1. [The pic2graph pipeline.][416]
8.1. [Glade Hello, World.][417]
8.2. [A sample m4 macro.][418]
8.3. [A sample XSLT program.][419]
8.4. [Taxonomy of languages — the pic source.][420]
8.5. [Synthetic example of a fetchmailrc.][421]
8.6. [RSA implementation using dc.][422]
9.1. [Example of fetchmailrc syntax.][423]
9.2. [Python structure dump of a fetchmail configuration.][424]
9.3. [copy_instance metaclass code.][425]
9.4. [Calling context for copy_instance.][426]
9.5. [ascii usage screen.][427]
9.6. [Desired output format for the star table.][428]
