Table of Contents

The Art of Unix Programming
Eric Steven Raymond
Dedication
Appendix燗.燝lossary of Abbreviations
Appendix燘.燫eferences
Bibliography
Appendix燙.燙ontributors
Evolution of C
Early History of C
C Standards
Culture? What Culture?
The Durability of Unix
The Case against Learning Unix Culture
What Unix Gets Wrong
What Unix Gets Right
Open-Source Software
Cross-Platform Portability and Open Standards
The Internet and the World Wide Web
The Open-Source Community
Flexibility All the Way Down
Unix Is Fun to Hack
The Lessons of Unix Can Be Applied Elsewhere
Basics of the Unix Philosophy
Rule of Modularity: Write simple parts connected by clean interfaces.
Rule of Clarity: Clarity is better than cleverness.
Rule of Composition: Design programs to be connected with other programs.
Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
Rule of Simplicity: Design for simplicity; add complexity only where you must.
Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
Rule of Transparency: Design for visibility to make inspection and debugging easier.
Rule of Robustness: Robustness is the child of爐ransparency and simplicity.
Rule of Representation: Fold knowledge into data, so爌rogram logic can be stupid and robust.
Rule of Least Surprise: In interface design, always do the爈east surprising thing.
Rule of Silence: When a program has nothing surprising to say, it should say nothing.
Rule of Repair: Repair what you can — but when you must fail, fail noisily and as soon as possible.
Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Rule of Diversity: Distrust all claims for “one true way”.
Rule of Extensibility: Design for the future, because it will be here sooner than you think.
The Unix Philosophy in One Lesson
Applying the Unix Philosophy
Attitude Matters Too
Origins and History of Unix, 1969-1995
Genesis: 1969–1971
Exodus: 1971–1980
TCP/IP and the Unix Wars: 1980-1990
Blows against the Empire: 1991-1995
The Open-Source Movement: 1998 and Onward
The Lessons of Unix History
The Elements of Operating-System Style
What Is the Operating System's Unifying Idea?
Multitasking Capability
Cooperating Processes
Internal Boundaries
File Attributes and Record Structures
Binary File Formats
Preferred User Interface Style
Intended Audience
Entry Barriers to Development
Operating-System Comparisons
VMS
MacOS
OS/2
Windows NT
BeOS
MVS
VM/CMS
Linux
What Goes Around, Comes Around
Encapsulation and Optimal Module Size
Compactness and Orthogonality
Compactness
Orthogonality
The SPOT Rule
Compactness and the Strong Single Center
The Value of Detachment
Software Is a Many-Layered Thing
Top-Down versus Bottom-Up
Glue Layers
Case Study: C Considered as Thin Glue
Libraries
Case Study: GIMP Plugins
Coding for Modularity
The Importance of Being Textual
Case Study: Unix Password File Format
Case Study: .newsrc Format
Case Study: The PNG Graphics File Format
Data File Metaformats
DSV Style
RFC 822 Format
Cookie-Jar Format
Record-Jar Format
XML
Windows INI Format
Unix Textual File Format Conventions
The Pros and Cons of File Compression
Application Protocol Design
Case Study: SMTP, the Simple Mail Transfer Protocol
Case Study: POP3, the Post Office Protocol
Case Study: IMAP, the Internet Message Access Protocol
Application Protocol Metaformats
The Classical Internet Application Metaprotocol
HTTP as a Universal Application Protocol
Case Study: The CDDB/freedb.org Database
Case Study: Internet Printing Protocol
BEEP: Blocks Extensible Exchange Protocol
XML-RPC, SOAP, and Jabber
Studying Cases
Case Study: audacity
Case Study: fetchmail's -v option
Case Study: GCC
Case Study: kmail
Case Study: SNG
Case Study: The Terminfo Database
Case Study: Freeciv Data Files
Designing for Transparency and Discoverability
The Zen of Transparency
Coding for Transparency and Discoverability
Transparency and Avoiding Overprotectiveness
Transparency and Editable Representations
Transparency, Fault Diagnosis, and Fault Recovery
Designing for Maintainability
Separating Complexity Control from Performance Tuning
Taxonomy of Unix IPC Methods
Handing off Tasks to Specialist Programs
Case Study: The mutt Mail User Agent
Pipes, Redirection, and Filters
Case Study: Piping to a Pager
Case Study: Making Word Lists
Case Study: pic2graph
Case Study: bc(1) and dc(1)
Anti-Case Study: Why Isn't fetchmail a Pipeline?
Wrappers
Case Study: Backup Scripts
Security Wrappers and Bernstein Chaining
Slave Processes
Case Study: scp and ssh
Peer-to-Peer Inter-Process Communication
Tempfiles
Signals
System Daemons and Conventional Signals
Case Study: fetchmail's Use of Signals
Sockets
Case Study: PostgreSQL
Case Study: Freeciv
Shared Memory
Problems and Methods to Avoid
Obsolescent Unix IPC Methods
System V IPC
Streams
Remote Procedure Calls
Threads — Threat or Menace?
Process Partitioning at the Design Level
Understanding the Taxonomy of Languages
Applying Minilanguages
Case Study: sng
Case Study: Regular Expressions
Case Study: Glade
Case Study: m4
Case Study: XSLT
Case Study: The Documenter's Workbench Tools
Case Study: fetchmail Run-Control Syntax
Case Study: awk
Case Study: PostScript
Case Study: bc and dc
Case Study: Emacs Lisp
Case Study: JavaScript
Designing Minilanguages
Choosing the Right Complexity Level
Extending and Embedding Languages
Writing a Custom Grammar
Macros — Beware!
Language or Application Protocol?
Data-Driven Programming
Case Study: ascii
Case Study: Statistical Spam Filtering
Case Study: Metaclass Hacking in fetchmailconf
Ad-hoc Code Generation
Case Study: Generating Code for the ascii Displays
Case Study: Generating HTML Code for a Tabular List
What Should Be Configurable?
Where Configurations Live
Run-Control Files
Case Study: The .netrc File
Portability to Other Operating Systems
Environment Variables
System Environment Variables
User Environment Variables
When to Use Environment Variables
Portability to Other Operating Systems
Command-Line Options
The -a to -z of Command-Line Options
Portability to Other Operating Systems
How to Choose among the Methods
Case Study: fetchmail
Case Study: The XFree86 Server
On Breaking These Rules
Applying the Rule of Least Surprise
History of Interface Design on Unix
Evaluating Interface Designs
Tradeoffs between CLI and Visual Interfaces
Case Study: Two Ways to Write a Calculator Program
Transparency, Expressiveness, and Configurability
Unix Interface Design Patterns
The Filter Pattern
The Cantrip Pattern
The Source Pattern
The Sink Pattern
The Compiler Pattern
The ed pattern
The Roguelike Pattern
The ‘Separated Engine and Interface’ Pattern
Configurator/Actor Pair
Spooler/Daemon Pair
Driver/Engine Pair
Client/Server Pair
The CLI Server Pattern
Language-Based Interface Patterns
Applying Unix Interface-Design Patterns
The Polyvalent-Program Pattern
The Web Browser as a Universal Front End
Silence Is Golden
Don't Just Do Something, Stand There!
Measure before Optimizing
Nonlocality Considered Harmful
Throughput vs. Latency
Batching Operations
Overlapping Operations
Caching Operation Results
Speaking of Complexity
The Three Sources of Complexity
Tradeoffs between Interface and Implementation Complexity
Essential, Optional, and Accidental Complexity
Mapping Complexity
When Simplicity Is Not Enough
A Tale of Five Editors
ed
vi
Sam
Emacs
Wily
The Right Size for an Editor
Identifying the Complexity Problems
Compromise Doesn't Work
Is Emacs an Argument against the Unix Tradition?
The Right Size of Software
Unix's Cornucopia of Languages
Interpreted Languages and Mixed Strategies
Language Evaluations
C
C Case Study: fetchmail
C++
C++ Case Study: The Qt Toolkit
Shell
Case Study: xmlto
Case Study: Sorcery Linux
Perl
A Small Perl Case Study: blq
A Large Perl Case Study: keeper
Tcl
Case Study: TkMan
Moodss: A Large Tcl Case Study
Python
A Small Python Case Study: imgsizer
A Medium-Sized Python Case Study: fetchmailconf
A Large Python Case Study: PIL
Java
Case Study: FreeNet
Emacs Lisp
Trends for the Future
Choosing an X Toolkit
A Developer-Friendly Operating System
Choosing an Editor
Useful Things to Know about vi
Useful Things to Know about Emacs
The Antireligious Choice: Using Both
Special-Purpose Code Generators
yacc and lex
Case Study: The fetchmailrc Grammar
Case Study: Glade
make: Automating Your Recipes
Basic Theory of make
make in Non-C/C++ Development
Case Study: make for Document-File Translation
Utility Productions
Generating Makefiles
makedepend
Imake
autoconf
automake
Version-Control Systems
Why Version Control?
Version Control by Hand
Automated Version Control
Unix Tools for Version Control
Source Code Control System (SCCS)
Revision Control System (RCS)
Concurrent Version System (CVS)
Other Version-Control Systems
Runtime Debugging
Profiling
Combining Tools with Emacs
Emacs and make
Emacs and Runtime Debugging
Emacs and Version Control
Emacs and Profiling
Like an IDE, Only Better
The Tale of J. Random Newbie
Transparency as the Key to Reuse
From Reuse to Open Source
The Best Things in Life Are Open
Where to Look?
Issues in Using Open-Source Software
Licensing Issues
What Qualifies as Open Source
Standard Open-Source Licenses
When You Need a Lawyer
Unix Standards
Standards and the Unix Wars
The Ghost at the Victory Banquet
Unix Standards in the Open-Source World
Specifications as DNA, Code as RNA
Programming for Portability
Portability and Choice of Language
C Portability
C++ Portability
Shell Portability
Perl Portability
Python Portability
Tcl Portability
Java Portability
Emacs Lisp Portability
Avoiding System Dependencies
Tools for Portability
Internationalization
Portability, Open Standards, and Open Source
Documentation Concepts
The Unix Style
The Large-Document Bias
Cultural Style
The Zoo of Unix Documentation Formats
troff and the Documenter's Workbench Tools
TeX
Texinfo
POD
HTML
DocBook
The Present Chaos and a Possible Way Out
DocBook
Document Type Definitions
Other DTDs
The DocBook Toolchain
Migration Tools
Editing Tools
Related Standards and Practices
SGML
XML-DocBook References
Best Practices for Writing Unix Documentation
Unix and Open Source
Best Practices for Working with Open-Source Developers
Good Patching Practice
Do send patches, don't send whole archives or files.
Send patches against the current version of the code.
Don't include patches for generated files.
Don't send patch bands that just tweak RCS or SCCS $-symbols.
Do use -c or -u format, don't use the default (-e) format.
Do include documentation with your patch.
Do include an explanation with your patch.
Do include useful comments in your code.
Don't take it personally if your patch is rejected
Good Project- and Archive-Naming Practice
Use GNU-style names with a stem and major.minor.patch numbering.
But respect local conventions where appropriate.
Try hard to choose a name prefix that is unique and easy to type.
Good Development Practice
Don't rely on proprietary code.
Use GNU Autotools.
Test your code before release.
Sanity-check your code before release.
Spell-check your documentation and READMEs before release.
Recommended C/C++ Portability Practices
Good Distribution-Making Practice
Make sure tarballs always unpack into a single new directory.
Include a README.
Respect and follow standard file-naming practices.
Design for upgradability.
Under Linux, provide RPMs.
Provide checksums.
Good Communication Practice
Announce to Freshmeat.
Announce to a relevant topic newsgroup.
Have a website.
Host project mailing lists.
Release to major archives.
The Logic of Licenses: How to Pick One
Why You Should Use a Standard License
Varieties of Open-Source Licensing
MIT or X Consortium License
BSD Classic License
Artistic License
General Public License
Mozilla Public License
Essence and Accident in Unix Tradition
Problems in the Design of Unix
A Unix File Is Just a Big Bag of Bytes
Unix Support for GUIs Is Weak
File Deletion Is Forever
Unix Assumes a Static File System
The Design of Job Control Was Badly Botched
The Unix API Doesn't Use Exceptions
ioctl(2) and fcntl(2) Are an Embarrassment
The Unix Security Model May Be Too Primitive
Unix Has Too Many Different Kinds of Names
File Systems Might Be Considered Harmful
Towards a Global Internet Address Space
Problems in the Environment of Unix
Problems in the Culture of Unix
Reasons to Believe
Community
Chapter�燙omplexity
As Simple As Possible, but燦o燬impler
Chapter�燙onfiguration
Starting on the Right Foot
Context
Chapter�Contrasts
Comparing the Unix Philosophy with Others
Internet FAQ Archives - Online Education
Internet FAQ Archives
Search the FAQ Archives
Other Information
Popular Usenet FAQs
Internet RFC/STD/FYI/BCP Archives
Contacting FAQ Maintainers; RFC Authors
Using this site
Frequently-Asked Questions
It's all about freedom...
...and it's all about me.
Design
Chapter�燚ocumentation
Explaining Your Code to燼燱eb-Centric World
Master Foo and the End User
Chapter�燜utures
Dangers and Opportunities
Chapter�Generation
Pushing the Specification Level Upwards
Master Foo Discourses on the Graphical User Interface
Origins and History of the Hackers, 1961-1995
At Play in the Groves of Academe: 1961-1980
Internet Fusion and the Free Software Movement: 1981-1991
Linux and the Pragmatist Reaction: 1991-1998
Chapter�History
A Tale of Two Cultures
IETF and the RFC Standards Process
Implementation
Chapter�營nterfaces
User-Interface Design Patterns in爐he燯nix Environment
Editor's Introduction
Chapter�燣anguages
To C or Not To C?
Master Foo and the Methodologist
Chapter�Minilanguages
Finding a Notation That Sings
Chapter�Modularity
Keeping It Clean, Keeping It Simple
Chapter�Multiprogramming
Separating Processes to燬eparate Function
Chapter�燨pen Source
Programming in the New Unix燙ommunity
Chapter�燨ptimization
Chapter�Philosophy
Philosophy Matters
Plan 9: The Way the Future Was
Chapter�燩ortability
Software Portability and燢eeping燯p燬tandards
Who Should Read This Book
How to Use This Book
Related References
Conventions Used in This Book
Our Case Studies
Author's Acknowledgements
Preface
Chapter�燫euse
On Not Reinventing the Wheel
Master Foo and the Script Kiddie
Master Foo and the Ten Thousand Lines
Chapter�Textuality
Good Protocols Make Good燩ractice
Chapter�燭ools
The Tactics of Development
Chapter�Transparency
Let There Be Light
Master Foo Discourses on the Two Paths
Unix and Object-Oriented Languages
Appendix燚.燫ootless Root
The Unix Koans of Master Foo
Master Foo Discourses on the Unix-Nature
Why Not C?
Master Foo and the Unix Zealot