Linux Server Hacks – Read Now and Download Mobi

Linux Server Hacks

Table of Contents

A Note Regarding Supplemental Files

How to Become a Hacker

The Hacker Attitude

1. 1. The world is full of fascinating problems waiting to be solved.

2. 2. No problem should ever have to be solved twice.

3. 3. Boredom and drudgery are evil.

4. 4. Freedom is good.

5. 5. Attitude is no substitute for competence.

Preface

How This Book is Organized

How to Use This Book

Conventions Used in This Book

How to Contact Us

Gotta Hack?

Credits

Acknowledgments

1. Server Basics

1.1. Hacks #1-22

Hack #1. Removing Unnecessary Services

1.1. See also:

Hack #2. Forgoing the Console Login

Hack #3. Common Boot Parameters

37. See also:

Hack #4. Creating a Persistent Daemon with init

4.1. See also:

Hack #5. n>&m: Swap Standard Output and Standard Error

Hack #6. Building Complex Command Lines

6.1. Hacking the Hack

Hack #7. Working with Tricky Files in xargs

7.1. Listing: albumize

Hack #8. Immutable Files in ext2/ext3

Hack #9. Speeding Up Compiles

Hack #10. At Home in Your Shell Environment

Hack #11. Finding and Eliminating setuid/setgid Binaries

Hack #12. Make sudo Work Harder

Hack #13. Using a Makefile to Automate Admin Tasks

13.1. Listing: Makefile.mail

13.2. Listing: Makefile.push

Hack #14. Brute Forcing Your New Domain Name

Hack #15. Playing Hunt the Disk Hog

Hack #16. Fun with /proc

Hack #17. Manipulating Processes Symbolically with procps

17.1. See also:

Hack #18. Managing System Resources per Process

18.1. See also:

Hack #19. Cleaning Up after Ex-Users

Hack #20. Eliminating Unnecessary Drivers from the Kernel

20.1. See also:

Hack #21. Using Large Amounts of RAM

Hack #22. hdparm: Fine Tune IDE Drive Parameters

2. Revision Control

2.1. Hacks #23-36

Hack #23. Getting Started with RCS

Hack #24. Checking Out a Previous Revision in RCS

Hack #25. Tracking Changes with rcs2log

Hack #26. Getting Started with CVS

26.1. Typical Uses

26.2. Creating a Repository

26.3. Importing a New Module

26.4. Environment Variables

26.5. See Also:

Hack #27. CVS: Checking Out a Module

Hack #28. CVS: Updating Your Working Copy

Hack #29. CVS: Using Tags

Hack #30. CVS: Making Changes to a Module

Hack #31. CVS: Merging Files

Hack #32. CVS: Adding and Removing Files and Directories

32.1. Removing Files

32.2. Removing Directories

Hack #33. CVS: Branching Development

Hack #34. CVS: Watching and Locking Files

Hack #35. CVS: Keeping CVS Secure

35.1. Remote Repositories

35.2. Permissions

35.3. Developer Machines

Hack #36. CVS: Anonymous Repositories

36.1. Creating an Anonymous Repository

36.2. Installing pserver

36.3. Using a Remote pserver

3. Backups

3.1. Hacks #37-44

Hack #37. Backing Up with tar over ssh

37.1. See also:

Hack #38. Using rsync over ssh

38.1. See also:

Hack #39. Archiving with Pax

39.1. Creating Archives

39.2. Expanding Archives

39.3. Interactive Restores

39.4. Recursively Copy a Directory

39.5. Incremental Backups

39.6. Skipping Files on Restore

39.7. See also:

Hack #40. Backing Up Your Boot Sector

40.1. See also:

Hack #41. Keeping Parts of Filesystems in sync with rsync

41.1. Listing: Balance-push.sh

Hack #42. Automated Snapshot-Style Incremental Backups with rsync

42.1. Extensions: Hourly, Daily, and Weekly Snapshots

42.2. Listing: make_snapshot.sh

42.3. Listing: Daily_snapshot_rotate.sh

42.4. Sample Output of ls -l /snapshot/home

42.5. See also:

Hack #43. Working with ISOs and CDR/CDRWs

43.1. See also:

Hack #44. Burning a CD Without Creating an ISO File

4. Networking

4.1. Hacks #45-53

Hack #45. Creating a Firewall from the Command Line of any Server

45.1. See also:

Hack #46. Simple IP Masquerading

46.1. See also:

Hack #47. iptables Tips & Tricks

47.1. Advanced iptables Features

47.2. See also:

Hack #48. Forwarding TCP Ports to Arbitrary Machines

48.1. See also:

Hack #49. Using Custom Chains in iptables

49.1. See also:

Hack #50. Tunneling: IPIP Encapsulation

50.1. See also:

Hack #51. Tunneling: GRE Encapsulation

51.1. See also:

Hack #52. Using vtun over ssh to Circumvent NAT

52.1. See also:

Hack #53. Automatic vtund.conf Generator

53.1. Listing: vtundconf

5. Monitoring

5.1. Hacks #54-65

Hack #54. Steering syslog

54.1. Mark Who?

54.2. Remote Logging

Hack #55. Watching Jobs with watch

55.1. See also:

Hack #56. What's Holding That Port Open?

Hack #57. Checking On Open Files and Sockets with lsof

57.1. See also:

Hack #58. Monitor System Resources with top

58.1. See also:

Hack #59. Constant Load Average Display in the Titlebar

59.1. Listing: tl

Hack #60. Network Monitoring with ngrep

60.1. Listing: go-ogle

60.2. See also:

Hack #61. Scanning Your Own Machines with nmap

61.1. See also:

Hack #62. Disk Age Analysis

62.1. Listing: diskage

Hack #63. Cheap IP Takeover

63.1. Listing: takeover

63.2. See also:

Hack #64. Running ntop for Real-Time Network Stats

64.1. See also:

Hack #65. Monitoring Web Traffic in Real Time with httptop

65.1. Listing: httptop

6. SSH

6.1. Hacks #66-71

Hack #66. Quick Logins with ssh Client Keys

66.1. Security Concerns

66.2. See Also:

Hack #67. Turbo-mode ssh Logins

67.1. See also:

Hack #68. Using ssh-Agent Effectively

Hack #69. Running the ssh-Agent in a GUI

69.1. See Also:

Hack #70. X over ssh

70.1. See also:

Hack #71. Forwarding Ports over ssh

71.1. See also:

7. Scripting

7.1. Hacks #72-75

Hack #72. Get Settled in Quickly with movein.sh

72.1. Listing: movein.sh

72.2. See also:

Hack #73. Global Search and Replace with Perl

73.1. See also:

Hack #74. Mincing Your Data into Arbitrary Chunks (in bash)

74.1. Listing: mince

Hack #75. Colorized Log Analysis in Your Terminal

8. Information Servers

8.1. Hacks #76-100

Hack #76. Running BIND in a chroot Jail

76.1. See also:

Hack #77. Views in BIND 9

77.1. Basic Syntax

77.2. Defining Zones in Views

77.3. Views in Slave Name Servers

Hack #78. Setting Up Caching DNS with Authority for Local Domains

78.1. See also:

Hack #79. Distributing Server Load with Round-Robin DNS

79.1. See also:

Hack #80. Running Your Own Top-Level Domain

Hack #81. Monitoring MySQL Health with mtop

81.1. See also:

Hack #82. Setting Up Replication in MySQL

82.1. See also:

Hack #83. Restoring a Single Table from a Large MySQL Dump

Hack #84. MySQL Server Tuning

84.1. See also:

Hack #85. Using proftpd with a mysql Authentication Source

85.1. See also:

Hack #86. Optimizing glibc, linuxthreads, and the Kernel for a Super MySQL Server

86.1. Step 1: Build glib

86.2. Step 2: The Kernel

86.3. Step 3: Build a New MySQL

86.4. Step 4: Expand the Maximum Filehandles at Boot

Hack #87. Apache Toolbox

87.1. See Also:

Hack #88. Display the Full Filename in Indexes

Hack #89. Quick Configuration Changes with IfDefine

Hack #90. Simplistic Ad Referral Tracking

90.1. Listing: referral-report.pl

Hack #91. Mimicking FTP Servers with Apache

Hack #92. Rotate and compress Apache Server Logs

92.1. Listing: logflume.pl

Hack #93. Generating an SSL cert and Certificate Signing Request

93.1. See also:

Hack #94. Creating Your Own CA

94.1. See also:

Hack #95. Distributing Your CA to Client Browsers

95.1. See also:

Hack #96. Serving multiple sites with the same DocumentRoot

Hack #97. Delivering Content Based on the Query String Using mod_rewrite

97.1. See also:

Hack #98. Using mod_proxy on Apache for Speed

Hack #99. Distributing Load with Apache RewriteMap

99.1. See Also:

Hack #100. Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite

Linux Server Hacks

Rob Flickenger

Editor

Rael Dornfest

Copyright © 2009 O'Reilly Media, Inc.

O'Reilly Media

A Note Regarding Supplemental Files

Supplemental files and examples for this book can be found at http://examples.oreilly.com/9780596004613/. Please use a standard desktop web browser to access these files, as they may not be accessible from all ereader devices.

All code files or examples referenced in the book will be available online. For physical books that ship with an accompanying disc, whenever possible, we’ve posted all CD/DVD content. Note that while we provide as much of the media content as we are able via free download, we are sometimes limited by licensing restrictions. Please direct any questions or concerns to [email protected].

How to Become a Hacker

The Jargon File contains a bunch of definitions of the term "hacker," most having to do with technical adeptness and a delight in solving problems and overcoming limits. If you want to know how to become a hacker, though, only two are really relevant.

There is a community, a shared culture, of expert programmers and networking wizards that traces its history back through decades to the first time-sharing minicomputers and the earliest ARPAnet experiments. The members of this culture originated the term "hacker." Hackers built the Internet. Hackers made the Unix operating system what it is today. Hackers run Usenet. Hackers make the Web work. If you are part of this culture, if you have contributed to it and other people in it know who you are and call you a hacker, you're a hacker.

The hacker mind-set is not confined to this software-hacker culture. There are people who apply the hacker attitude to other things, like electronics or music — actually, you can find it at the highest levels of any science or art. Software hackers recognize these kindred spirits elsewhere and may call them "hackers" too — and some claim that the hacker nature is really independent of the particular medium the hacker works in. But in the rest of this document, we will focus on the skills and attitudes of software hackers, and the traditions of the shared culture that originated the term "hacker."

There is another group of people who loudly call themselves hackers, but aren't. These are people (mainly adolescent males) who get a kick out of breaking into computers and breaking the phone system. Real hackers call these people "crackers" and want nothing to do with them. Real hackers mostly think crackers are lazy, irresponsible, and not very bright — being able to break security doesn't make you a hacker any more than being able to hotwire cars makes you an automotive engineer. Unfortunately, many journalists and writers have been fooled into using the word "hacker" to describe crackers; this irritates real hackers no end.

The basic difference is this: hackers build things, crackers break them.

If you want to be a hacker, keep reading. If you want to be a cracker, go read the alt.2600 newsgroup and get ready to do five to ten in the slammer after finding out you aren't as smart as you think you are. And that's all I'm going to say about crackers.

The Hacker Attitude

Hackers solve problems and build things, and they believe in freedom and voluntary mutual help. To be accepted as a hacker, you have to behave as though you have this kind of attitude yourself. And to behave as though you have the attitude, you have to really believe the attitude.

But if you think of cultivating hacker attitudes as just a way to gain acceptance in the culture, you'll miss the point. Becoming the kind of person who believes these things is important for you — for helping you learn and keeping you motivated. As with all creative arts, the most effective way to become a master is to imitate the mind-set of masters — not just intellectually but emotionally as well.

Or, as the following modern Zen poem has it:

So if you want to be a hacker, repeat the following things until you believe them.

1. The world is full of fascinating problems waiting to be solved.

Being a hacker is a lot of fun, but it's a kind of fun that takes a lot of effort. The effort takes motivation. Successful athletes get their motivation from a kind of physical delight in making their bodies perform, pushing themselves past their own physical limits. Similarly, to be a hacker you have to get a basic thrill from solving problems, sharpening your skills, and exercising your intelligence.

If you aren't the kind of person that feels this way naturally, you'll need to become one in order to make it as a hacker. Otherwise you'll find your hacking energy is zapped by distractions like sex, money, and social approval.

(You also have to develop a kind of faith in your own learning capacity — a belief that even though you may not know all of what you need to solve a problem, if you tackle just a piece of it and learn from that, you'll learn enough to solve the next piece — and so on, until you're done.)

2. No problem should ever have to be solved twice.

Creative brains are a valuable, limited resource. They shouldn't be wasted on re-inventing the wheel when there are so many fascinating new problems waiting out there.

To behave like a hacker, you have to believe that the thinking time of other hackers is precious — so much so that it's almost a moral duty for you to share information, solve problems, and then give the solutions away just so other hackers can solve new problems instead of having to perpetually re-address old ones.

(You don't have to believe that you're obligated to give all your creative product away, though the hackers that do are the ones that get most respect from other hackers. It's consistent with hacker values to sell enough of it to keep you in food and rent and computers. It's fine to use your hacking skills to support a family or even get rich, as long as you don't forget your loyalty to your art and your fellow hackers while doing it.)

3. Boredom and drudgery are evil.

Hackers (and creative people in general) should never be bored or have to drudge at stupid repetitive work, because when this happens it means they aren't doing what only they can do — solve new problems. This wastefulness hurts everybody. Therefore boredom and drudgery are not just unpleasant but actually evil.

To behave like a hacker, you have to believe this enough to want to automate away the boring bits as much as possible, not just for yourself but for everybody else (especially other hackers).

(There is one apparent exception to this. Hackers will sometimes do things that may seem repetitive or boring to an observer as a mind-clearing exercise, or in order to acquire a skill or have some particular kind of experience you can't have otherwise. But this is by choice — nobody who can think should ever be forced into a situation that bores them.)

4. Freedom is good.

Hackers are naturally anti-authoritarian. Anyone who can give you orders can stop you from solving whatever problem you're being fascinated by — and, given the way authoritarian minds work, will generally find some appallingly stupid reason to do so. So the authoritarian attitude has to be fought wherever you find it, lest it smother you and other hackers.

(This isn't the same as fighting all authority. Children need to be guided and criminals restrained. A hacker may agree to accept some kinds of authority in order to get something he wants more than the time he spends following orders. But that's a limited, conscious bargain; the kind of personal surrender authoritarians want is not on offer.)

Authoritarians thrive on censorship and secrecy. And they distrust voluntary cooperation and information-sharing — they only like "cooperation" that they control. So to behave like a hacker, you have to develop an instinctive hostility to censorship, secrecy, and the use of force or deception to compel responsible adults. And you have to be willing to act on that belief.

5. Attitude is no substitute for competence.

To be a hacker, you have to develop some of these attitudes. But copping an attitude alone won't make you a hacker, any more than it will make you a champion athlete or a rock star. Becoming a hacker will take intelligence, practice, dedication, and hard work.

Therefore, you have to learn to distrust attitude and respect competence of every kind. Hackers won't let posers waste their time, but they worship competence — especially competence at hacking, but competence at anything is good. Competence at demanding skills that few can master is especially good, and competence at demanding skills that involve mental acuteness, craft, and concentration is best.

If you revere competence, you'll enjoy developing it in yourself — the hard work and dedication will become a kind of intense play rather than drudgery. That attitude is vital to becoming a hacker.

The complete essay can be found online at http://www.catb.org/~esr/faqs/hacker-howto.html and in an appendix to the "The Cathedral and the Bazaar" book (O'Reilly.)

— Eric S. Raymond^[1]

^[1] Eric S. Raymond is the author of the New Hacker's Dictionary, based on the Jargon File, and the famous "Cathedral and the Bazaar" essay that served as a catalyst for the Open Source movement. The following is an excerpt from his 1996 essay, "What is a hacker?" Raymond argues that hackers are ingenious at solving interesting problems, an idea that is the cornerstone of O'Reilly's new Hacks series.

Preface

A hacker does for love what others would not do for money.

—/usr/games/fortune

The word hack has many connotations. A "good hack" makes the best of the situation of the moment, using whatever resources are at hand. An "ugly hack" approaches the situation in the most obscure and least understandable way, although many "good hacks" may also appear unintelligible to the uninitiated.

The effectiveness of a hack is generally measured by its ability to solve a particular technical problem, inversely proportional to the amount of human effort involved in getting the hack running. Some hacks are scalable and some are even sustainable. The longest running and most generally accepted hacks become standards and cause many more hacks to be invented. A good hack lasts until a better hack comes along.

A hack reveals the interface between the abstract and wonderfully complex mind of the designer, and the indisputable and vulgar experience of human needs. Sometimes, hacks may be ugly and only exist because someone had an itch that needed scratching. To the engineer, a hack is the ultimate expression of the Do-It-Yourself sentiment: no one understands how a hack came to be better than the person who felt compelled to solve the problem in the first place. If a person with a bent for problem solving thinks a given hack is ugly, then they are almost always irresistibly motivated to go one better — and hack the hack, something that we encourage the readers of this book to do.

In the end, even the most capable server, with the most RAM and running the fastest (and most free) operating system on the planet, is still just a fancy back-scratcher fixing the itch of the moment, until a better, faster and cheaper back-scratcher is required.

Where does all of this pseudo-philosophical rambling get you? Hopefully, this background will give you some idea of the mindset that prompted the compiling of this collection of solutions that we call Linux Server Hacks. Some are short and simple, while some are quite complex. All of these hacks are designed to solve a particular technical problem that the designer simply couldn't let go without "scratching." I hope that some of them will be directly applicable to an "itch" or two that you may have felt yourself as a new or experienced administrator of Linux servers.

How This Book is Organized

A competent sysadmin must be a jack-of-all-trades. To be truly effective, you'll need to be able to handle every problem the system throws at you, from power on to halt. To assist you in the time in between, I present this collection of time-saving and novel approaches to daily administrative tasks.

• Server Basics begins by looking at some of the most common sorts of tasks that admins encounter: manipulating the boot process, effectively working with the command line, automating common tasks, watching (and regulating) how system resources are used, and tuning various pieces of the Linux kernel to make everything run more efficiently. This isn't an introduction to system administration but a look at some very effective and non-obvious techniques that even seasoned sysadmins may have overlooked.
• Revision Control gives a crash-course in using two fundamental revision control systems, RCS and CVS. Being able to recall arbitrary previous revisions of configuration files, source code, and documentation is a critical ability that can save your job. Too many professional admins are lacking in revision control basics (preferring instead to make the inevitable, but unsupportable .old or .orig backup). This section will get you up and running quickly, giving you commands and instructions that are succinct and to the point.
• The next section, Backups, looks at quick and easy methods for keeping spare copies of your data. I pay particular attention to network backups, rsync, and working with ISOs. I'll demonstrate some of the enormous flexibility of standard system backup tools and even present one way of implementing regular "snapshot" revisions of a filesystem (without requiring huge amounts of storage).
• Networking is my favorite section of this entire book. The focus isn't on basic functionality and routing, but instead looks at some obscure but insanely useful techniques for making networks behave in unexpected ways. I'll set up various kinds of IP tunnels (both encrypted and otherwise), work with NAT, and show some advanced features that allow for interesting behavior based on all kinds of parameters. Did you ever want to decide what to do with a packet based on its data contents? Take a look at this section.
• Monitoring is an eclectic mix of tips and tools for finding out exactly what your server is up to. It looks at some standard (and some absolutely required "optional") packages that will tell you volumes about who is using what, when, and how on your network. It also looks at a couple of ways to mitigate inevitable service failures and even help detect when naughty people attempt to do not-so-nice things to your network.
• Truly a font of hackery unto itself, the SSH section describes all sorts of nifty uses for ssh, the cryptographically strong (and wonderfully flexible) networking tool. There are a couple of versions of ssh available for Linux, and while many of the examples will work in all versions, they are all tested and known to work with OpenSSH v3.4p1.
• Scripting provides a short digression by looking at a couple of odds and ends that simply couldn't fit on a single command line. These hacks will save you time and will hopefully serve as examples of how to do some nifty things in shell and Perl.
• Information Services presents three major applications for Linux: BIND 9, MySQL, and Apache. This section assumes that you're well beyond basic installation of these packages, and are looking for ways to make them deliver their services faster and more efficiently, without having to do a lot of work yourself. You will see methods for getting your server running quickly, helping it scale to very large installations and behave in all sorts of slick ways that save a lot of configuration and maintenance time.

How to Use This Book

You may find it useful to read this book from cover to cover, as the hacks do build on each other a bit from beginning to end. However, each hack is designed to stand on its own as a particular example of one way to accomplish a particular task. To that end, I have grouped together hacks that fit a particular theme into sections, but I do cross-reference quite a bit between hacks from different sections (and also to more definitive resources on the subject). Don't consider a given section as a cut-and-dried chapter with rigidly defined subject boundaries but more as a convenient way of collecting similar (and yet independent) hacks. You may want to read this book much like the way most people browse web pages online: follow whatever interests you, and if you get lost, follow the links within the piece to find more information.

Conventions Used in This Book

The following is a list of the typographical conventions used in this book:

Italic

Used to indicate new terms, URLs, filenames, file extensions, directories, commands and options, and program names.

Constant Width

Used to show code examples, the contents of files, or the output from commands.

Constant Width Bold

Used in examples and tables to show commands or other text that should be typed literally.

Constant Width Italic

Used in examples and tables to show text that should be replaced with user-supplied values.

The thermometer icons, found next to each hack, indicate the relative complexity of the hack:

beginner

moderate

expert

How to Contact Us

We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors, inaccuracies, bugs, misleading or confusing statements, and typos that you find in this book.

You can write to us at:

To ask technical questions or to comment on the book, send email to:

Visit the web page for Linux Server Hacks to find additional support information, including examples and errata. You can find this page at:

For more information about this book and others, see the O'Reilly web site:

Gotta Hack?

Got a good hack you'd like to share with others? Go to the O'Reilly Hacks web site at:

You'll find book-related resources, sample hacks and new hacks contributed by users. You'll find information about additional books in the Hacks series.

Credits

• Rob Flickenger authored the majority of hacks in this book.
  Rob has worked with Linux since Slackware 3.5. He was previously the system administrator of the O'Reilly Network (an all-Linux shop, naturally) and is the author of Building Wireless Community Networks, also by O'Reilly.
• Rael Dornfest ([Hack #87])
  Rael is a Researcher at the O'Reilly & Associates focusing on technologies just beyond the pale. He assesses, experiments, programs, and writes for the O'Reilly network and O'Reilly publications.
• Seann Herdejurgen ([Hack #47], [Hack #62])
  Seann has been working with UNIX since 1987 and now architects high availability solutions as a senior systems engineer with D-Tech corporation in Dallas, Texas. He holds a Master's degree in Computer Science from Texas A&M University. He may be reached at: http://seann.herdejurgen.com/.
• Dru Lavigne ([Hack #39])
  Dru is an instructor at a private technical college in Kingston, ON where she teaches the fundamentals of TCP/IP networking, routing, and security. Her current hobbies include matching every TCP and UDP port number to its associated application(s) and reading her way through all of the RFCs.
• Cricket Liu ([Hack #77])
  Cricket matriculated at the University of California's Berkeley campus, that great bastion of free speech, unencumbered Unix, and cheap pizza. He worked for a year as Director of DNS Product Management for VeriSign Global Registry Services, and is a co-author of DNS and BIND, also published by O'Reilly & Associates.
• Mike Rubel ([Hack #42])
  Mike (http://www.mikerubel.org) studied Mechanical Engineering at Rutgers University (B.S. 1998) and Aeronautics at Caltech (M.S. 1999), where he is now a graduate student. He has enjoyed using Linux and GNU software for several years in the course of his numerical methods research.
• Jennifer Vesperman (all of the CVS pieces except [Hack #36] were adapted from her online CVS pieces for O'ReillyNet)
  Jenn contributes to open source, as a user, writer and occasional programmer. Her coding experience ranges from the hardware interface to an HDLC card to the human interface of Java GUIs. Jenn is the current coordinator and co-sysadmin for Linuxchix.org.
• Schuyler Erle (contributed code for httptop, mysql-table-restore, balance-push, find-whois, and vtundgen)
  By day, Schuyler is a mild-mannered Internet Systems Developer for O'Reilly & Associates. By night, he crusades for justice and freedom as a free software hacker and community networking activist.
• Morbus Iff ([Hack #89], [Hack #90], [Hack #91])
  Kevin Hemenway, better known as Morbus Iff, is the creator of disobey.com, which bills itself as "content for the discontented." Publisher and developer of more home cooking than you could ever imagine, he'd love to give you a Fry Pan of Intellect upside the head. Politely, of course. And with love.

Acknowledgments

I would like to thank my family and friends for their support and encouragement. Thanks especially to my dad for showing me "proper troubleshooting technique" at such an early age and inevitably setting me on the path of the Hacker before I had even seen a computer.

Of course, this book would be nothing without the excellent contributions of all the phenomenally talented hackers contained herein. But of course, our hacks are built by hacking the shoulders of giants (to horribly mix a metaphor), and it is my sincere hope that you will in turn take what you learn here and go one better, and most importantly, tell everyone just how you did it.

Chapter 1. Server Basics

Hacks #1-22

A running Linux system is a complex interaction of hardware and software where invisible daemons do the user's bidding, carrying out arcane tasks to the beat of the drum of the uncompromising task master called the Linux kernel.

A Linux system can be configured to perform many different kinds of tasks. When running as a desktop machine, the visible portion of Linux spends much of its time controlling a graphical display, painting windows on the screen, and responding to the user's every gesture and command. It must generally be a very flexible (and entertaining) system, where good responsiveness and interactivity are the critical goals.

On the other hand, a Linux server generally is designed to perform a couple of tasks, nearly always involving the squeezing of information down a network connection as quickly as possible. While pretty screen savers and GUI features may be critical to a successful desktop system, the successful Linux server is a high performance appliance that provides access to information as quickly and efficiently as possible. It pulls that information from some sort of storage (like the filesystem, a database, or somewhere else on the network) and delivers that information over the network to whomever requested it, be it a human being connected to a web server, a user sitting in a shell, or over a port to another server entirely.

It is under these circumstances that a system administrator finds their responsibilities lying somewhere between deity and janitor. Ultimately, the sysadmin's job is to provide access to system resources as quickly (and equitably) as possible. This job involves both the ability to design new systems (that may or may not be rooted in solutions that already exist) and the talent (and the stomach) for cleaning up after people who use that system without any concept of what "resource management" really means.

The most successful sysadmins remove themselves from the path of access to system resources and let the machines do all of the work. As a user, you know that your sysadmin is effective when you have the tools that you need to get the job done and you never need to ask your sysadmin for anything. To pull off (that is, to hack) this impossible sounding task requires that the sysadmin anticipate what the users' needs will be and make efficient use of the resources that are available.

To begin with, I'll present ways to optimize Linux to perform only the work that is required to get the job done and not waste cycles doing work that you're not interested in doing. You'll see some examples of how to get the system to do more of the work of maintaining itself and how to make use of some of the more obscure features of the system to make your job easier. Parts of this section (particularly Command Line and Resource Management) include techniques that you may find yourself using every day to help build a picture of how people are using your system and ways that you might improve it.

These hacks assume that you are already familiar with Linux. In particular, you should already have root on a running Linux system available with which to experiment and should be comfortable with working on the system from the command line. You should also have a good working knowledge of networks and standard network services. While I hope that you will find these hacks informative, they are certainly not a good introduction to Linux system administration. For in-depth discussion on good administrative techniques, I highly recommend the Linux Network Administrator's Guide and Essential System Administration, both by O'Reilly and Associates.

The hacks in this chapter are grouped together into the following five categories: Boot Time, Command Line, Automation, Resource Management, and Kernel Tuning.

Boot Time

1. Removing Unnecessary Services
2. Forgoing the Console Login
3. Common Boot Parameters
4. Creating a Persistent Daemon with init

Command Line

1. n>&m: Swap Standard Output and Standard Error
2. Building Complex Command Lines
3. Working with Tricky Files in xargs
4. Immutable Files in ext2/ext3
5. Speeding Up Compiles

Automation

1. At home in your shell environments
2. Finding and eliminating setuid/setgid binaries
3. Make sudo work harder for you
4. Using a Makefile to automate admin tasks
5. Brute forcing your new domain name

Resource Management

1. Playing Hunt the Disk Hog
2. Fun with /proc
3. Manipulating processes symbolically with procps
4. Managing system resources per process
5. Cleaning up after ex-users

Kernel Tuning

1. Eliminating unnecessary drivers from the kernel
2. Using large amounts of RAM
3. hdparm: fine tune IDE drive parameters

Hack #1. Removing Unnecessary Services

Fine tune your server to provide only the services you really want to serve

When you build a server, you are creating a system that should perform its intended function as quickly and efficiently as possible. Just as a paint mixer has no real business being included as an espresso machine attachment, extraneous services can take up resources and, in some cases, cause a real mess that is completely unrelated to what you wanted the server to do in the first place. This is not to say that Linux is incapable of serving as both a top-notch paint mixer and making a good cup of coffee simultaneously — just be sure that this is exactly what you intend before turning your server loose on the world (or rather, turning the world loose on your server).

When building a server, you should continually ask yourself: what do I really need this machine to do? Do I really need FTP services on my web server? Should NFS be running on my DNS server, even if no shares are exported? Do I need the automounter to run if I mount all of my volumes statically?

To get an idea of what your server is up to, simply run a ps ax . If nobody is logged in, this will generally tell you what your server is currently running. You should also see what programs for which your inetd is accepting connections, with either a grep -v ^# /etc/inetd.conf or (more to the point) netstat -lp . The first command will show all uncommented lines in your inetd.conf, while the second (when run as root) will show all of the sockets that are in the LISTEN state, and the programs that are listening on each port. Ideally, you should be able to reduce the output of a ps ax to a page of information or less (barring preforking servers like httpd, of course).

Here are some notorious (and typically unnecessary) services that are enabled by default in many distributions:

portmap, rpc.mountd, rpc.nfsd

These are all part of the NFS subsystem. Are you running an NFS server? Do you need to mount remote NFS shares? Unless you answered yes to either of these questions, you don't need these daemons running. Reclaim the resources that they're taking up and eliminate the potential security risk.

smbd and nmbd

These are the Samba daemons. Do you need to export SMB shares to Windows boxes (or other machines)? If not, then these processes can be safely killed.

automount

The automounter can be handy to bring up network (or local) filesystems on demand, eliminating the need for root privileges when accessing them. This is especially handy on client desktop machines, where a user needs to use removable media (such as CDs or floppies) or to access network resources. But on a dedicated server, the automounter is probably unnecessary. Unless your machine is providing console access or remote network shares, you can kill the automounter (and set up all of your mounts statically, in /etc/fstab).

named

Are you running a name server? You don't need named running if you only need to resolve network names; that's what /etc/resolv.conf and the bind libraries are for. Unless you're running name services for other machines, or are running a caching DNS server (see [Hack #78]), then named isn't needed.

lpd

Do you ever print to this machine? Chances are, if it's serving Internet resources, it shouldn't be accepting print requests anyway. Remove the printer daemon if you aren't planning on using it.

inetd

Do you really need to run any services from inetd? If you have ssh running in standalone mode, and are only running standalone daemons (such as Apache, BIND, MySQL, or ProFTPD) then inetd may be superfluous. In the very least, review which services are being accepted with the grep command grep -v ^# /etc/inetd.conf. If you find that every service can be safely commented out, then why run the daemon? Remove it from the boot process (either by removing it from the system rc's or with a simple chmod -x /usr/sbin/inetd).

telnet, rlogin, rexec, ftp

The remote login, execution, and file transfer functionality of these venerable daemons has largely been supplanted by ssh and scp, their cryptographically secure and tremendously flexible counterparts. Unless you have a really good reason to keep them around, it's a good idea to eliminate support for these on your system. If you really need to support ftp connections, you might try the mod_sql plugin for proftpd (see [Hack #85]).

finger, comsat, chargen, echo, identd

The finger and comsat services made sense in the days of an open Internet, where users were curious but generally well-intentioned. In these days of stealth portscans and remote buffer overflow exploits, running extraneous services that give away information about your server is generally considered a bad idea. The chargen and echo ports were once good for testing network connectivity, but are now too inviting for a random miscreant to fiddle with (and perhaps connect to each other to drive up server load quickly and inexpensively).

Finally, the identd service was once a meaningful and important source of information, providing remote servers with an idea of which users were connecting to their machines. Unfortunately, in these days of local root exploits and desktop Linux machines, installing an identd that (perish the thought!) actually lies about who is connected has become so common that most sites ignore the author information anyway. Since identd is a notoriously shaky source of information, why leave it enabled at all?

To eliminate unnecessary services, first shut them down (either by running service stop in /etc/rc.d/init.d/, removing them from /etc/inetd.conf, or by killing them manually). Then to be sure that they don't start again the next time the machine reboots, remove their entry from /etc/rc.d/*. Once you have your system trimmed down to only the services you intend to serve, reboot the machine and check the process table again.

If you absolutely need to run insecure services on your machine, then you should use tcp wrappers or local firewalling to limit access to only the machines that absolutely need it.

See also:

• [Hack #45]
• [Hack #78]
• [Hack #85]

Hack #2. Forgoing the Console Login

All of the access, none of the passwords

It will happen to you one day. You'll need to work on a machine for a friend or client who has "misplaced" the root password on which you don't have an account.

If you have console access and don't mind rebooting, traditional wisdom beckons you to boot up in single user mode. Naturally, after hitting Control-Alt-Delete, you simply wait for it to POST and then pass the parameter single to the booting kernel. For example, from the LILO prompt:

LILO: linux single

On many systems, this will happily present you with a root shell. But on some systems (notably RedHat), you'll run into the dreaded emergency prompt:

Give root password for maintenance
(or type Control-D for normal startup)

If you knew the root password, you wouldn't be here! If you're lucky, the init script will actually let you hit ^C at this stage and will drop you to a root prompt. But most init processes are "smarter" than that, and trap ^C. What to do? Of course, you could always boot from a rescue disk and reset the password, but suppose you don't have one handy (or that the machine doesn't have a CD-ROM drive).

All is not lost! Rather than risk running into the above mess, let's modify the system with extreme prejudice, right from the start. Again, from the LILO prompt:

LILO: linux init=/bin/bash

What does this do? Rather than start /sbin/init and proceed with the usual /etc/rc.d/* procedure, we're telling the kernel to simply give us a shell. No passwords, no filesystem checks (and for that matter, not much of a starting environment!) but a very quick, shiny new root prompt.

Unfortunately, that's not quite enough to be able to repair your system. The root filesystem will be mounted read-only (since it never got a chance to be checked and remounted read/write). Also, networking will be down, and none of the usual system daemons will be running. You don't want to do anything more complicated than resetting a password (or tweaking a file or two) at a prompt like this. Above all: don't hit ^D or type Exit! Your little shell (plus the kernel) constitutes the entire running Linux system at the moment. So, how can you manipulate the filesystem in this situation, if it is mounted read-only? Try this:

# mount -o remount,rw /

That will force the root filesystem to be remounted read-write. You can now type passwd to change the root password (and if the original admin lost the password, consider the ramifications of giving them access to the new one. If you were the original admin, consider writing it in invisible ink on a post-it note and sticking it to your screen, or stitching it into your underwear, or maybe even taking up another hobby).

Once the password is reset, DO NOT REBOOT. Since there is no init running, there is no process in place for safely taking the system down. The quickest way to shutdown safely is to remount root again:

# mount -o remount,ro /

With the root partition readonly, you can confidently hit the Reset button, bring it up in single-user mode, and begin your actual work.

Hack #3. Common Boot Parameters

Manipulate kernel parameters at boot time

As we saw in [Hack #2], it is possible to pass parameters to the kernel at the LILO prompt allowing you to change the program that is first called when the system boots. Changing init (with the init=/bin/bash line) is just one of many useful options that can be set at boot time. Here are more common boot parameters:

single

Boots up in single user mode.

root=

Changes the device that is mounted as /. For example:

root=/dev/sdc4

will boot from the fourth partition on the third scsi disk (instead of whatever your boot loader has defined as the default).

hdX=

Adjusts IDE drive geometry. This is useful if your BIOS reports incorrect information:

hda=3649,255,63 hdd=cdrom

This defines the master/primary IDE drive as a 30GB hard drive in LBA mode, and the slave/secondary IDE drive as a CD-ROM.

console=

Defines a serial port console on kernels with serial console support. For example:

console=ttyS0,19200n81

Here we're directing the kernel to log boot messages to ttyS0 (the first serial port), at 19200 baud, no parity, 8 data bits, 1 stop bit. Note that to get an actual serial console (that you can log in on), you'll need to add a line to /etc/inittab that looks something like this:

s1:12345:respawn:/sbin/agetty 19200 ttyS0 vt100

nosmp

Disables SMP on a kernel so enabled. This can help if you suspect kernel trouble on a multiprocessor system.

mem=

Defines the total amount of available system memory. See [Hack #21].

ro

Mounts the / partition read-only (this is typically the default, and is remounted read-write after fsck runs).

rw

Mounts the / partition read-write. This is generally a bad idea, unless you're also running the init hack. Pass your init line along with rw, like this:

init=/bin/bash rw

to eliminate the need for all of that silly mount -o remount,rw / stuff in [Hack #2]. Congratulations, now you've hacked a hack.

You can also pass parameters for SCSI controllers, IDE devices, sound cards, and just about any other device driver. Every driver is different, and typically allows for setting IRQs, base addresses, parity, speeds, options for auto-probing, and more. Consult your online documentation for the excruciating details.

See also:

• man bootparam
• /usr/src/linux/Documentation/*

Hack #4. Creating a Persistent Daemon with init

Make sure that your process stays up, no matter what

There are a number of scripts that will automatically restart a process if it exits unexpectedly. Perhaps the simplest is something like:

$ while : ; do echo "Run some code here..."; sleep 1; done

If you run a foreground process in place of that echo line, then the process is always guaranteed to be running (or, at least, it will try to run). The : simply makes the while always execute (and is more efficient than running /bin/true, as it doesn't have to spawn an external command on each iteration). Definitely do not run a background process in place of the echo, unless you enjoy filling up your process table (as the while will then spawn your command as many times as it can, one every second). But as far as cool hacks go, the while approach is fairly lacking in functionality.

What happens if your command runs into an abnormal condition? If it exits immediately, then it will retry every second, without giving any indication that there is a problem (unless the process has its own logging system or uses syslog). It might make sense to have something watch the process, and stop trying to respawn it if it returns too quickly after a few tries.

There is a utility already present on every Linux system that will do this automatically for you: init . The same program that brings up the system and sets up your terminals is perfectly suited for making sure that programs are always running. In fact, that is its primary job.

You can add arbitrary lines to /etc/inittab specifying programs you'd like init to watch for you:

zz:12345:respawn:/usr/local/sbin/my_daemon

The inittab line consists of an arbitrary (but unique) two character identification string (in this case, zz), followed by the runlevels that this program should be run in, then the respawn keyword, and finally the full path to the command. In the above example, as long as my_daemon is configured to run in the foreground, init will respawn another copy whenever it exits. After making changes to inittab, be sure to send a HUP to init so it will reload its configuration. One quick way to do this is:

# kill -HUP 1

If the command respawns too quickly, then init will postpone execution for a while, to keep it from tying up too many resources. For example:

zz:12345:respawn:/bin/touch /tmp/timestamp

This will cause the file /tmp/timestamp to be touched several times a second, until init decides that enough is enough. You should see this message in /var/log/messages almost immediately:

Sep 8 11:28:23 catlin init: Id "zz" respawning too fast: disabled for 5 minutes

In five minutes, init will try to run the command again, and if it is still respawning too quickly, it will disable it again.

Obviously, this method is fine for commands that need to run as root, but what if you want your auto-respawning process to run as some other user? That's no problem: use sudo:

zz:12345:respawn:/usr/bin/sudo -u rob /bin/touch /tmp/timestamp

Now that touch will run as rob, not as root. If you're trying these commands as you read this, be sure to remove the existing /tmp/timestamp before trying this sudo line. After sending a HUP to init, take a look at the timestamp file:

rob@catlin:~# ls -al /tmp/timestamp 
-rw-r--r-- 1 rob users 0 Sep 8 11:28 /tmp/timestamp

The two drawbacks to using init to run arbitrary daemons are that you need to comment out the line in inittab if you need to bring the daemon down (since it will just respawn if you kill it) and that only root can add entries to inittab. But for keeping a process running that simply must stay up no matter what, init does a great job.

See also:

• [Hack #12]

Hack #5. n>&m: Swap Standard Output and Standard Error

Direct standard out and standard error to wherever you need them to go

By default, a command's standard error goes to your terminal. The standard output goes to the terminal or is redirected somewhere (to a file, down a pipe, into backquotes).

Sometimes you want the opposite. For instance, you may need to send a command's standard output to the screen and grab the error messages (standard error) with backquotes. Or, you might want to send a command's standard output to a file and the standard error down a pipe to an error-processing command. Here's how to do that in the Bourne shell. (The C shell can't do this.)

File descriptors 0, 1, and 2 are the standard input, standard output, and standard error, respectively. Without redirection, they're all associated with the terminal file /dev/tty. It's easy to redirect any descriptor to any file — if you know the filename. For instance, to redirect file descriptor to errfile, type:

$ command
             2> errfile

You know that a pipe and backquotes also redirect the standard output:

$ command
             | ...\
$ var=`
            command`

But there's no filename associated with the pipe or backquotes, so you can't use the 2> redirection. You need to rearrange the file descriptors without knowing the file (or whatever) that they're associated with. Here's how.

Let's start slowly. By sending both standard output and standard error to the pipe or backquotes, the Bourne shell operator n>&m rearranges the files and file descriptors. It says "make file descriptor n point to the same file as file descriptor m." Let's use that operator on the previous example. We'll send standard error to the same place standard output is going:

$ command
             2>&1 | ...
$ var=`
            command
             2>&1`

In both those examples, 2>&1 means "send standard error (file descriptor 2) to the same place standard output (file descriptor 1) is going." Simple, eh?

You can use more than one of those n>&m operators. The shell reads them left-to-right before it executes the command.

"Oh!" you might say, "To swap standard output and standard error — make stderr go down a pipe and stdout go to the screen — I could do this!"

$ command
             2>&1 1>&2 | ... 
            (wrong...)

Sorry, Charlie. When the shell sees 2>&1 1>&2, the shell first does 2>&1. You've seen that before — it makes file descriptor 2 (stderr) go the same place as file descriptor 1 (stdout). Then, the shell does 1>&2. It makes stdout (1) go the same place as stderr (2), but stderr is already going the same place as stdout, down the pipe.

This is one place that the other file descriptors, 3 through 9, come in handy. They normally aren't used. You can use one of them as a "holding place" to remember where another file descriptor "pointed." For example, one way to read the operator 3>&2 is "make 3 point the same place as 2". After you use 3>&2 to grab the location of 2, you can make 2 point somewhere else. Then, make 1 point to where 2 used to (where 3 points now).

The command line you want is one of these:

$ command
             3>&2 2>&1 1>&3 | ...
$ var=`
            command
             3>&2 2>&1 1>&3`

Open files are automatically closed when a process exits. But it's safer to close the files yourself as soon as you're done with them. That way, if you forget and use the same descriptor later for something else (for instance, use F.D. 3 to redirect some other command, or a subprocess uses F.D. 3), you won't run into conflicts. Use m<&- to close input file descriptor m and m>&- to close output file descriptor m. If you need to close standard input, use <&- ; >&- will close standard output.

Hack #6. Building Complex Command Lines

Build simple commands into full-fledged paragraphs for complex (but meaningful) reports

Studying Linux (or indeed any Unix) is much like studying a foreign language. At some magical point in the course of one's studies, halting monosyllabic mutterings begin to meld together into coherent, often used phrases. Eventually, one finds himself pouring out entire sentences and paragraphs of the Unix Mother Tongue, with one's mind entirely on the problem at hand (and not on the syntax of any particular command). But just as high school foreign language students spend much of their time asking for directions to the toilet and figuring out just what the dative case really is, the path to Linux command-line fluency must begin with the first timidly spoken magic words.

Your shell is very forgiving, and will patiently (and repeatedly) listen to your every utterance, until you get it just right. Any command can serve as the input for any other, making for some very interesting Unix "sentences." When armed with the handy (and probably over-used) up arrow, it is possible to chain together commands with slight tweaks over many tries to achieve some very complex behavior.

For example, suppose that you're given the task of finding out why a web server is throwing a bunch of errors over time. If you type less error_log, you see that there are many "soft errors" relating to missing (or badly linked) graphics:

[Tue Aug 27 00:22:38 2002] [error] [client 17.136.12.171] File does not
exist: /htdocs/images/spacer.gif
[Tue Aug 27 00:31:14 2002] [error] [client 95.168.19.34] File does not
exist: /htdocs/image/trans.gif
[Tue Aug 27 00:36:57 2002] [error] [client 2.188.2.75] File does not
exist: /htdocs/images/linux/arrows-linux-back.gif
[Tue Aug 27 00:40:37 2002] [error] [client 2.188.2.75] File does not
exist: /htdocs/images/linux/arrows-linux-back.gif
[Tue Aug 27 00:41:43 2002] [error] [client 6.93.4.85] File does not
exist: /htdocs/images/linux/hub-linux.jpg
[Tue Aug 27 00:41:44 2002] [error] [client 6.93.4.85] File does not
exist: /htdocs/images/xml/hub-xml.jpg
[Tue Aug 27 00:42:13 2002] [error] [client 6.93.4.85] File does not
exist: /htdocs/images/linux/hub-linux.jpg
[Tue Aug 27 00:42:13 2002] [error] [client 6.93.4.85] File does not
exist: /htdocs/images/xml/hub-xml.jpg

and so on. Running a logging package (like analog) reports exactly how many errors you have seen in a day but few other details (which is how you were probably alerted to the problem in the first place). Looking at the logfile directly gives you every excruciating detail but is entirely too much information to process effectively.

Let's start simple. Are there any errors other than missing files? First we'll need to know how many errors we've had today:

$ wc -l error_log
1265 error_log

And how many were due to File does not exist errors?

$ grep "File does not exist:" error_log | wc -l
1265 error_log

That's a good start. At least we know that we're not seeing permission problems or errors in anything that generates dynamic content (like cgi scripts.) If every error is due to missing files (or typos in our html that point to the wrong file) then it's probably not a big problem. Let's generate a list of the filenames of all bad requests. Hit the up arrow and delete that wc -l:

$ grep "File does not exist:" error_log | awk '{print $13}' | less

That's the sort of thing that we want (the 13th field, just the filename), but hang on a second. The same couple of files are repeated many, many times. Sure, we could email this to the web team (all whopping 1265 lines of it), but I'm sure they wouldn't appreciate the extraneous spam. Printing each file exactly once is easy:

$ grep "File does not exist:" error_log | awk '{print $13}' | sort | uniq |
less

This is much more reasonable (substitute a wc -l for that less to see just how many unique files have been listed as missing). But that still doesn't really solve the problem. Maybe one of those files was requested once, but another was requested several hundred times. Naturally, if there is a link somewhere with a typo in it, we would see many requests for the same "missing" file. But the previous line doesn't give any indication of which files are requested most. This isn't a problem for bash; let's try out a command line for loop.

$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
| uniq`; do \
echo -n "$x : "; grep $x error_log | wc -l; done

We need those backticks ( `) to actually execute our entire command from the previous example and feed the output of it to a for loop. On each iteration through the loop, the $x variable is set to the next line of output of our original command (that is, the next unique filename reported as missing). We then grep for that filename in the error_log, and count how many times we see it. The echo at the end just prints it in a somewhat nice report format.

I call it a somewhat nice report because not only is it full of single hit errors (which we probably don't care about), the output is very jagged, and it isn't even sorted! Let's sort it numerically, with the biggest hits at the top, numbers on the left, and only show the top 20 most requested "missing" files:

$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
| uniq`; do \
grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
head -20

That's much better and not even much more typing than the last try. We need the tr to eliminate the trailing newline at the end of wc's output (why it doesn't have a switch to do this, I'll never know). Your output should look something like this:

595 : /htdocs/images/pixel-onlamp.gif.gif
156 : /htdocs/image/trans.gif
139 : /htdocs/images/linux/arrows-linux-back.gif
68 : /htdocs/pub/a/onjava/javacook/images/spacer.gif
50 : /htdocs/javascript/2001/03/23/examples/target.gif

From this report, it's very simple to see that almost half of our errors are due to a typo on a popular web page somewhere (note the repeated .gif.gif in the first line). The second is probably also a typo (should be images/, not image/). The rest are for the web team to figure out:

$ ( echo "Here's a report of the top 20 'missing' files in the error_log.";
echo; \
for x in `grep "File does not exist" error_log | awk '{print $13}' | sort |
uniq`; do \
grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
head -20 )\
| mail -s "Missing file report" [email protected]

and maybe one hardcopy for the weekly development meeting:

$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
| uniq`; do \
grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
head -20 \
| enscript

Hacking the Hack

Once you get used to chunking groups of commands together, you can chain their outputs together indefinitely, creating any sort of report you like out of a live data stream. Naturally, if you find yourself doing a particular task regularly, you might want to consider turning it into a shell script of its own (or even reimplementing it in Perl or Python for efficiency's sake, as every | in a command means that you've spawned yet another program). On a modern (and unloaded) machine, you'll hardly notice the difference, but it's considered good form to clean up the solution once you've hacked it out. And on the command line, there's plenty of room to hack.

Hack #7. Working with Tricky Files in xargs

Deal with many files containing spaces or other strange characters

When you have a number of files containing spaces, parentheses, and other "forbidden" characters, dealing with them can be daunting. This is a problem that seems to come up frequently, with the recent explosive popularity of digital music. Luckily, tab completion in bash makes it simple to handle one file at a time. For example:

rob@catlin:~/Music$ ls
Hallucinogen - The Lone Deranger
Misc - Pure Disco
rob@catlin:~/Music$ rm -rf Misc[TAB]
rob@catlin:~/Music$ rm -rf Misc\ -\ Pure\ Disco/

Hitting the Tab key for [TAB] above replaces the command line with the line below it, properly escaping any special characters contained in the file. That's fine for one file at a time, but what if we want to do a massive transformation (say, renaming a bunch of mp3s to include an album name)? Take a look at this:

rob@catlin:~/Music$ cd Hall[TAB]
rob@catlin:~/Music$ cd Hallucinogen\ -\ The\ Lone\ Deranger/
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ ls
Hallucinogen - 01 - Demention.mp3
Hallucinogen - 02 - Snakey Shaker.mp3
Hallucinogen - 03 - Trancespotter.mp3
Hallucinogen - 04 - Horrorgram.mp3
Hallucinogen - 05 - Snarling (Remix).mp3
Hallucinogen - 06 - Gamma Goblins Pt. 2.mp3
Hallucinogen - 07 - Deranger.mp3
Hallucinogen - 08 - Jiggle of the Sphinx.mp3
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$

When attempting to manipulate many files at once, things get tricky. Many system utilities break on whitespace (yielding many more chunks than you intended) and will completely fall apart if you throw a ) or a { at them. What we need is a delimiter that is guaranteed never to show up in a filename, and break on that instead.

Fortunately, the xargs utility will break on NULL characters, if you ask it to nicely. Take a look at this script:

Listing: albumize

#!/bin/sh

if [ -z "$ALBUM" ]; then
echo 'You must set the ALBUM name first (eg. export ALBUM="Greatest Hits")'
exit 1
fi

for x in *; do
echo -n $x; echo -ne '\000'
echo -n `echo $x|cut -f 1 -d '-'`
echo -n " - $ALBUM - "
echo -n `echo $x|cut -f 2- -d '-'`; echo -ne '\000'
done | xargs -0 -n2 mv

We're actually doing two tricky things here. First, we're building a list consisting of the original filename followed by the name to which we'd like to mv it, separated by NULL characters, for all files in the current directory. We then feed that entire list to an xargs with two switches: -0 tells it to break on NULLs (instead of newlines or whitespace), and -n2 tells it to take two arguments at a time on each pass, and feed them to our command (mv).

Save the script as ~/bin/albumize. Before you run it, set the $ALBUM environment variable to the name that you'd like injected into the filename just after the first -. Here's a trial run:

rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ export ALBUM="The Lone Deranger"
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ albumize
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ ls
Hallucinogen - The Lone Deranger - 01 - Demention.mp3
Hallucinogen - The Lone Deranger - 02 - Snakey Shaker.mp3
Hallucinogen - The Lone Deranger - 03 - Trancespotter.mp3
Hallucinogen - The Lone Deranger - 04 - Horrorgram.mp3
Hallucinogen - The Lone Deranger - 05 - Snarling (Remix).mp3
Hallucinogen - The Lone Deranger - 06 - Gamma Goblins Pt. 2.mp3
Hallucinogen - The Lone Deranger - 07 - Deranger.mp3
Hallucinogen - The Lone Deranger - 08 - Jiggle of the Sphinx.mp3
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$

What if you would like to remove the album name again? Try this one, and call it ~/bin/dealbumize:

#!/bin/sh

for x in *; do
echo -n $x; echo -ne '\000'
echo -n `echo $x|cut -f 1 -d '-'`; echo -n ' - '
echo -n `echo $x|cut -f 3- -d '-'`; echo -ne '\000'
done | xargs -0 -n2 mv

and simply run it (no $ALBUM required):

rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ dealbumize
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ ls
Hallucinogen - 01 - Demention.mp3
Hallucinogen - 02 - Snakey Shaker.mp3
Hallucinogen - 03 - Trancespotter.mp3
Hallucinogen - 04 - Horrorgram.mp3
Hallucinogen - 05 - Snarling (Remix).mp3
Hallucinogen - 06 - Gamma Goblins Pt. 2.mp3
Hallucinogen - 07 - Deranger.mp3
Hallucinogen - 08 - Jiggle of the Sphinx.mp3
rob@catlin:~/Music/Hallucinogen - The Lone Deranger$

The -0 switch is also popular to team up with the -print0 option of find (which, naturally, prints matching filenames separated by NULLs instead of newlines). With find and xargs on a pipeline, you can do anything you like to any number of files, without ever running into the dreaded Argument list too long error:

rob@catlin:~/Pit of too many files$ ls *
bash: /bin/ls: Argument list too long

A find/xargs combo makes quick work of these files, no matter what they're called:

rob@catlin:/Pit of too many files$ find -type f -print0 | xargs -0 ls

To delete them, just replace that trailing ls with an rm, and away you go.

Hack #8. Immutable Files in ext2/ext3

Create files that even root can't manipulate

Here's a puzzle for you. Suppose we're cleaning up /tmp, and run into some trouble:

root@catlin:/tmp# rm -rf junk/
rm: cannot unlink `junk/stubborn.txt': Operation not permitted
rm: cannot remove directory `junk': Directory not empty
root@catlin:/tmp# cd junk/
root@catlin:/tmp/junk# ls -al
total 40
drwxr-xr-x 2 root root 4096 Sep 4 14:45 ./
drwxrwxrwt 13 root root 4096 Sep 4 14:45 ../
-rw-r--r-- 1 root root 29798 Sep 4 14:43 stubborn.txt
root@catlin:/tmp/junk# rm ./stubborn.txt 
rm: remove write-protected file `./stubborn.txt'? y
rm: cannot unlink `./stubborn.txt': Operation not permitted

What's going on? Are we root or aren't we? Let's try emptying the file instead of deleting it:

root@catlin:/tmp/junk# cp /dev/null stubborn.txt 
cp: cannot create regular file `stubborn.txt': Permission denied
root@catlin:/tmp/junk# > stubborn.txt 
bash: stubborn.txt: Permission denied

Well, /tmp certainly isn't mounted read-only. What is going on?

In the ext2 and ext3 filesystems, there are a number of additional file attributes that are available beyond the standard bits accessible through chmod. If you haven't seen it already, take a look at the manpages for chattr and its companion, lsattr .

One of the very useful new attributes is -i, the immutable flag. With this bit set, attempts to unlink, rename, overwrite, or append to the file are forbidden. Even making a hard link is denied (so you can't make a hard link, then edit the link). And having root privileges makes no difference when immutable is in effect:

root@catlin:/tmp/junk# ln stubborn.txt another.txt
ln: creating hard link `another.txt' to `stubborn.txt': Operation not permitted

To view the supplementary ext flags that are in force on a file, use lsattr:

root@catlin:/tmp/junk# lsattr
---i--------- ./stubborn.txt

and to set flags a la chmod, use chattr:

root@catlin:/tmp/junk# chattr -i stubborn.txt 
root@catlin:/tmp/junk# rm stubborn.txt 
root@catlin:/tmp/junk#

This could be terribly useful for adding an extra security step on files you know you'll never want to change (say, /etc/rc.d/* or various configuration files.) While little will help you on a box that has been r00ted, immutable files probably aren't vulnerable to simple overwrite attacks from other processes, even if they are owned by root.

There are hooks for adding compression, security deletes, undeletability, synchronous writes, and a couple of other useful attributes. As of this writing, many of the additional attributes aren't implemented yet, but keep watching for new developments on the ext filesystem.

Hack #9. Speeding Up Compiles

Make sure you're keeping all processors busy with parallel builds

If you're running a multiprocessor system (SMP) with a moderate amount of RAM, you can usually see significant benefits by performing a parallel make when building code. Compared to doing serial builds when running make (as is the default), a parallel build is a vast improvement.

To tell make to allow more than one child at a time while building, use the -j switch:

rob@mouse:~/linux$ make -j4; make -j4 modules

Some projects aren't designed to handle parallel builds and can get confused if parts of the project are built before their parent dependencies have completed. If you run into build errors, it is safest to just start from scratch this time without the -j switch.

By way of comparison, here are some sample timings. They were performed on an otherwise unloaded dual PIII/600 with 1GB RAM. Each time I built a bzImage for Linux 2.4.19 (redirecting STDOUT to /dev/null), and removed the source tree before starting the next test.

            time make bzImage:

real 7m1.640s
user 6m44.710s
sys 0m25.260s

time make -j2 bzImage:

real 3m43.126s
user 6m48.080s
sys 0m26.420s

time make -j4 bzImage:

real 3m37.687s
user 6m44.980s
sys 0m26.350s

time make -j10 bzImage:

real 3m46.060s
user 6m53.970s
sys 0m27.240s

As you can see, there is a significant improvement just by adding the -j2 switch. We dropped from 7 minutes to 3 minutes and 43 seconds of actual time. Increasing to -j4 saved us about five more seconds, but jumping all the way to -j10 actually hurt performance by a few seconds. Notice how user and system seconds are virtually the same across all four runs. In the end, you need to shovel the same sized pile of bits, but -j on a multi-processor machine simply lets you spread it around to more people with shovels.

Of course, bits all eventually end up in the bit bucket anyway. But hey, if nothing else, performance timings are a great way to keep your cage warm.

Hack #10. At Home in Your Shell Environment

Make bash more comfortable through environment variables

Consulting a manpage for bash can be a daunting read, especially if you're not precisely sure what you're looking for. But when you have the time to devote to it, the manpage for bash is well worth the read. This is a shell just oozing with all sorts of arcane (but wonderfully useful) features, most of which are simply disabled by default.

Let's start by looking at some useful environment variables, and some useful values to which to set them:

export PS1=`echo -ne "\033[0;34m\u@\h:\033[0;36m\w\033[0;34m\$\033[0;37m "`

As you probably know, the PS1 variable sets the default system prompt, and automatically interprets escape sequences such as \u (for username) and \w (for the current working directory.) As you may not know, it is possible to encode ANSI escape sequences in your shell prompt, to give your prompt a colorized appearance. We wrap the whole string in backticks (` ) in order to get echo to generate the magic ASCII escape character. This is executed once, and the result is stored in PS1. Let's look at that line again, with boldface around everything that isn't an ANSI code:

export PS1=`echo -ne "\033[0;34m\u@\h:\033[0;36m\w\033[0;34m\$\033[0;37m "`

You should recognize the familiar \u@\h:\w\$prompt that we've all grown to know and love. By changing the numbers just after each semicolon, you can set the colors of each part of the prompt to your heart's content.

Along the same lines, here's a handy command that is run just before bash gives you a prompt:

export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'

(We don't need backticks for this one, as bash is expecting it to contain an actual command, not a string.) This time, the escape sequence is the magic string that manipulates the titlebar on most terminal windows (such as xterm, rxvt, eterm, gnometerm, etc.). Anything after the semicolon and before the \007 gets printed to your titlebar every time you get a new prompt. In this case, we're displaying your username, the host you're logged into, and the current working directory. This is quite handy for being able to tell at a glance (or even while within vim) to which machine you're logged in, and to what directory you're about to save your file. See [Hack #59] if you'd like to update your titlebar in real time instead of at every new bash prompt.

Have you ever accidentally hit ^D too many times in a row, only to find yourself logged out? You can tell bash to ignore as many consecutive ^D hits as you like:

export IGNOREEOF=2

This makes bash follow the Snark rule ("What I tell you three times is true") and only log you out if you hit ^D three times in a row. If that's too few for you, feel free to set it to 101 and bash will obligingly keep count for you.

Having a directory just off of your home that lies in your path can be extremely useful (for keeping scripts, symlinks, and other random pieces of code.) A traditional place to keep this directory is in bin underneath your home directory. If you use the ~ expansion facility in bash, like this:

export PATH=$PATH:~/bin

then the path will always be set properly, even if your home directory ever gets moved (or if you decide you want to use this same line on multiple machines with potentially different home directories — as in movein.sh). See [Hack #72].

Did you know that just as commands are searched for in the PATH variable (and manpages are searched for in the MANPATH variable), directories are likewise searched for in the CDPATH variable every time you issue a cd? By default, it is only set to ".", but can be set to anything you like:

export CDPATH=.:~

This will make cd search not only the current directory, but also your home directory for the directory you try to change to. For example:

rob@caligula:~$ ls
bin/ devel/ incoming/ mail/ test/ stuff.txt
rob@caligula:~$ cd /usr/local/bin
rob@caligula:/usr/local/bin$ cd mail
bash: cd: mail: No such file or directory
rob@caligula:/usr/local/bin$ export CDPATH=.:~
rob@caligula:/usr/local/bin$ cd mail
/home/rob/mail
rob@caligula:~/mail$

You can put as many paths as you like to search for in CDPATH, separating each with a : (just as with the PATH and MANPATH variables.)

We all know about the up arrow and the history command. But what happens if you accidentally type something sensitive on the command line? Suppose you slip while typing and accidentally type a password where you meant to type a command. This accident will faithfully get recorded to your ~/.bash_history file when you logout, where another unscrupulous user might happen to find it. Editing your .bash_history manually won't fix the problem, as the file gets rewritten each time you log out.

To clear out your history quickly, try this from the command line:

export HISTSIZE=0

This completely clears out the current bash history and will write an empty .bash_history on logout. When you log back in, your history will start over from scratch but will otherwise work just as before. From now on, try to be more careful!

Do you have a problem with people logging into a machine, then disconnecting their laptop and going home without logging back out again? If you've ever run a w and seen a bunch of idle users who have been logged in for several days, try setting this in their environment:

export TMOUT=600

The TMOUT variable specifies the number of seconds that bash will wait for input on a command line before logging the user out automatically. This won't help if your users are sitting in a vi window but will alleviate the problem of users just sitting at an idle shell. Ten minutes might be a little short for some users, but kindly remind them that if they don't like the system default, they are free to reset the variable themselves.

This brings up an interesting point: exactly where do you go to make any of these environment changes permanent? There are several files that bash consults when starting up, depending on whether the shell was called at login or from within another shell.

From bash(1), on login shells:

...it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable . . . . When a login shell exits, bash reads and executes commands from the file ~/.bash_logout , if it exists.

For all other shells:

bash reads and executes commands from ~/.bashrc , if that file exists.

For the full definition of what constitutes a login shell (and for a whole bunch of information about the environment you work in every day), consult bash(1).

Hack #11. Finding and Eliminating setuid/setgid Binaries

Eliminate potential root exploits before they have a chance to happen

While running Linux as a server, one guiding principle that has served me well is to continually ask, "what am I trying to achieve?" Does it make sense for a web server to have the printing subsystem installed? Should a system with no console have gpm installed? Usually, extra software packages just take up unnecessary disk space, but in the case of setuid or setgid binaries, the situation could be far worse.

While distribution maintainers work very hard to ensure that all known exploits for setuid and setgid binaries have been removed, it seems that a few new unexpected exploits come out every month or two. Especially if your server has more shell users than yourself, you should regularly audit the setuid and setgid binaries on your system. Chances are you'll be surprised at just how many you'll find.

Here's one command for finding all of the files with a setuid or setgid bit set:

root@catlin:~# find / -perm +6000 -type f -exec ls -ld {} \; > setuid.txt &

This will create a file called setuid.txt that contains the details of all of the matching files present on your system. It is a very good idea to look through this list, and remove the s bits of any tools that you don't use.

Let's look through what we might find on a typical system:

-rws--x--x 1 root bin 35248 May 30 2001 /usr/bin/at
-rws--x--x 1 root bin 10592 May 30 2001 /usr/bin/crontab

Not much surprise here. at and crontab need root privileges in order to change to the user that requested the at job or cron job. If you're paranoid, and you don't use these facilities, then you could remove the setuid bits with:

# chmod a-s /usr/bin/{at,crontab}

Generally speaking, it's a bad idea to disable cron (as so many systems depend on timed job execution). But when was the last time you used at? Do your users even know what it's for? Personally, I find at a nice shortcut to setting up a full-blown cron job, and wouldn't like to part with it. But if there is no call for it on your particular system, you should consider defanging it. With the setuid bit removed, the commands will no longer be available to regular users but will still work fine as root.

-rws--x--x 1 root bin 11244 Apr 15 2001 /usr/bin/disable-paste

This is part of the gpm package (a mouse driver for the Linux console). Do you have a mouse attached to the console of this machine? Do you use it in text mode? If not, then why leave a setuid root binary in place that will never even be called?

-r-s--s--x 1 root lp 14632 Jun 18 2001 /usr/bin/lpq
-r-s--s--x 1 root lp 15788 Jun 18 2001 /usr/bin/lpr
-r-s--s--x 1 root lp 15456 Jun 18 2001 /usr/bin/lprm
-r-xr-s--x 1 root lp 23772 Jun 18 2001 /usr/sbin/lpc

These are all part of the printing subsystem. Does this machine actually use lp to print?

-rws--x--x 1 root bin 33760 Jun 18 2000 /usr/bin/chage
-rws--x--x 1 root bin 29572 Jun 18 2000 /usr/bin/chfn
-rws--x--x 1 root bin 27188 Jun 18 2000 /usr/bin/chsh
-rws--x--x 1 root bin 35620 Jun 18 2000 /usr/bin/passwd

These are all necessary for users to be able to set their passwords, shell, and finger information. Does your site use finger information at all? If, like most sites, you have a roster somewhere else (probably on the web) that isn't kept in sync with user's GECOS field, then this information is generally useless (except for the user's "real" name, which is still used in some email clients). Do you really need to allow users to change this information on their own, without admin intervention?

-r-xr-sr-x 1 root tty 9768 Jun 21 2001 /usr/bin/wall
-r-xr-sr-x 1 root tty 8504 Jun 21 2001 /usr/bin/write

Both wall and write need to be setgid tty to write to other user's terminals. This is generally a safe operation, but can be abused by miscreants who like to write bad data (or lots of bad data) to other user's terminals. If you don't need to provide this functionality to other users, why not disable the setgid bit? If you do, root will still be able to send walls and writes (for example, when a message is sent by shutdown when rebooting the system).

-rwsr-xr-x 1 root bin 14204 Jun 3 2001 /usr/bin/rcp
-rwsr-xr-x 1 root bin 10524 Jun 3 2001 /usr/bin/rlogin
-r-sr-xr-x 1 root bin 7956 Jun 3 2001 /usr/bin/rsh

The r* commands are left from an age (perhaps not so long ago), before the days of ssh . Do you need to provide the r commands to your users? Is there anything that ssh and scp can't do that you absolutely need rsh and rcp for? More than likely, once you've worked with ssh for a while, you'll never miss the r commands, and can safely remove the potential ticking time bomb of a disused setuid rsh installation. If you're looking for interesting things to do with ssh, check out any of the ssh hacks elsewhere in this book.

-r-sr-xr-x 1 root bin 10200 Jun 3 2001 /usr/bin/traceroute
-r-sr-xr-x 1 root bin 15004 Jun 3 2001 /bin/ping

The traceroute and ping commands need the setuid root bit to be able to create ICMP packets. If you want your users to be able to run these network diagnostic tools, then they'll need to be setuid root. Otherwise, remove setuid and then only root will be able to run ping and traceroute.

-r-sr-sr-- 1 uucp uucp 83344 Feb 10 2001 /usr/bin/uucp
-r-sr-sr-- 1 uucp uucp 36172 Feb 10 2001 /usr/bin/uuname
-r-sr-sr-- 1 uucp uucp 93532 Feb 10 2001 /usr/bin/uustat
-r-sr-sr-- 1 uucp uucp 85348 Feb 10 2001 /usr/bin/uux
-r-sr-sr-- 1 uucp uucp 65492 Feb 10 2001 /usr/lib/uucp/uuchk
-r-sr-sr-- 1 uucp uucp 213832 Feb 10 2001 /usr/lib/uucp/uucico
-r-sr-sr-- 1 uucp uucp 70748 Feb 10 2001 /usr/lib/uucp/uuconv
-r-sr-sr-- 1 uucp uucp 315 Nov 22 1995 /usr/lib/uucp/uusched
-r-sr-sr-- 1 uucp uucp 95420 Feb 10 2001 /usr/lib/uucp/uuxqt

When was the last time you connected to another machine with UUCP? Have you ever set up the UUCP system? I have been a network admin for ten years, and in that time, I have never come across a live UUCP installation. That's not to say that UUCP isn't useful, just that in these days of permanently connected TCP/IP networks, UUCP is becoming extremely uncommon. If you're not using UUCP, then leaving setuid and setgid binaries online to support it doesn't make much sense.

Do any of the binaries in the examples above have potential root (or other privilege elevation) exploits? I have no idea. But I do know that by removing unnecessary privileges, I minimize my exposure to the possibility that an exploit might be run on this system if one is discovered.

I have tried to present this hack as a process, not as a solution. This list is certainly by no means definitive. As you build a server, always keep in mind what the intended use is, and build the system accordingly. Whenever possible, remove privileges (or even entire packages) that provide functionality that you simply don't need. Consult the manpage if you ever wonder about what a particular binary is supposed to be doing, and why it is installed setuid (and when the manpage fails you, remember to use the source).

Hack #12. Make sudo Work Harder

Use sudo to let other users do your evil bidding, without giving away the machine

The sudo utility can help you delegate some system responsibilities to other people, without giving away full root access. It is a setuid root binary that executes commands on an authorized user's behalf, after they have entered their current password.

As root, run /usr/sbin/visudo to edit the list of users who can call sudo. The default sudo list looks something like this:

root ALL=(ALL) ALL

Unfortunately, many system admins tend to use this entry as a template and grant unrestricted root access to all other admins unilaterally:

root ALL=(ALL) ALL
rob ALL=(ALL) ALL
jim ALL=(ALL) ALL
david ALL=(ALL) ALL

While this may allow you to give out root access without giving away the root password, it is really only a useful method when all of the sudo users can be completely trusted. When properly configured, the sudo utility allows for tremendous flexibility for granting access to any number of commands, run as any arbitrary uid.

The syntax of the sudo line is:

            user machine=(effective user) command

The first column specifies the sudo user. The next column defines the hosts in which this sudo entry is valid. This allows you to easily use a single sudo configuration across multiple machines.

For example, suppose you have a developer who needs root access on a development machine, but not on any other server:

peter beta.oreillynet.com=(ALL) ALL

The next column (in parentheses) specifies the effective user that may run the commands. This is very handy for allowing users to execute code as users other than root:

peter lists.oreillynet.com=(mailman) ALL

Finally, the last column specifies all of the commands that this user may run:

david ns.oreillynet.com=(bind) /usr/sbin/rndc,/usr/sbin/named

If you find yourself specifying large lists of commands (or, for that matter, users or machines), then take advantage of sudo's Alias syntax. An Alias can be used in place of its respective entry on any line of the sudo configuration:

User_Alias ADMINS=rob,jim,david
User_Alias WEBMASTERS=peter,nancy

Runas_Alias DAEMONS=bind,www,smmsp,ircd

Host_Alias WEBSERVERS=www.oreillynet.com,www.oreilly.com,www.perl.com

Cmnd_Alias PROCS=/bin/kill,/bin/killall,/usr/bin/skill,/usr/bin/top
Cmnd_Alias APACHE=/usr/local/apache/bin/apachectl

WEBMASTERS WEBSERVERS=(www) APACHE
ADMINS ALL=(DAEMONS) ALL

It is also possible to specify system groups in place of the user specification to allow any user who belongs to that group to execute commands. Just preface the group with a %, like this:

%wwwadmin WEBSERVERS=(www) APACHE

Now any user who is part of the wwwadmin group can execute apachectl as the www user on any of the web server machines.

One very useful feature is the NOPASSWD: flag. When present, the user won't have to enter his password before executing the command:

rob ALL=(ALL) NOPASSWD: PROCS

This will allow the user rob to execute kill, killall, skill, and top on any machine, as any user, without entering a password.

Finally, sudo can be a handy alternative to su for running commands at startup out of the system rc files:

(cd /usr/local/mysql; sudo -u mysql ./bin/safe_mysqld &)

sudo -u www /usr/local/apache/bin/apachectl start

For that to work at boot time, you'll need the default line root ALL=(ALL) ALL to be present.

Use sudo with the usual caveats that apply to setuid binaries. Particularly if you allow sudo to execute interactive commands (like editors) or any sort of compiler or interpreter, you should assume that it is possible that the sudo user will be able to execute arbitrary commands as the effective user. Still, under most circumstances this isn't a problem and is certainly preferable to giving away undue access to root privileges.

Hack #13. Using a Makefile to Automate Admin Tasks

Makefiles make everything (not just gcc) faster and easier

You probably know the make command from building projects (probably involving gcc) from source. But not many people also realize that since it keeps track of file modification times, it can be a handy tool for making all sorts of updates whenever arbitrary files are updated.

Here's a Makefile that is used to maintain sendmail configuration files:

Listing: Makefile.mail

M4= m4
CFDIR= /usr/src/sendmail-8.12.5/cf
CHMOD= chmod
ROMODE= 444
RM= rm -f

.SUFFIXES: .mc .cf

all: virtusers.db aliases.db access.db sendmail.cf

access.db: access.txt
makemap -v hash access < access.txt

aliases.db: aliases
newaliases

virtusers.db: virtusers.txt
makemap -v hash virtusers < virtusers.txt

.mc.cf:
$(RM) $@
$(M4) ${CFDIR}/m4/cf.m4 $*.mc > $@ || ( $(RM) $@ && exit 1 )
$(CHMOD) $(ROMODE) $@

With this installed as /etc/mail/Makefile, you'll never have to remember to run newaliases when editing your sendmail aliases file, or the syntax of that makemap command when you update virtual domain or access control settings. And best of all, when you update your master mc configuration file (you are using mc and not editing the sendmail.cf by hand, right?) then it will build your new .cf file for you — all by simply typing make. Since make keeps track of files that have been recently updated, it takes care of rebuilding only what needs to be rebuilt.

Here's another example, used to push Apache configuration files to another server (say, in around-robinApachesetup, which you can learn more about in [Hack #99]. Just put this in your /usr/local/apache/conf directory:

Listing: Makefile.push

#
# Makefile to push *.conf to the slave, as needed.
#
SLAVE= www2.oreillynet.com
APACHE= /usr/local/apache
RM= /bin/rm
TOUCH= /bin/touch
SSH= /usr/local/bin/ssh
SCP= /usr/local/bin/scp

.SUFFIXES: .conf .ts

all: test restart sites.ts globals.ts httpd.ts

configtest: test

test:
@echo -n "Testing Apache configuration: "
@$(APACHE)/bin/apachectl configtest

restart:
$(APACHE)/bin/apachectl restart

.conf.ts:
@$(RM) -f $@
@$(SCP) $*.conf $(SLAVE):$(APACHE)/conf
@$(SSH) $(SLAVE) $(APACHE)/bin/apachectl restart
@$(TOUCH) $@

This example is a little trickier because we're not actually building any new files, so it's difficult for make to tell if any of the configuration files have actually changed. We fake out make by creating empty .ts files (short for TimeStamp) that only serve to hold the time and date of the last update. If the real files (httpd.conf, sites.conf, or globals.conf) have changed, then we first run an apachectl configtest to verify that we have a good configuration. If all goes well, it will then restart the local Apache, copy the newly changed files over to the slave server, then restart Apache on the slave server. Finally, we touch the relevant .ts files so we won't process them again until the .conf files change.

This saves a lot of typing and is significantly quicker (and safer) than doing it all by hand on each update.

Hack #14. Brute Forcing Your New Domain Name

Find exactly the domain you'd like to register, whatever it turns out to be

There are many tools available online that will assist in performing whois queries for you to determine if your favorite domain name is still available, and if not, who has registered it. These tools are usually web based and allow you to submit a few queries at a time (and frequently suggest several inane alternatives if your first choice is taken).

If you're not so much interested in a particular name as in finding one that matches a pattern, why not let the command line do the work for you? Suppose you wanted to find a list of all words that end in the letters "st":

cat /usr/share/dict/words | grep 'st$' | sed 's/st$/.st/' | \
while read i; do \
(whois $i | grep -q '^No entries found') && echo $i; sleep 60; \
done | tee list_of_st_domains.txt

This will obligingly supply you with a visual running tab of all available words that haven't yet been registered to el Republica Democratica de Sào Tomé e Príncipe (the domain registrar for the st TLD). This example searches the system dictionary and tries to find the whois record for each matching word, one at a time, every 60 seconds. It saves any nonexistent records to a file called list_of_st_domains.txt, and shows you its progress as it runs. Replace that st with any two letter TLD (like us or to) to brute force the namespace of any TLD you like.

Some feel that the domain name land grab is almost turning the Internet into a corporate ghetto, but I don't subscribe to that idea. I actually find the whole situation quite humorous.

Hack #15. Playing Hunt the Disk Hog

Browse your filesystem for heavy usage quickly with a handy alias

It always seems to happen late on a Saturday night. You're getting paged because a partition on one of the servers (probably the mail server) is dangerously close to full.

Obviously, running a df will show what's left:

rob@magic:~$ df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda1 7040696 1813680 4863600 27% /
/dev/sda2 17496684 13197760 3410132 79% /home
/dev/sdb1 8388608 8360723 27885 100% /var/spool/mail

But you already knew that the mail spool was full (hence, the page that took you away from an otherwise pleasant, non-mailserver related evening). How can you quickly find out who's hogging all of the space?

Here's a one-liner that's handy to have in your .profile:

alias ducks='du -cks * |sort -rn |head -11'

Once this alias is in place, running ducks in any directory will show you the total in use, followed by the top 10 disk hogs, in descending order. It recurses subdirectories, which is very handy (but can take a long time to run on a heavily loaded server, or in a directory with many subdirectories and files in it). Let's get to the bottom of this:

rob@magic:~$ cd /var/spool/mail
rob@magic:/var/spool/mail$ ducks
8388608 total
1537216 rob
55120 phil
48800 raw
43175 hagbard
36804 mal
30439 eris
30212 ferris
26042 nick
22464 rachael
22412 valis

Oops! It looks like my mail spool runneth over. Boy, I have orders of magnitude more mail than any other user. I'd better do something about that, such as appropriate new hardware and upgrade the /var/spool/mail partition. 😉

As this command recurses subdirectories, it's also good for running a periodic report on home directory usage:

root@magic:/home# ducks
[ several seconds later ]
13197880 total
2266480 ferris
1877064 valis
1692660 hagbard
1338992 raw
1137024 nick
1001576 rob
925620 phil
870552 shared
607740 mal
564628 eris

For running simple spot checks while looking for disk hogs, ducks can save many keystrokes (although if we called it something like ds, it would save even more, but wouldn't be nearly as funny.)

Hack #16. Fun with /proc

Directly view the kernel's running process table and system variables

The /proc filesystem contains a representation of the kernel's live process table. By manipulating files and directories in /proc, you can learn about (and fiddle with) all sorts of parameters in the running system. Be warned that poking around under /proc as root can be extraordinarily dangerous, as root has the power to overwrite virtually anything in the process table. One slip of a redirector, and Linux will very obligingly blow away your entire kernel memory, without so much as a "so long and thanks for all the kcore."

Here are some examples of interesting things to do with /proc. In these examples, we'll assume that you're using a recent kernel (about 2.4.18 or so) and that you are logged in as root. Unless you're root, you will only be able to view and modify processes owned by your uid.

First, let's take a look at a lightly loaded machine:

root@catlin:/proc# ls
1/ 204/ 227/ 37/ bus/ hermes/ loadavg scsi/ version
1039/ 212/ 228/ 4/ cmdline ide/ locks self@
1064/ 217/ 229/ 5/ cpuinfo interrupts meminfo slabinfo
1078/ 220/ 230/ 6/ devices iomem misc stat
194/ 222/ 231/ 698/ dma ioports modules swaps
197/ 223/ 232/ 7/ driver/ irq/ mounts sys/
2/ 224/ 233/ 826/ execdomains kcore net/ sysvipc/
200/ 225/ 254/ 827/ filesystems kmsg partitions tty/
202/ 226/ 3/ apm fs/ ksyms pci uptime

The directories consisting of numbers contain information about every process running on the system. The number corresponds to the PID. The rest of the files and directories correspond to drivers, counters, and many other internals of the running kernel. The interface operates just as any other file or device in the system, by reading from and writing to each entry as if it were a file. Suppose you want to find out which kernel is currently booted:

root@catlin:/proc# cat version
Linux version 2.4.18 (root@catlin) (gcc version 2.95.3 20010315 (release))
#2 Sat Jun 22 19:01:17 PDT 2002

Naturally, you could find much of that out by simply running uname -a, but this way cuts out the middle man (and actually does what uname does internally).

Interested in how much RAM we have installed? Take a look at kcore:

root@catlin:/proc# ls -l kcore
-r-------- 1 root root 201330688 Aug 28 21:39 kcore

Looks like we have 192MB installed in this machine (201330688/1024/1024 == 192, more or less). Notice the restricted file permissions? That is the system's defense against anyone attempting to read the memory directly. Of course, you're root and can do whatever you like. There's nothing preventing you from running grep or strings on kcore and looking for interesting tidbits. This is the system memory, and things get cached in there until overwritten (making it possible to hunt down accidentally lost data or track down naughty people doing things they oughtn't). Grovelling over kcore is actually not much fun and is typically only a method used by the very desperate (or very bored).

Some other notable status files:

root@catlin:/proc# cat interrupts 
CPU0
0: 34302408 XT-PIC timer
1: 2 XT-PIC keyboard
2: 0 XT-PIC cascade
3: 289891 XT-PIC orinoco_cs
8: 1 XT-PIC rtc
9: 13933886 XT-PIC eth0
10: 25581 XT-PIC BusLogic BT-958
14: 301982 XT-PIC ide0
NMI: 0
ERR: 0

These are the system counters for every interrupt that has ever been called, its number, and the driver that called it.

root@catlin:/proc# cat partitions 
major minor #blocks name

8 0 8971292 sda
8 1 8707198 sda1
8 2 257040 sda2
3 64 29316672 hdb
3 65 29310561 hdb1

This is a list of all of the hard disk partitions (and devices) that were discovered at boot, along with their respective sizes. Here we have a 9GB SCSI disk, and a 30GB IDE disk. All available partitions are represented here, regardless of whether they are currently mounted (making it a handy reference to see if there are any unmounted disks on the system).

Let's leave the system parameters and take a look at the structure of an individual process.

root@catlin:/proc# cd 1
root@catlin:/proc/1# ls -l
total 0
-r--r--r-- 1 root root 0 Aug 28 22:05 cmdline
lrwxrwxrwx 1 root root 0 Aug 28 22:05 cwd -> //
-r-------- 1 root root 0 Aug 28 22:05 environ
lrwxrwxrwx 1 root root 0 Aug 28 22:05 exe -> /sbin/init*
dr-x------ 2 root root 0 Aug 28 22:05 fd/
-r--r--r-- 1 root root 0 Aug 28 22:05 maps
-rw------- 1 root root 0 Aug 28 22:05 mem
lrwxrwxrwx 1 root root 0 Aug 28 22:05 root -> //
-r--r--r-- 1 root root 0 Aug 28 22:05 stat
-r--r--r-- 1 root root 0 Aug 28 22:05 statm
-r--r--r-- 1 root root 0 Aug 28 22:05 status

There are three interesting symlinks in this directory. cwd points to the current working directory of this process (you can, for example, cd /proc/3852/cwd to land in the directory from which process ID 3852 was run.) The exe link points to full path to the binary that was called, and root points to the notion of the root directory that this process has. The root link will almost always be /, unless it has executed a chroot.

The cmdline and environ files contain the command line as it was originally called and the processes complete environment. These are separated by NULL characters, so to see them in a more human readable form, try this:

root@catlin:/proc/1# cat environ |tr '\0' '\n'
HOME=/
TERM=linux
BOOT_IMAGE=catlin

(or for a better example, try this on /proc/self/environ, the environment of the currently running process):

root@catlin:/proc/1# cat /proc/self/environ |tr '\0' '\n'
PWD=/proc/1
HOSTNAME=catlin.nocat.net
MOZILLA_HOME=/usr/lib/netscape
ignoreeof=10
LS_OPTIONS= --color=auto -F -b -T 0
MANPATH=/usr/local/man:/usr/man:/usr/X11R6/man
LESSOPEN=|lesspipe.sh %s
PS1=\u@\h:\w\$
PS2=>
...

and so on. This can be tremendously handy for use in shell scripts (or other programs) where you need specific information about running processes. Just use it with care, and remember that unprivileged users can usually only access information about their own processes.

In closing, here's a practical example of one use for /proc . By checking the output of ps against the running process table, you can see if the two agree:

# ls -d /proc/* |grep [0-9]|wc -l; ps ax |wc -l

This will give you a quick spot check of the number of running processes versus the number that ps actually reports. Many rootkits install a hacked ps that allows a miscreant to hide processes (by simply not displaying them when ps runs). You may hide from ps, but it's much more difficult to hide from /proc. If the second number is considerably larger than the first (particularly if you run it several times in a row) then you might want to consider taking your box offline for further inspection. Quickly.

Hack #17. Manipulating Processes Symbolically with procps

Signal and renice processes by name, terminal, or username (instead of PID)

If you often find yourself running a ps awux |grep something just to find the PID of a job you'd like to kill, then you should take a look at some of the more modern process manipulation packages.

Probably the best known process tools package is procps , the same package that includes the Linux version of the ubiquitous top command. The top tool is so tremendously useful and flexible that it deserves its own discussion. You can learn more about the top tool in [Hack #58].

Among the other nifty utilities included in procps:

skill lets you send signals to processes by name, terminal, username, or PID. snice does the same but renices processes instead of sending them signals.

For example, to freeze the user on terminal pts/2:

# skill -STOP pts/2

To release them from the grip of sleeping death, try this:

# skill -CONT pts/2

Or to renice all of luser's processes to 5:

# snice +5 luser

pkill is similar to skill, but with more formal parameters. Rather than attempting to guess whether you are referring to a username, process name, or terminal, you specify them explicitly with switches. For example, these two commands do the same thing:

# skill -KILL rob bash
# pkill -KILL -u rob bash

pkill may take slightly more typing, but is guaranteed to be unambiguous (assuming that you happened to have a user and a process with the same name, for example).

pgrep works just like pkill, but instead of sending a signal to each process, it simply prints the matching PID(s) on STDOUT:

$ pgrep httpd
3211
3212
3213
3214
3215
3216

Finally, vmstat gives you a nice, easily parsed pinpoint measurement of virtual memory and cpu statistics:

$ vmstat 
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 5676 6716 35804 58940 0 0 9 9 7 9 0 0 29

If you'd like to watch how usage changes over time, give it a number on the command line. The number represents the number of seconds to pause before displaying the results of another measurement.

Learning how to use the lesser known procps utilities can save you lots of typing, not to mention wincing. Use skill or pkill to avoid accidentally mistyping a PID argument to kill, and suddenly bringing sshd (and the wrath of many angry users) down upon your unfortunate sysadmin head.

See also:

• ftp://people.redhat.com/johnsonm/procps/
• [Hack #58]
• manpages for pkill, pgrep, skill, snice, and vmstat

Hack #18. Managing System Resources per Process

Prevent user processes from running away with all system resources

Whether intentionally or accidentally, it is entirely possible for a single user to use up all system resources, leading to poor performance or outright system failure. One frequently overlooked way to deal with resource hogs is to use the ulimit functionality of bash.

To prevent a process (or any of its children) from creating enormous files, try specifying a ulimit -f (with the maximum file size specified in kilobytes).

rob@catlin:/tmp$ ulimit -f 100
rob@catlin:/tmp$ yes 'Spam spam spam spam SPAM!' > spam.txt
File size limit exceeded
rob@catlin:/tmp$ ls -l spam.txt
-rw-r--r-- 1 rob users 102400 Sep 4 17:05 spam.txt
rob@catlin:/tmp$

Users can decrease their own limits, but not increase them (as with nice and renice). This means that ulimits set in /etc/profile cannot be increased later by users other than root:

rob@catlin:/tmp$ ulimit -f unlimited
bash: ulimit: cannot modify limit: Operation not permitted

Note that nothing is preventing a user from creating many files, all as big as their ulimit allows. Users with this particular temperament should be escorted to a back room and introduced to your favorite LART. Or alternately, you could look into introducing disk quotas (although this is usually less than fun to implement, if a simple stern talking to will fix the problem).

Likewise, ulimit can limit the maximum number of children that a single user can spawn:

rob@catlin:~$ cat > lots-o-procs
#!/bin/bash
export RUN=$((RUN + 1))
echo $RUN...
$0
^D
rob@catlin:~$ ulimit -u 10
rob@catlin:~$ ./lots-o-procs
1...
2...
3...
4...
5...
6...
7...
8...
9...
./lots-o-procs: fork: Resource temporarily unavailable
rob@catlin:~$

This limits the number of processes for a single user across all terminals (and back grounded processes). It has to be this way, because once a process is forked, it disassociates itself from the controlling terminal. (And how would you count it against a given subshell then?)

One other very useful ulimit option is -v, maximum virtual memory size. Once this ceiling is reached, processes will exit with Segmentation fault (which isn't ideal, but will keep the system from crashing as it runs out of RAM and swap). If you have a particularly badly behaving process that shows significant bloat (like Apache + mod_perl and poorly written CGI code, for example) you could set a ulimit to act as an "emergency brake" while debugging the real source of the trouble. Again, specify the limit in kilobytes.

To see all available ulimit settings, use -a:

rob@catlin:~$ ulimit -a 
core file size (blocks) 0
data seg size (kbytes) unlimited
file size (blocks) unlimited
max locked memory (kbytes) unlimited
max memory size (kbytes) unlimited
open files 1024
pipe size (512 bytes) 8
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 1536
virtual memory (kbytes) unlimited

You can see that before setting system-wide hard limits, user processes can grow to be quite large. In tcsh, the analogous command you're after is limit:

rob@catlin:~> limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 1536
openfiles 1024

Setting system resource limits may sound draconian but is a much better alternative to the downward spiral of a user process gone amok.

See also:

• http://www.tuxedo.org/~esr/jargon/html/entry/LART.html

Hack #19. Cleaning Up after Ex-Users

Make sure you close the door all the way when a user takes their leave

It happens. Plans change, companies shift focus, and people move on. At some point, every admin has had to clean up shell access after someone has left the company, happily or otherwise.

I am personally of the opinion that if one doesn't trust one's users from the beginning, then they will one day prove themselves untrustworthy. (Of course, I've never had to admin an open shell server at an ISP, either.) At any rate, building trust with your users from the beginning will go a long way toward being able to sleep at night later on.

When you do have to lock old accounts up, it's best to proceed strategically. Don't assume that just because you ran a passwd -l that the user in question can't regain access to your machine. Let's assume that we're locking up after an account called luser. Here are some obvious (and some not so obvious) things to check on in the course of cleaning up:

passwd -l luser

Obviously, locking the user's password is a good first step.

chsh -s /bin/true luser

This is another popular step, changing the user's login shell to something that exits immediately. This generally prevents a user from gaining shell access to the server. But be warned, if sshd is running on this box, and you allow remote RSA or DSA key authentication, then luser can still forward ports to any machine that your server can reach! With a command like this:

luser@evil:~$ ssh -f -N -L8000:private.intranet.server.com:80 old.server.com

luser has just forwarded his local port 8000 to your internal intranet server's http port. This is allowed since luser isn't using a password (he is using an RSA key) and isn't attempting to execute a program on old.server.com (since he specified the -N switch).

Obviously, you should remove ~luser/.ssh/authorized_keys* and prevent luser from using his ssh key in the first place. Likewise, look for either of these files:

~luser/.shosts
~luser/.rhosts

This usually isn't a problem unless you're running rsh, or have enabled this functionality in ssh. But you never know if a future admin will decide to enable .shosts or .rhosts functionality, so it's better to remove them now, if they exist.

Did luser have any sudo privileges? Check visudo to be sure.

How about cron jobs or at jobs?

crontab -u luser -e
atq

For that matter, is luser running any jobs right now?

ps awux |grep -i ^luser

or as in [Hack #17], you can use:

 skill -KILL luser

Could luser execute cgi programs from his home directory (or somewhere else)?

find ~luser/public_html/ -perm +111

What about PHP or other embedded scripting languages?

find ~luser ~public_html/ -name '*.php*'

Does luser have any email forwarding set up? Forwarders can frequently be made to execute arbitrary programs.

less ~luser/.forward
grep -C luser /etc/mail/aliases

Finally, does luser own any files in strange places?

find / -user luser > ~root/luser-files.report

One safe (and quick) way of ensuring that all of luser's personal configuration files are invalidated is to mv /home/luser /home/luser.removed. This will keep the contents of luser's home directory intact, without worrying about having missed other possible points of entry. Note that this will break a legitimate .forward file (and also ~luser/public_html and any other publicly accessible data that luser might have kept in his home directory), so if you go this route be sure to take that into account (say, by adding an appropriate entry to the system aliases file, and moving a cleaned version of public_html/ back to /home/luser/public_html/.

Look at configuration files for any system software to which luser had access, particularly services that run as privileged users. Even something as innocuous as a user-supplied Apache configuration file could be used to provide shell access later.

This list is by no means exhaustive but is meant to demonstrate that there is a lot more to revoking access than simply locking a user's password. If this user ever had root access, all bets are off. Access could later be granted by anything from a Trojan system binary to an invisible kernel module to having simply changed the root password.

Get to know your shell users long before the day you have to say goodbye. This job is difficult enough without having to worry about whom you can trust.

Hack #20. Eliminating Unnecessary Drivers from the Kernel

Keep your kernel optimized for quick booting and long-term stability

Linux will run on an enormous variety of computer hardware. There is support for all manner of hardware, from critical components (such as hard disk drives and RAM) to more exotic devices (such as USB scanners and video capture boards). The kernel that ships with most Linux distributions aims to be complete (and safe) at the expense of possibly being less efficient than it could be, by including support for as many devices as possible.

As your machine boots, take a look at the messages the kernel produces. You may find it probing for all sorts of hardware (particularly SCSI controllers and Ethernet cards) that you don't actually have installed. If your distribution hides the kernel boot messages, try the dmesg command (probably piped through less) to see what kernel messages were generated at boot.

To make your kernel boot quickly and at the same time eliminate the possibility that an unnecessary device driver might be causing problems with your installed hardware, you should trim down the drivers that the kernel attempts to load to fit your hardware.

There are two schools of thought on managing kernel drivers. Some people prefer to build a kernel with all of the functionality they need built into it, without using loadable kernel modules. Others prefer to build a more lightweight kernel and load the drivers they need when the system boots. Of course, both methods have their advantages and disadvantages.

For example, the monolithic kernel (without loadable modules) is guaranteed to boot, even if something happens to the drivers under /lib/modules. Some admins even prefer to build a kernel with no loadable module support at all, to discourage the possibility of Trojan horse device drivers being loaded by a random miscreant down the road. On the other hand, if a new piece of hardware is added to the system, then you will need to rebuild your kernel to accommodate it.

If you use loadable modules, then you have enormous flexibility in how you load device drivers. You can alter the order that drivers load and even pass parameters to various modules as you load the driver, all without rebooting. The downside is that good copies of the modules must exist under /lib/modules, or else the system can't load its drivers. When you build a new kernel, you'll need to remember to run a make modules_install, and to keep your kernel module utilities (like modprobe and lsmod) up to date. Building a kernel with loadable modules can also help if your monolithic kernel is too large to boot (by loading extra device drivers after the kernel boots and the filesystems are mounted).

Regardless of the method that you choose for your system, you'll need to build a kernel with the minimal functionality that is required to boot. This includes drivers for the IDE, SCSI, or other bus (maybe floppy disk or even network card?) that your machine boots from. You will also need support for the filesystem that your root partition is installed on (likely ext2, ext3, or reiserfs). Make sure that you build a kernel with support for the amount of RAM that your machine has installed (see [Hack #21]). Select a processor that matches your hardware to be sure that all appropriate optimizations are turned on. Be sure to build an SMP kernel if you have more than one processor.

To build a customized kernel, unpack the kernel sources somewhere that has enough room (say, in /usr/local/src/linux). Run a make menuconfig (or, if you're running X, make xconfig). Select your drivers carefully (hitting Y for built-in drivers, and M for loadable modules.) Remember that you're building a kernel for a server and don't include extraneous drivers. Does your server really need sound support, even if it is built into your motherboard? What about USB? Unless you have a specific use for a particular piece of hardware that is directly related to its job as a server, don't bother installing a driver for it. You should also consider disabling unused hardware in the BIOS, wherever possible, to conserve system resources and reduce the possibility of hardware resource conflicts.

After you're finished selecting which bits of the kernel to build, it will save your configuration to a file called .config at the top of the kernel source tree. Save this file for later, because it will make upgrading your kernel much easier down the road (by copying it to the top of the new kernel tree, and running make oldconfig). Now build the new kernel (and modules, if applicable), install it, and give it a try. Be sure to try out all of your devices after installing a new kernel, and watch your boot messages carefully. Track down any unexpected warnings or errors now, before they cause trouble at some future date.

Finally, if you're doing kernel work remotely, don't forget to build in support for your network devices! There's nothing quite like the pain of realizing that the network drivers aren't built just after you issue a shutdown -r now. Hopefully that console is accessible by someone on call (who will discretely boot your old kernel for you, without telling too many of your friends).

Building a kernel well-tailored to your hardware can be challenging, but is necessary to make your server run as efficiently as it can. Don't be discouraged if it takes a couple of tries to build the perfect Linux kernel, the effort is well worth it.

See also:

• Running Linux, Fourth Edition (O'Reilly)
• [Hack #21]

Hack #21. Using Large Amounts of RAM

Be sure that Linux is using all of your available system RAM

Linux is capable of addressing up to 64 GB of physical RAM on x86 systems. But if you want to accommodate more than 960 MB RAM, you'll have to let the system know about it.

First of all, your Linux kernel must be configured to support the additional RAM. Typically, the default kernel configuration will address up to 960 MB RAM. If you install more than that in a machine, it will simply be ignored. (The common complaint is that you've just installed 1 GB, and yet a `free' only reports 960MB, even though it counts to 1024 MB at post time.)

The way that the kernel addresses its available system memory is dictated by the High Memory Support setting (a.k.a. the CONFIG_NOHIGHMEM define.) Depending on the amount of RAM you intend to use, set it accordingly:

up to 960MB: off
up to 4GB: 4GB
more than 4GB: 64GB

Be warned that selecting 64 GB requires a processor capable of using Intel Physical Address Extension (PAE) mode. According to the kernel notes, all Intel processors since the Pentium Pro support PAE, but this setting won't work on older processors (and the kernel will refuse to boot, which is one reason that it isn't on by default). Make your selection and rebuild your kernel, as in [Hack #20].

Once the kernel is built and installed, you may have to tell your boot loader how much RAM is installed, so it can inform the kernel at boot time (as not every BIOS is accurate in reporting the total system RAM at boot.) To do this, add the mem= kernel parameter in your bootloader configuration. For example, suppose we have a machine with 2GB RAM installed.

If you're using Lilo, add this line to /etc/lilo.conf:

append="mem=2048M"

If you're using Grub, try this in your /etc/grub.conf:

kernel /boot/vmlinuz-2.4.19 mem=2048M

If you're running loadlin, just pass it on the loadlin line:

c:\loadlin c:\kernel\vmlinuz root=/dev/hda3 ro mem=2048M

Although, if you're running loadlin, why are you reading a book on Linux Server Hacks? 😉

Hack #22. hdparm: Fine Tune IDE Drive Parameters

Get the best possible performance from your IDE hardware

If you're running a Linux system with at least one (E)IDE hard drive, and you've never heard of hdparm, read on.

By default, most Linux distributions use the default kernel parameters when accessing your IDE controller and drives. These settings are very conservative and are designed to protect your data at all costs. But as many have come to discover, safe almost never equals fast. And with large volume data processing applications, there is no such thing as "fast enough."

If you want to get the most performance out of your IDE hardware, take a look at the hdparm(8) command. It will not only tell you how your drives are currently performing, but will let you tweak them to your heart's content.

It is worth pointing out that under some circumstances, these commands CAN CAUSE UNEXPECTED DATA CORRUPTION! Use them at your own risk! At the very least, back up your box and bring it down to single-user mode before proceeding.

Let's begin. Now that we're in single user mode (which we discussed in [Hack #2]), let's find out how well the primary drive is currently performing:

hdparm -Tt /dev/hda

You should see something like:

/dev/hda:
Timing buffer-cache reads: 128 MB in 1.34 seconds =95.52 MB/sec
Timing buffered disk reads: 64 MB in 17.86 seconds = 3.58 MB/sec

What does this tell us? The -T means to test the cache system (i.e., the memory, CPU, and buffer cache). The -t means to report stats on the disk in question, reading data not in the cache. The two together, run a couple of times in a row in single-user mode, will give you an idea of the performance of your disk I/O system. (These are actual numbers from a PII/350/128M Ram/EIDE HD; your numbers will vary.)

But even with varying numbers, 3.58 MB/sec is pathetic for the above hardware. I thought the ad for the HD said something about 66 MB per second!!?!? What gives?

Let's find out more about how Linux is addressing this drive:

# hdparm /dev/hda

/dev/hda:
multcount = 0 (off)
I/O support = 0 (default 16-bit)
unmaskirq = 0 (off)
using_dma = 0 (off)
keepsettings = 0 (off)
nowerr = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 1870/255/63, sectors = 30043440, start = 0

These are the defaults. Nice, safe, but not necessarily optimal. What's all this about 16-bit mode? I thought that went out with the 386!

These settings are virtually guaranteed to work on any hardware you might throw at it. But since we know we're throwing something more than a dusty, 8-year-old, 16-bit multi-IO card at it, let's talk about the interesting options:

multcount

Short for multiple sector count. This controls how many sectors are fetched from the disk in a single I/O interrupt. Almost all modern IDE drives support this. The manpage claims:

When this feature is enabled, it typically reduces operating system overhead for disk I/O by 30-50%. On many systems, it also provides increased data throughput of anywhere from 5% to 50%.

I/O support

This is a big one. This flag controls how data is passed from the PCI bus to the controller. Almost all modern controller chipsets support mode 3, or 32-bit mode w/sync. Some even support 32-bit async. Turning this on will almost certainly double your throughput (see below).

unmaskirq

Turning this on will allow Linux to unmask other interrupts while processing a disk interrupt. What does that mean? It lets Linux attend to other interrupt-related tasks (i.e., network traffic) while waiting for your disk to return with the data it asked for. It should improve overall system response time, but be warned: not all hardware configurations will be able to handle it. See the manpage.

using_dma

DMA can be a tricky business. If you can get your controller and drive using a DMA mode, do it. However, I have seen more than one machine hang while playing with this option. Again, see the manpage.

Let's try out some turbo settings:

# hdparm -c3 -m16 /dev/hda

/dev/hda:
setting 32-bit I/O support flag to 3
setting multcount to 16
multcount = 16 (on)
I/O support = 3 (32-bit w/sync)

Great! 32-bit sounds nice. And some multi-reads might work. Let's re-run the benchmark:

# hdparm -tT /dev/hda

/dev/hda:
Timing buffer-cache reads: 128 MB in 1.41 seconds =90.78 MB/sec
Timing buffered disk reads: 64 MB in 9.84 seconds = 6.50 MB/sec

Hmm, almost double the disk throughput without really trying! Incredible.

But wait, there's more: we're still not unmasking interrupts, using DMA, or even a using decent PIO mode! Of course, enabling these gets riskier. The manpage mentions trying Multiword DMA mode2, so let's try this:

# hdparm -X34 -d1 -u1 /dev/hda

Unfortunately this seems to be unsupported on this particular box (it hung like an NT box running a Java application) So, after rebooting it (again in single-user mode), I went with this:

# hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda

/dev/hda:
setting 32-bit I/O support flag to 3
setting multcount to 16
setting unmaskirq to 1 (on)
setting using_dma to 1 (on)
setting xfermode to 66 (UltraDMA mode2)
multcount = 16 (on)
I/O support = 3 (32-bit w/sync)
unmaskirq = 1 (on)
using_dma = 1 (on)

And then checked:

# hdparm -tT /dev/hda

/dev/hda:
Timing buffer-cache reads: 128 MB in 1.43 seconds =89.51 MB/sec
Timing buffered disk reads: 64 MB in 3.18 seconds =20.13 MB/sec

20.13 MB/sec. A far cry from the miniscule 3.58 with which we started.

Did you notice how we specified the -m16 and -c3 switch again? That's because it doesn't remember your hdparm settings between reboots. Be sure to add the above line to your /etc/rc.d/* scripts once you're sure the system is stable (and preferably after your fsck runs; running an extensive filesystem check with your controller in a flaky mode may be a good way to generate vast quantities of entropy, but it's no way to administer a system. At least not with a straight face).

If you can't find hdparm on your system (usually in /sbin or /usr/sbin), get it from the source at http://metalab.unc.edu/pub/Linux/system/hardware/.

Chapter 2. Revision Control

Hacks #23-36

If you take administration seriously, you will likely spend untold hours fine-tuning your system to behave just so. Unfortunately, it isn't always possible to define when a given application is "working" and when it is "broken"; usually functionality is much more finely graded, somewhere in the grey area between up and down. Even making a small configuration change can bring about subtle effects that aren't seen for days (or weeks) down the road.

This is where Revision Control can be a powerful ally. Beyond providing simple backups, any revision control package worth its weight in bits will keep track of the changes in any file, when it was changed, who changed it, and why the change was made. This information can be of untold value when working on a complex system, particularly if more than one person is responsible for its upkeep. By keeping critical configuration files in systems such as RCS or CVS, you will be able to "roll back" to any previous revision of a file, and analyze the differences between what was present then and the version you have online right now.

In this section, the goal is to provide the specific syntax you need to make effective use of RCS and CVS, with useful examples of how you might use them. While RCS is extremely handy for maintaining file revisions on a single machine, CVS offers extensive archiving and merging capabilities, keeping a central repository somewhere on the network. Understandably, CVS is a much more complicated tool than RCS. Hacks 23 through 25 offer a crash course in RCS, while 26 through 36 will get you up to speed quickly on CVS, from simple commands to setting up your own anonymous CVS repository.

Naturally, this chapter assumes that you're already comfortable working with files from the command line, even if you haven't worked with RCS or CVS before. If you're looking for in-depth information about CVS, take a look at the excellent book Open Source Development with CVS by Karl Fogel (CoriolisOpen Press).

Hack #23. Getting Started with RCS

Let RCS manage your system files, and keep a revision history

RCS is a revision control system, useful for keeping multiple revisions of configuration files, shell scripts, and any other text files you have lying around. Unlike CVS, there is no functionality for using RCS with remote machines; in RCS, everything is intended to be kept in the local filesystem.

RCS keeps all of its revisions in a directory called RCS/ (under the directory you're currently in). To start a new RCS repository, simply make the directory:

root@catlin:/etc# mkdir RCS

There, that wasn't so bad, was it? Now, we'll need to initialize a file into the new RCS repository before working with it. Let's work with syslog.conf:

root@catlin:/etc# ci -i syslog.conf
RCS/syslog.conf,v <-- syslog.conf
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> Here's the syslog.conf for Catlin
initial revision: 1.1
done

The initial ci -i does a "check in and initialize," to give RCS a good first copy of the file. RCS prompts for an initial description, and then removes the file from the current directory. You probably don't want to leave it like that for too long, so after you've initialized the file in RCS, check it out:

root@catlin:/etc# co syslog.conf
RCS/syslog.conf,v --> syslog.conf
revision 1.1
done

That copies the file back into the current directory. But we're not ready to start working on it just yet. First, let's check it out with a lock to prevent other people from making updates while we work on it.

root@catlin:/etc# co -l syslog.conf
RCS/syslog.conf,v --> syslog.conf
revision 1.1 (locked)
done

Now you can edit the file to your heart's content. Once you're finished with it, check it back in and unlock it with ci -u:

root@catlin:/etc# ci -u syslog.conf
syslog.conf,v <-- syslog.conf
new revision: 1.3; previous revision: 1.2
enter log message, terminated with single '.' or end of file:
>> Added remote logging to the new security log host
>> .
done

Be sure to use meaningful log entries, because you won't remember what you meant by "made a couple of edits" in six months, when your code starts doing strange things.

If you have trouble checking anything in or out, you've probably forgotten the -l (on co) or -u (on ci) switch at some point. You can usually start over by making a backup copy, then checking the file back out again, and copying it back, assuming that you've just edited syslog.conf:

root@catlin:/etc# ci -u syslog.conf
syslog.conf,v <-- syslog.conf
ci: syslog.conf,v: no lock set by root
root@catlin:/etc#

Uh-oh. Better back it up and start again from scratch:

root@catlin:/etc# cp syslog.conf syslog.conf.backup
root@catlin:/etc# co -l syslog.conf
syslog.conf,v --> syslog.conf
revision 1.4 (locked)
done
root@catlin:/etc# cp syslog.conf.backup syslog.conf
root@catlin:/etc# ci -u syslog.conf
syslog.conf,v <-- syslog.conf
new revision: 1.5; previous revision: 1.4
enter log message, terminated with single '.' or end of file:
>> commented out the FIFO line
>> .
done

We'll see some more fun things to do with RCS in the next couple of hacks.

Hack #24. Checking Out a Previous Revision in RCS

Save yourself with RCS revisions

Inevitably, you will break things so horribly that you have no hope of getting them back again. Suppose you've just done a complicated edit to your httpd.conf , and you get the error message that strikes fear (and occasionally heartburn) in the heart of admins everywhere:

root@catlin:/usr/local/apache/conf# ../bin/apachectl restart
apachectl restart: httpd not running, trying to start
apachectl restart: httpd could not be started

Oh, dear. What have you done? With the web server down, people aren't even getting your friendly 404 page, they're seeing Connection Refused errors. You could dig into httpd.conf desperately with vi to figure out exactly what you (or perhaps the admin before you) did to deserve this.

But if you're using RCS, things aren't so bleak. To see what has changed between your last checkout and this one, use rcsdiff :

root@catlin:/usr/local/apache/conf# rcsdiff httpd.conf
===================================================================
RCS file: RCS/httpd.conf,v
retrieving revision 1.1
diff -r1.1 httpd.conf
458c458
< ErrorLog /usr/local/apache/logs/error_log
---
> ErrorLog :wq/usr/local/apache/logs/error_log

There we are. Evidently, a :wq accidentally got inserted at line 458. Ah, must have missed that ESC key. Make your fix, start Apache, and check it back in.

But what if the changes were more extensive than that? Suppose you make a big change to a configuration file on Tuesday morning, and gremlins start to manifest themselves around Thursday afternoon. How can you get back to a known good copy?

Simple: check out a particular revision using the -r switch:

root@catlin:/etc# co -l -r1.2 syslog.conf
syslog.conf,v --> syslog.conf
revision 1.2
done

Presto, you're back to the state you were in when you checked the file in at rev 1.2. Remember with RCS, it makes sense to save early and often.

Hack #25. Tracking Changes with rcs2log

See at a glance who's been editing your files, and why

RCS really shows its strength when more than one person needs to edit a particular file. Having multiple admins can sometimes lead to a blame game when things aren't working properly. In these cases, it's obvious that somebody changed something, but nobody will admit to it. If you keep track of changes in RCS, then it's simple to tell who's been changing what:

root@www:/usr/local/apache/htdocs# rcs2log index.html
2002-08-14 rob <rob@mouse>

* index.html: meeting announcement

2002-07-30 rob <rob@mouse>

* index.html: gre.tunnel announcement

2002-07-12 sderle <sderle@mouse>

* index.html: added Marin and shuffled around a bit.

2002-07-10 rob <rob@mouse>

* index.html: meeting announcement + v0.81

2002-07-01 jim <jim@mouse>

* index.html: *** empty log message ***

2002-06-20 rob <rob@mouse>

* index.html: meeting reminder

Hmm, what's that empty log message doing there? It might be worth talking to Jim (and possibly doing an rcsdiff between it and the previous revision). But how can you tell which versions correspond to which log entries? Try the -v switch on rcs2log:

root@www:/usr/local/apache/htdocs# rcs2log -v index.html
2002-08-14 rob <rob@mouse>

* index.html 1.54: meeting announcement

2002-07-30 rob <rob@mouse>

* index.html 1.53: gre.tunnel announcement

2002-07-12 sderle <sderle@mouse>

* index.html 1.52: added Marin and shuffled around a bit.

2002-07-10 rob <rob@mouse>

* index.html 1.51: meeting announcement + v0.81

2002-07-01 jim <jim@mouse>

* index.html 1.50: *** empty log message ***

2002-06-20 rob <rob@mouse>

* index.html 1.49: meeting reminder

Now do a diff between whichever revisions you like:

root@www:/usr/local/apache/htdocs# rcsdiff -r1.49 -r1.50 index.html
===================================================================
RCS file: RCS/index.html,v
retrieving revision 1.49
retrieving revision 1.50
diff -r1.49 -r1.50
199a200,202
> <dt><a href="http://labs.google.com/">Google Labs</a></dt>
> <dd>Take a look at the strange and wondrous things that Google is up to...</dd>
>

So it was just a link addition. Still, it might be easy to just hit that ^D when prompted, but it's generally considered bad form to not make any log entry.

At any rate, once you master the simple ci, co, rcsdiff, and rcs2log commands, you have some very flexible revision control tools at your disposal. Use them well, and they'll probably save your life one day. Or in the very least, they'll save the next best thing: your data.

Hack #26.  Getting Started with CVS

The ins and outs of the Concurrent Versioning System

Concurrent Versioning System (CVS) is a system for managing simultaneous development of files. It is commonly used in large programming projects and is also useful to system administrators, technical writers, and anyone who needs to manage files.

CVS stores files in a central repository. It is set up to be accessible to all users of the files, using standard Unix permissions. Commands are given to "check out" a copy of a file for development and "commit" changes back to the repository. It also scans the files as they are moved to and from the repository, to prevent one person's work from overwriting another's.

This system ensures that a history of the file is retained, which is extremely useful when the boss decides he wants a feature you trashed months ago. It also ensures that backing up the repository is enough to backup a project (providing all necessary files are kept in repository).

CVS is usually used to help manage projects but can also be used for individual files.

Typical Uses

CVS is designed for developers, either individually or in teams. For individuals, CVS provides a repository from which you can work from home, the office, or the client site without having to haul disks around. It also provides version control, allowing rollbacks without loss of data. For teams, it also keeps a record of who changed which lines of a file and prevents direct overwriting of each other's work.

System administrators can keep configuration files in CVS. You can make a change, cvs commit it, test it. If it fails, roll back the change, even if you only discover the failure six months down the track.

Administrators can keep a CVS tree of the configurations for server farms. Adding a new server? Just use cvs to checkout the config tree for that type of server. Committing all changes also helps you keep track of who did what, when.

In this section, we'll take a look at what you need to get CVS up and running quickly, both as a client and as a server.

Creating a Repository

The repository needs to be hosted on a machine with sufficient disk space to store all your files and all the data for the changes you expect. As a rule of thumb, put the repository on a partition with enough room for three times the expected final size of the module. Then use the "Scotty principle" and double your estimate — the project will expand. If you intend to store binary files, multiply by ten. After your first project, you'll have a feel for how much space to allow.

First, ensure that all CVS users have valid user accounts and can access the repository machine from all the machines they intend to use.

Now create your repository root directory. Repositories are often stored in /home/cvsroot or /usr/local/cvsroot. Use cvs init to set up the directory as a CVS repository.

cvs -d /home/cvsroot init

Debian Linux has a script, cvs-makerepos , that will build a repository based on pre-existing Debian configuration scripts. See man cvs-makerepos for more information and man cvsconfig for an automated system for configuring a Debian CVS repository.

In general, most CVS servers tend to use a single repository, with multiple Modules contained within it. For example, your repository might live in /home/cvsroot, but could contain many projects (such as tools, email, dns, myproj, etc.).

Importing a New Module

Before loading your project into CVS, consider its structure. Moving or renaming files and directories can damage the CVS record of the files' history. Deleting a directory can cost you the record of all its files and subdirectories. For this reason, CVS has no facility for moving or renaming files and directories, or removing directories.

Make your initial directory structure — even if it's just one directory. Add any initial files you want. From within the root directory of your project, use the command cvs -d repository import nameofmodule vendortag releasetag.

For most cases, you will not need to know about vendor tags and release tags. CVS requires them to be present, but you can simply use the name of the module as the vendor tag, and the current version as the release tag.

/home/jenn$ cd example
/home/jenn/example$ cvs -d /home/cvsroot import example example_project ver_0-1

Environment Variables

If you are only working with one repository, and would like to save yourself some typing, set the $CVSROOT environment variable to the repository's full path, like this:

export CVSROOT=/home/cvsroot/

With $CVSROOT set, you can now omit the -d repository portion of any CVS command. If you need to temporarily work with another repository, either reset your $CVSROOT, or use the -d switch to override your existing $CVSROOT setting.

If the CVS repository resides on a different machine, then you need to tell CVS how to access it. For remote repositories, $CVSROOT is set to :method:user@host:path. It should look something like :ext:[email protected]:/home/cvs.

By default, CVS uses rsh to access a repository on a remote machine. While this may have been a very logical transport choice a few years ago, running rsh on a production server is now generally considered a Very Bad Idea. Fortunately, the ext access method tells CVS to look in the $CVS_RSH environment variable, and run that program to connect to the machine that contains the repository. Most people set the $CVS_RSH variable to ssh , and set up ssh client keys (as in [Hack #66]) on the repository machine. This greatly simplifies working with your repository, as you won't have to type in a password every time you make an update.

Incidentally, the other common access method is pserver, and these days is generally used for accessing anonymous public CVS repositories. For an example of how to work with anonymous CVS (and set up your own public repository), see [Hack #66].

See Also:

• CVS in a Nutshell (O'Reilly)
• Anonymous CVS
• Quick logins with ssh keys

Hack #27. CVS: Checking Out a Module

How to get a working copy of a CVS Module

CVS stores the files in a central repository, but users work from a working copy of a file.

Make a directory to do your work in (I tend to use ~/cvs), then cd into that directory. The checkout syntax is cvs checkout module To checkout a module called example, try cvs checkout example.

The checkout will put a copy of that module's files and subdirectories into your cvs directory.

cvs$ ls
example
cvs$ cd example; ls
CVS src
cvs/example$ cd CVS; ls
Entries Repository Root

The cvs directory is a special directory that CVS uses for its own purposes. CVS/Entries lists files and subdirectories CVS knows about. CVS/Repository contains the path to the corresponding directory in the repository. CVS/Root contains the path to the repository, so you won't need to use the -d repository-path option again for these files.

Note that CVS/Root overrides the $CVSROOT environment variable, so if you change the repository, you should check out the module again. Alternately, if you're in the middle of a big edit when you realize that your repository needs to be changed, try this perl one-liner, as shown in [Hack #73]:

cvs/src$ ls
CVS Makefile sample.h sample.c

The src directory contains the source files for the example project. sample.c, sample.h, and Makefile are ordinary files in the working copy. In the repository, they are stored in a format that tracks the changes.

Hack #28. CVS: Updating Your Working Copy

Receiving recent changes in your CVS Module

Every day before you start work, and any time someone else may have made and committed changes, cd into your working directory and run cvs update. This checks your working copies against the repository files and imports any changed files for you. cvs update -d also gives you any new directories.

Update reports on the status of each file as it checks it:

U

File updated successfully

A

File added but not yet committed (need to run a cvs commit)

R

File removed but not yet committed (need to run a cvs commit)

M

File modified in your working directory; the file in the repository was changed and your working directory file was older than the last time CVS checked it or the repository had changes that the system could safely merge

C

Conflict between the repository copy, and your copy, which requires human intervention

?

File the file is in your working directory but not the repository; CVS doesn't know what to do with it

Hack #29. CVS: Using Tags

Tagging a Module revision in CVS

Tags assign a symbolic name to a revision of a file or a set of files. cvs tag tags the repository versions of all the files in the current working directory and its subdirectories.

The cvs tag command is based on a timestamp. It assigns the symbolic name to the version of the file or directory that is closest to and older than the timestamp; it does not look at the working directory.

Note that CVS doesn't allow the `.' character in tags. Specify the filename after the tag name, as in the following example:

cvs tag tagname filename
cvs tag tagname

Otherwise, the command tags the repository versions of all files in the current working directory:

cvs tag -c tagname

Use the -c option to cause the process to abort if the repository copy differs from the working copy:

cvs/example$ cvs tag release-1-0 src/sample.c
cvs/example/src$ cvs tag release-1-0 
cvs/example/src$ cvs tag -c release-1-0

To retrieve a tagged version, use the -r flag to checkout or update. If you checkout or update in your usual working directory, the tagged version will overwrite the existing files, as in:

cvs checkout -r tagname
cvs update -r tagname

Here's an example:

cvs$ mkdir example-rel-1.0
cvs/example-rel-1.0$ cvs checkout -r release-1-0

or

cvs/example$ cvs update -r release-1-0

Hack #30. CVS: Making Changes to a Module

Checking in your changes in CVS

Once your files are checked out, edit them and compile them normally. Apply the updates to the repository with cvs commit. This command needs to be run higher in the hierarchy than all the files you have changed — you can always run it from the base of your working copy.

You can also cvs commit filename, which will commit a single file or recursively commit a directory.

Different project teams have different opinions on how often to do a cvs commit. Good rules of thumb include "every time you have a clean compile," and "every day before lunch and before you leave."

cvs/example$ cvs commit
cvs commit: Examining .
cvs commit: Examining src
[email protected]'s password:

CVS examines each directory and subdirectory below the current working directory. Any file that CVS knows will be checked for changes. If your cvs repository is not on the local machine, CVS will ask for a password for the remote machine, unless you have already set up your ssh host keys (as shown in [Hack #66]) and have eliminated the need for passwords.

CVS then opens whichever editor is the default in your environment — based on the $CVSEDITOR or $EDITOR environment variables. Add change-notes for the appropriate files, as in:

CVS:------------------------------------------------------------------
CVS: Enter Log. Lines beginning with 'CVS:' are removed automatically
CVS:
CVS: Committing in .
CVS:
CVS: Modified Files:
CVS: example/src/sample.h example/src/sample.c
CVS:------------------------------------------------------------------

I strongly recommend meaningful change-notes — if you're trying to do a rollback and all you have are messages that say "fixed a few bugs," you'll not know which version to roll back to without using cvs diff.

If there is a potential conflict, cvs commit fails. Correct this by running a cvs update on the repository — CVS will attempt to merge the files, and will ask for human help if it cannot do this without losing data:

cvs server: Up-to-date check failed for 'cvs_intro.html'
cvs [server aborted]: correct above errors first!
cvs commit: saving log message in /tmp/cvst7onmJ

Hack #31. CVS: Merging Files

Resolving update conflicts in CVS

If CVS can't merge a modified file successfully with the copy in the repository, it announces the conflict in the output of cvs update. The original file is stored in .#file.version in the file's working directory, and the results of the merge are stored as the original filename:

cvs/example$ cvs update
[email protected]'s password:
cvs server: Updating .
RCS file: /home/cvs/example/sample.c,v
retrieving revision 1.3
retrieving revision 1.4
Merging differences between 1.3 and 1.4 into sample.c
rcsmerge: warning: conflicts during merge
cvs server: conflicts found in sample.c
C sample.c

CVS writes the merge with the conflicting lines surrounded by CVS tags. CVS can't automatically merge conflicts where the same line is changed in both versions of a file:

<<<<<<< sample.c
Deliberately creating a conflict.
=======
Let's make a conflict.
>>>>>>> 1.4

Hack #32. CVS: Adding and Removing Files and Directories

Adding and removing files in your Module

Files that CVS doesn't know what to do with are reported with a question mark after the commit process and during a cvs update. They need to be added to the repository before CVS will recognize them.

Use cvs add filename to mark a new file for inclusion. CVS doesn't put the file in the repository until you do a cvs commit.

Directories are added with the same command. Files within a directory can't be added until the directory is added.

Removing Files

To mark a file for removal from the working copies, use cvs remove filename. Before CVS will remove a file from the repository, you have to actually delete it from the filesystem. CVS doesn't actually remove the file entirely, it puts the file in a special subdirectory in the repository called Attic.

Removing Directories

Directories cannot be removed from the repository using CVS commands. If a directory is no longer required, empty the directory with cvs remove, and use cvs update -P and cvs checkout -P when retrieving a working copy. The -P flag ensures that empty directories are not retrieved.

If you must, you can remove a directory by using rmdir on the repository. Do this on a copy of the repository first and check that you aren't breaking anything: If the directory in the repository has an Attic subdirectory, you will lose archived copies of files formerly stored there.

If you remove a directory from the repository, you should have all your users remove their existing working copies, and check out fresh copies of the module.

Hack #33. CVS: Branching Development

Setting up a development branch in CVS

If you need to fix a bug in an older version of your code without changing current code, or modify a configuration set for staging servers without modifying the set for your production servers, you might need to branch your modules. A branch allows storage and retrieval of a variation of the main module, without affecting the main module. Changes on the branch can be merged in later. To make a branch, use cvs tag -b branchtag, as in:

cvs/example$ cvs tag -b release-1-0-patches

Retrieve a branch using either checkout or update. Checkout will create a new directory for the branch, and update will overwrite your current working directory with the branch.

            cvs checkout -r branchtag
cvs update -r branchtag

Example:

cvs/example-rel-1.0$ cvs checkout -r release-1-0-patches
cvs/example$ cvs update -r release-1-0-patches

Branches can be merged back into the main trunk, using the conflict resolution system invoked by cvs update and cvs commit.

            cvs checkout module
cvs update -j branchtag

Example:

/tmp/example$ cvs checkout example
/tmp/example$ cvs update -j release-1-0-patches

Or in a single command:

            cvs checkout -j branchtag module

The following example:

/tmp/example$ cvs checkout -j release-1-0-patches example

resolves any conflicts the system reports, then use cvs commit.

Hack #34. CVS: Watching and Locking Files

Setting up an email watch on files in CVS

Unlike many versioning systems, CVS doesn't have file locking — it doesn't prevent simultaneous editing of files. However, you can set files to be watched, so CVS will mail watchers when a file is being edited. If files are being watched, developers need to use cvs edit and cvs unedit to release a file for editing. Unwatched files can be edited without notifying CVS in any way.

To set up files for being watched, use:

            cvs watch on (files)
cvs watch off (files)

To set yourself as a watcher, use:

            cvs watch add (files)
cvs watch remove (files)

or:

            cvs watch add -a edit|unedit|commit|all (files)
cvs watch remove -a edit|unedit|commit|all (files)

The special CVS file notify determines what occurs when a watched file is changed. It defaults to sending mail to the user's username on the CVS server. If your users have other addresses, set up the file users in the repository's CVSROOT directory. Entries should be in the format user:email, one to a line.

jenn:[email protected]

Hack #35. CVS: Keeping CVS Secure

Protecting your users and your code base in CVS

Remote Repositories

If the repository is on the local machine, both access and security are fairly straightforward. You can set the $CVSROOT environment variable to the root directory of the CVS repository, or call checkout with the -d directory option.

If the repository is on a remote machine, it is necessary to tell CVS which machine it is on, and what method will be used to access the machine. There are several methods available, but for security and simplicity, I prefer to use SSH. The syntax for defining a remote $CVSROOT is :method:[[user]:[password]@]hostname[:[port]]:/path/to/repository. For example:

:ext:[email protected]:/usr/local/cvsroot

(Note that info cvs disagrees slightly with what my copy of CVS actually does. I have included the syntax that works for me — a colon between the host and the path. Use the syntax that works on your system.)

To use SSH, we use the ext method. This method uses an external-to-CVS rsh or rsh-compatible program to communicate with the CVS server. To tell CVS to use SSH instead of rsh, set the environment variable $CVS_RSH to SSH. Ensure that SSH is set up on the server and on all clients, that SSH keys are generated and that users have usernames and passwords on both machines. If the usernames are the same, the user@ part of the CVSROOT string is not necessary. If a standard SSH port is used, the port is not necessary.

               cvs -d :ext:cvs.example.com.au:/usr/local/cvsroot checkout sample<code>

Permissions

The files in the repository are all read-only. Permissions to those files shouldn't be changed. To control access, use the directory permissions. Most administrators make a group for the people who should have access to the module, and ensure that the group has write access for the directory.

If using a remote repository, set the root directory of the module setgid to ensure that all directories beneath it are made with the correct permissions. If using a local repository, $CVSUMASK can be set to control the permissions of files and directories in the repository.

Developer Machines

Securing the project involves securing the repository and securing all checked out copies — typically your developer's machines. It's not enough to ensure that the repository is safe and all transmissions are properly encrypted if someone can walk into your developer's office on his day off and burn a CD of your code. Maintain the usual physical and Net-based security for your development machines, prototype and demonstration copies, and any other places the code gets checked out to.

Hack #36. CVS: Anonymous Repositories

Create your own read-only anonymous CVS repository

Creating an Anonymous Repository

The pserver access method allows users to login to remote repositories by supplying a username and password that is checked against a password file that CVS maintains, or against the system's /etc/passwd. The pserver method unfortunately passes all credentials in the clear, so at best an intervening snooper might capture a login to your CVS repository, or at worst could compromise the entire machine. For this reason, most people use ssh as their transport when using remote repositories. Please see [Hack #35], for further details.

Obviously, if you want to provide read-only access to your source tree to the public, using ssh as your transport would be unnecessary (and impractical). This is where pserver shows its real usefulness: allowing easy anonymous repository access.

Before we get anonymous CVS running, first we'll need to set the repository machine up to use the traditional pserver method.

Installing pserver

As we'll be using pserver for anonymous CVS access, we'll need to create a user that has no permissions to write to anything in your repository. Create a user called anonymous (or if 9-letter usernames bother you, cvsanon is another common choice.) Set its shell to /bin/true, its home directory to something innocuous (like /var/empty), put it in its own group, and lock its password (a passwd -l is a quick way to do that.) This user will never login; it's just a placeholder account for CVS to setuid to later.

Next we'll create a password file for CVS. Put the following in a file called CVSROOT/passwd, under your repository directory:

anonymous:23MLN3ne5kvBM

If you created a user account called cvsanon, use this line instead:

anonymous:23MLN3ne5kvBM:cvsanon

In the CVS passwd file, the left hand entry is the CVS login name, followed by the encrypted password, and finally ending in an optional system login name to map the CVS login to. If it is omitted, CVS will look up the first entry in the system's password file and use that. The encrypted string is the word anonymous.

To be absolutely sure that the anonymous user can't ever make changes to the repository, add the line anonymous to the CVSROOT/readers file under your repository. This file flags any users contained with in it (one per line) as read-only users.

Now we want to tell CVS to never accept regular system users under the pserver method (to prevent wayward users from habitually using their system logins with pserver.) This is set up in CVSROOT/config, under your repository directory. Uncomment the line that says SystemAuth=no and then only users specified in CVSROOT/passwd can login using pserver. Note that this will have no effect on CVS users that use ext and ssh; they still use simple filesystem permissions for access control, and never consult the CVS passwd file.

Finally, we can tell the system to actually accept pserver connections. CVS doesn't ever run as a daemon; it expects to be launched from inetd. It runs on port 2401, so add the following line to your /etc/services:

pserver 2401/tcp

And add this entry to /etc/inetd.conf :

pserver stream tcp nowait root /usr/bin/cvs cvs --allow-root=/usr/local/cvsroot pserver

Substitute your repository's directory for /usr/local/cvsroot. Now just skill -HUP inetd (and see [Hack #23] if you don't have skill installed) and away you go.

Using a Remote pserver

To test your new anonymous repository, first set your $CVSROOT environment variable to this:

:pserver:[email protected]:/usr/local/cvsroot

Before you can do a cvs checkout, you'll first need to login to pserver with cvs login. When prompted for a password, enter anonymous. Now proceed with cvs checkout module, and you should see an update as normal. Verify that you can't perform a cvs checkin, and you're ready to publish your anonymous CVS details to the world.

Chapter 3. Backups

Hacks #37-44

Be assured that one day, the unthinkable will happen. Hard drives wear. Tapes stretch and break. You might even accidentally pass the wrong switch to a command (or be in the wrong directory, or on the wrong machine) and suddenly find yourself cursing at having hit the Enter key, as your data evaporates into the ether. It is at these moments that you learn the value of a properly designed and executed backup solution.

While we can get you started with some interesting approaches to backing up your data, this section is by no means comprehensive. As a systems administrator, you should design a backup plan that fits the needs of your organization, implement the plan, and continually revise it as necessary as time goes on. Definitely do not assume that just because you set up a cron job to run a backup that the backups themselves are good (or even that the job itself is running properly). Watch your logs. Buy new backup media before the old media develops problems. Make it a policy to test your backups by doing a full restore from backup as often as you deem necessary to sleep well at night. And even then, consider keeping another copy of your data, off-site if need be. For an extremely detailed look at everything that goes into building a secure data backup policy, be sure to check out Unix Backups & Recovery by W. Curtis Preston (O'Reilly).

In this section, we'll take a look at some methods for keeping copies of your data ready for when that day of reckoning comes. Every installation is different, and each site needs its own degree of backup functionality (ranging from weekly differentials to 5-minute snapshots). While we won't try to define the perfect backup policy for your site, we will see some tools you can use to develop your own.

Hack #37. Backing Up with tar over ssh

Copy arbitrary bits of the filesystem between servers using ssh and tar

Shuffling files between servers is simple with scp :

root@inky:~# scp some-archive.tgz blinky:/

Or even copying many files at once:

root@pinky:~/tmp# scp clyde:/usr/local/etc/* .

But scp isn't designed to traverse subdirectories and preserve ownership and permissions. Fortunately, tar is one of the very early (and IMHO, most brilliant) design decisions in ssh to make it behave exactly as any other standard Unix command. When it is used to execute commands without an interactive login session, ssh simply accepts data on STDIN and prints the results to STDOUT. Think of any pipeline involving ssh as an easy portal to the machine you're connecting to. For example, suppose you want to backup all of the home directories on one server to an archive on another:

root@inky~# tar zcvf - /home | ssh pinky "cat > inky-homes.tgz"

Or even write a compressed archive directly to a tape drive on the remote machine:

root@blinky~# tar zcvf - /var/named/data | ssh clyde "cat > /dev/tape"

Suppose you wanted to just make a copy of a directory structure from one machine directly into the filesystem of another. In this example, we have a working Apache on the local machine but a broken copy on the remote side. Let's get the two in sync:

root@clyde:~# cd /usr/local
root@clyde:/usr/local# tar zcf - apache/ \  
| ssh pacman "cd /usr/local; mv apache apache.bak; tar zpxvf -"

This moves /usr/local/apache/ on pacman to /usr/local/apache.bak/, then creates an exact copy of /usr/local/apache/ from clyde, preserving permissions and the entire directory structure. You can experiment with using compression on both ends or not (with the z flag to tar), as performance will depend on the processing speed of both machines, the speed (and utilization) of the network, and whether you're already using compression in ssh.

Finally, let's assume that you have a large archive on the local machine and want to restore it to the remote side without having to copy it there first (suppose it's really huge, and you have enough space for the extracted copy, but not enough for a copy of the archive as well):

root@blinky~# ssh pinky "cd /usr/local/pacland; tar zpvxf -" \  
< really-big-archive.tgz

Or alternately, from the other direction:

root@pinky:/usr/local/pacland# ssh blinky "cat really-big-archive.tgz" \  
| tar zpvxf -

If you encounter problems with archives created or extracted on the remote end, check to make sure that nothing is written to the terminal in your ~/.bashrc on the remote machine. If you like to run /usr/games/fortune or some other program that writes to your terminal, it's a better idea to keep it in ~/.bash_profile or ~/.bash_login than in ~/.bashrc, because you're only interested in seeing what fortune has to say when there is an actual human being logging in and definitely not when remote commands are executed as part of a pipeline. You can still set environment variables or run any other command you like in ~/.bashrc, as long as those commands are guaranteed never to print anything to STDOUT or STDERR.

Using ssh keys to eliminate the need for passwords makes slinging around arbitrary chunks of the filesystem even easier (and easily scriptable in cron, if you're so inclined). Take a look at the hacks listed below for an example of how to make it very easy (and secure) to connect to any of your servers with ssh.

See also:

• [Hack #66]
• [Hack #67]
• [Hack #68]

Hack #38. Using rsync over ssh

Keep large directory structures in sync quickly with rsync

While tar over ssh is ideal for making remote copies of parts of a filesystem, rsync is even better suited for keeping the filesystem in sync between two machines. Typically, tar is used for the initial copy, and rsync is used to pick up whatever has changed since the last copy. This is because tar tends to be faster than rsync when none of the destination files exist, but rsync is much faster than tar when there are only a few differences between the two filesystems.

To run an rsync over ssh, pass it the -e switch, like this:

root@rover:~# rsync -ave ssh greendome:/home/ftp/pub/ /home/ftp/pub/

Notice the trailing / on the file spec from the source side (on greendome.) On the source specification, a trailing / tells rsync to copy the contents of the directory, but not the directory itself. To include the directory as the top level of whatever is being copied, leave off the /:

root@village:~# rsync -ave ssh bcnu:/home/six .

This will keep a copy of the ~root/six/ directory on village in sync with whatever is present on bcnu:/home/six/.

By default, rsync will only copy files and directories, but not remove them from the destination copy when they are removed from the source. To keep the copies exact, include the -- delete flag:

six@jammer:~/public_html# rsync -ave ssh  -- delete greendome:~one/reports .

Now when old reports are removed from ~one/reports/ on greendome, they're also removed from ~six/public_html/reports/ on jammer, every time this command is run. If you run a command like this in cron, leave off the v switch. This will keep the output quiet (unless rsync has a problem running, in which case you'll receive an email with the error output).

Using ssh as your transport for rsync traffic has the advantage of encrypting the data over the network and also takes advantage of any trust relationships you already have established using ssh client keys. For keeping large, complex directory structures in sync between two machines (especially when there are only a few differences between them), rsync is a very handy (and fast) tool to have at your disposal.

See also:

• man rsync
• [Hack #66]
• [Hack #68]
• [Hack #42]

Hack #39. Archiving with Pax

Make easy, portable archives using pax

pax stands for "portable archive exchange," as it was designed specifically to allow portability between different versions of Unix. There's also a bit of wry humor in the name, as pax attempts to bring some "peace" to the long-standing battle over which is better: tar or cpio. The pax utility can be used to create either type of archive, and during a restore, it automatically detects the type of archive for you. We'll start with some examples of basic pax usage, then move on to some fancier stuff.

Creating Archives

To back up the contents of your home directory, invoke write mode using the w switch:

cd
pax -wf home.pax .

In this example, I went to my home directory with cd, then told pax to write (w) to a file (f) named home.pax the contents of the current directory (.). When you use pax, it's very important to remember to include that f switch to indicate the name of the archive you'd like to create. If you forget the f, weird characters will be sent to your screen, accompanied by horrible, pained noises. Also, if you want to watch as pax does its thing, simply add the v, or verbose, switch to the switch portion of the command.

To see what type of file you've just created, use the file command:

file home.pax
home.pax: POSIX tar archive

To see the contents of that archive, tell pax which archive file you'd like to view, using the f switch:

pax -f home.pax |more

Since my archive file is rather large, I piped this output to the more command so I could read the contents of the archive one page at a time. If you also include the v switch, you'll get an ls -l-type output of the archive contents. Again, don't forget to specify the name of the archive with the f switch, or nothing will happen, except that you'll lose your prompt until you press Ctrl-c.

Expanding Archives

To restore (or use read mode on) an archive, first cd into the destination directory, then use the r switch. For example, I'll restore the backup named home.pax into the test subdirectory of my home directory:

cd test
pax -rvf ~/home.pax

The pax utility can also restore tar and cpio archives. It is able to automatically detect the correct format for you; however, you should use the file utility before attempting the restore to determine whether or not the archive is compressed. If it is, you'll need to include the z switch.

As an example, I have a file called backup.old located in my home directory (~). I'll first use the file utility:

file backup.old
backup: gzip compressed data, deflated, last modified:
Sat Aug 17 14:21:12 2002, os: Unix

Since this backup is compressed, I'll use this command to restore it to the test directory:

cd test
pax -rvzf ~/backup.old

I have another file in my home directory called backup:

file ~/backup
backup: cpio archive

This file isn't compressed, so i'll restore it, like so:

pax -rvf ~/backup

The fact that the first backup happened to be a tar archive and the second a cpio archive didn't confuse pax; however, I would have received some strange error messages if I had forgotten to inform pax that the first archive was compressed.

Interactive Restores

You can do some pretty funky things when restoring with pax. For example, you can do an interactive rename/restore by including the i switch. Issuing the following command:

pax -rif ~/backup

will start an interactive restore of the archive named backup into the current directory. In interactive mode, pax will display the name of each file, one at a time, and prompt you to either rename it as it's restored, restore it with the original name, or to skip it and not restore it:

ATTENTION: pax interactive file rename operation.
drwxr-xr-x Aug 17 15:08 .
Input new name, or a "." to keep the old name, or a "return" to skip this file.
Input >
Skipping file.

Here, I pressed Enter as I didn't want to change the name of "." or the current directory:

ATTENTION: pax interactive file rename operation.
drwxr-xr-x Jul 26 16:10 file1
input new name, or a "." to keep the old name, or a "return" to skip this file.
input > old
Processing continues, name changed to: old

ATTENTION: pax interactive file rename operation.
-rw-r--r-- Jun 11 00:20 file2
input new name, or a "." to keep the old name, or a "return" to skip this file.
input > .
Processing continues, name unchanged.

You'll note that I changed the name of file1 to old and kept file2 as is. A listing of the restored directory will show two files: one named old and one named file2.

Recursively Copy a Directory

One of the most powerful features of pax is that it is able to very quickly copy a complete directory structure to another portion of your hard drive, using copy mode. To use copy mode:

1. cd into the source directory
2. Ensure the destination directory exists; if it doesn't, use mkdir to create it
3. Issue this command: pax -rw . destination_directory

Note that you don't include the f switch in copy mode, as an archive file doesn't get created. Instead, the old directory structure is directly recreated into the new directory structure. This can be much easier to remember than the tar equivalent:

tar cf - . | (cd destination_directory; tar vpxf -)

Also note that you never want to do this:

cd
mkdir test
pax -rw . test

In the previous example, I cd'd into my home directory, made a subdirectory named test, then invoked copy mode. In doing so, I ended up in an endless loop of test subdirectories, each containing the contents of my home directory. If I hadn't interrupted this cycle with a Ctrl-c, pax would have continued ad infinitum, where infinitum is defined as the point where I run out of disk space. That's what this section of man pax refers to.

Warning: the destination directory must not be one of the file operands or a member of a file hierarchy rooted at one of the file operands. The result of a copy under these conditions is unpredictable.

However, this works beautifully and almost instantaneously:

su
Password:
cd ~user1/big_project
mkdir ~user2/big_project
chown user2 ~user2/big_project
pax -rw . ~user2/big_project

Voila, the entire big_project directory structure is now also in the second user's home directory. When using copy mode, you'll have to become the superuser as you'll be copying out of your home directory, so you can avoid the endless loop situation. If you have to make the new directory, it will be owned by root; if need be, use the chown command like I did to ensure that it has the desired ownership before doing the copy operation. You'll also want to take a look at man pax first to see how you want to handle the permissions of the copied directory structure.

It is also possible to interactively copy a directory structure by including the i switch:

pax -rwi . ~user2/big_project

Similarly to the previous interactive example, pax will display each filename, one at a time, so you can decide which files to copy over and which files to rename as you do so.

Incremental Backups

Now, let's do something useful with the pax command. I'll demonstrate how to create an incremental backup system. In this example, the user "genisis" would like to back up any changes she made to her home directory on a daily basis.

First, I'll become the superuser to create a directory to hold the backups:

su
Password:
mkdir /usr/backups

I'll then create a subdirectory and give the user "genisis" ownership of that subdirectory:

mkdir /usr/backups/genisis
chown genisis /usr/backups/genisis

I'll then leave the superuser account and as the user "genisis," cd into my home directory:

exit
cd

I'll then do a full backup of my home directory and save it to an archive file called Monday:

pax -wvf /usr/backups/genisis/Monday .

Now that I have a full backup, I can take daily incremental backups to just back up each day's changes. So when I'm finished with my work on Tuesday, I'll issue this command:

pax -wv -T 0000 -f /usr/backups/genisis/Tuesday .

Notice that I included the time switch (-T) and specified a time of midnight (0000). This tells pax to only back up the files that have changed since midnight, so it will catch all of the files that changed today. On Wednesday, I'll repeat that command but will change the archive name to Wednesday.

If you have the disk space and want to keep backups for longer than a week, modify your archive names to something like: Aug01, Aug02, etc. It's still a good idea to do a full backup once a week, followed by incremental backups the other days of that week. If disk space is an issue, include the z switch so the backups will be compressed. Also note that the T switch can be much pickier than I've demonstrated; see man pax for the details.

Skipping Files on Restore

To restore all of the files except file3, use this command:

pax -rvf ~/backup -c './file3'

The c switch is the exception switch. Note that your exception pattern (in my case, file3) needs to be enclosed in single quotes (the key next to your Enter key). Either use the literal pattern like I did (to pax, this file is known as ./file3, not file3) or use a wildcard, like so:

pax -rvf ~/backup -c '*file3'

If you use a wildcard (*) at the beginning of your pattern as in the above example, you will exclude all files that end with "file3" — for example: file3, myfile3, thatfile3.

You can also specify which file to restore by using the n, or pattern matching, switch. The following will just restore file2:

pax -rvf ~/backup -n './file2'

The n switch differs from the c switch in that it will only restore the first file that matches the pattern. This means that this command will not restore file3, myfile3, and thatfile3:

pax -rvf ~/backup -n '*file3'

Since file3 is the first file to match the expression, it will be the only file that will be restored.

The c and n switches are also useful when creating an archive; use them to specify which file you'd like to back up, or which file(s) you don't want to back up.

See also:

• Original article: http://www.onlamp.com/pub/a/bsd/2002/08/22/freeBSD_Basics.html
• pax source code (part of the ast-open package), http://www.research.att.com/sw/download/

Hack #40. Backing Up Your Boot Sector

Keep a copy of your boot sector packed away for a rainy day

Installing a boot loader (such as LILO) can be trickier than one might like. Particularly when using IDE hardware, it's easy to get yourself into trouble and work a system into a state that makes it impossible to boot without a rescue disk.

One common mistake when using IDE hardware is to install a kernel on a partition that extends beyond the 1024th cylinder. The symptom is very strange, because a machine will boot fine at first, but installing a kernel later (after the machine has been used for some time) makes LILO throw an error and will refuse to come up on the next boot. This can be a very confusing symptom, since "it used to work." Most likely, the kernel that was installed when the system was built the first time happens to reside on a space on disk before cylinder 1024. After the system software and user data are installed, the disk begins to fill up. When the disk contains about 500 MB (or 1GB on some BIOS) of data, any new kernels will necessarily lie (at least in part) beyond cylinder 1024 — which is inaccessible to the BIOS at boot time.

Modern versions of LILO will refuse to install such a kernel, but some older versions simply throw a warning, and install anyway. One way to deal with this is to make a small (say, 10 MB) partition at the beginning of your install process (as /dev/hda1, the first partition on the disk), and mount it under /boot. Now when you install new kernels, always copy them to /boot, and they are guaranteed to work with your BIOS, as they will necessarily be contained well before the 1024 cylinder limit.

At any rate, installing the boot loader shouldn't be taken lightly. Once you have your boot loader up and working as you want it, you should consider making a backup of the entire boot sector of your boot drive.

For IDE:

dd if=/dev/hda of=bootsector.bin bs=512 count=1

For SCSI:

dd if=/dev/sda of=bootsector.bin bs=512 count=1

Alternately, you can copy the boot sector directly to a floppy disk:

dd if=/dev/hda of=/dev/fd0 bs=512 count=1

Be extremely careful when specifying your if and of statements, as confusing them could obliterate your boot sector. It's probably a better idea to create a file and copy it to removable media as you would any other file. Trusting a floppy disk with critical system data may be a fun party trick, but after your first (or thousandth) floppy fails at a critical moment, you'll wish you had another copy.

Now if you ever end up with a munged boot sector, you can restore it quickly (after booting from rescue media) like this:

dd if=bootsector.bin of=/dev/hda

Or if you backed it up to floppy directly, against medical advice:

dd if=/dev/fd0 of=/dev/hda bs=512 count=1

Naturally, substitute sda for hda if you're using SCSI.

See also:

• Lilo's README (or other format of documentation under doc/ from the lilo distribution)

Hack #41. Keeping Parts of Filesystems in sync with rsync

Use rsync over ssh to mirror exactly what you want, to any number of servers

For O'Reilly's web publishing system, we built a web "cluster" that spreads the work of one machine across several, increasing both performance and reliability. But how do you keep the actual data in the filesystem of each server in sync with each other?

One method is to use NFS to provide a common filesystem for your data. While making it simple to update content across all of the servers simultaneously, NFS has some significant drawbacks. File locking on NFS is notoriously tricky. Performance tuning on NFS is generally considered something of a black art. (When it works, it's very good, and when it doesn't . . . that's why we're on call, right?) But probably the biggest drawback to a monolithic NFS server is that it introduces a single point of failure for your entire enterprise. If the NFS server becomes unavailable (or overloaded) for any reason, all of the machines dependent on it will suffer.

One alternative approach is to asynchronously update servers using a tool such as rsync. With the cost of disk space at an all-time low, having a locally cached copy of your data makes sense: not only will the single point of failure be eliminated, but the files will be served much faster from the local disk than they could be from network storage.

It is very straightforward to add an rsync job to cron on each of the web servers, containing something like this:

rsync -ae ssh master.machine.com:/usr/local/apache/htdocs/ \
  /usr/local/apache/htdocs/

Assuming that you have your ssh keys set up in advance (see [Hack #66]), this will update the local Apache document root with the current copy on master.machine.com over an encrypted ssh session. As long as updates are always made to master.machine.com, they will be copied faithfully to each of your other servers on the next pass.

The biggest drawback to this approach is that updates must happen asynchronously. If this job runs every five minutes, then the copy will be at most five minutes old (and about three minutes old, on average). As long as your application can tolerate the "be there in a minute" nature of rsync'ing large file structures, this method can buy you huge wins in terms of actual server performance and reliability. One strong caveat to using a bare rsync in cron is that you must take care that the job finishes before the next one is run. Otherwise, if a server gets particularly busy, it can enter a downward spiral where launching a new rsync drives up the load, which makes the running rsync take longer, which means that it is still running when the next one runs, which drives up the load.

Using rsync has a couple of advantages over using tar for this application. Using tar over ssh (see [Hack #37]) is generally quicker when making a full copy of a directory tree from scratch, but rsync is much faster when only a few files have changed on the master copy. It runs an analysis pass first to determine which files have been updated and then only transfers the changes.

But suppose that our web cluster has more complicated requirements. Assume that we are attempting to divide the load (see [Hack #99]) of serving pages into two groups: Application Servers (which run a heavy mod_perl-enabled Apache, driving our content management system) and lightweight front-end Apache servers, which only serve copies of static data (such as images and file archives). In this case, it would be a waste of space to copy the entire document tree to all of the servers, since the front-end machines will only ever serve a few particular types of files.

Using the exclude-from feature of rsync, you can supply a file that specifies what parts of the filesystem rsync will consider when syncing. Create a file that looks something like this in /usr/local/etc/balance.front for your static servers:

### Stuff we definitely don't want to mirror.
#
- logs/

### the entire document root
#
+ /usr/
+ /usr/local/
+ /usr/local/apache/
+ /usr/local/apache/htdocs/
+ /usr/local/apache/htdocs/**

### user public_html directories
#
+ /home/
+ /home/*/
+ /home/*/public_html/
+ /home/*/public_html/**

# Images, archives, etc.
#
+ *.jpg
+ *.gif
+ *.png
+ *.pdf
+ *.mp3
+ *.zip
+ *.tgz
+ *.gz

# Exclude everything else.
#
- *

And create a similar file for syncing to your Application Servers:

### Stuff we definitely don't want to mirror.
#
- logs/
- *.tmp
- *.swp

### the entire document root
#
+ /usr/
+ /usr/local/
+ /usr/local/apache/
+ /usr/local/apache/htdocs/
+ /usr/local/apache/htdocs/**

# Exclude everything else.
#
- *

Now, create a list of each type of server (front and back), one machine per line. Call the files /usr/local/etc/servers.front and /usr/local/etc/servers.back, respectively.

For example, put this in servers.front:

tiberius
caligula

and this in servers.back:

augustus
claudius
germanicus
posthumous
castor

Finally, rather than calling rsync directly from cron on each of your web servers, try this bit of shell code in cron on the authoritative master machine.

Listing: Balance-push.sh

#!/bin/bash

#
# balance-push - Push content from the master server (localhost)
# to multiple front- and back-end servers, in parallel.
#

# $FRONT_END lists the servers that receive the front-end (e.g. static content) updates.
#
FRONT_END=$(cat /usr/local/etc/servers.front)

# $BACK_END lists the hosts that receive the full back-end (e.g. everything) updates.
#
BACK_END=$(cat /usr/local/etc/servers.back)

# $TARGET specifies the filesystem root on the remote host to push to.
# Normally, you want this to be /, unless you're doing testing.
#
TARGET=/

# $EXCLUDE specifies the prefix of the per-mode rsync exclude files.
# For example, if your exclude files are /usr/local/etc/balance.front and
# /usr/local/etc/balance.back, set this to "/usr/local/etc/balance". The
# per-mode extensions will be added.
#
EXCLUDE=/usr/local/etc/balance

# $LOCK_DIR specifies a path to put the lock files in.
#
LOCK_DIR=/var/tmp

######## Ignore the shell functions behind the curtain. ########

PATH=/bin:/usr/bin:/usr/local/bin

lock (  ) {
local lockfile="$LOCK_DIR/balance.$1.lock"
if [ -f $lockfile ]; then
if kill -0 $(cat $lockfile); then
echo "$0 appears to be already running on $1."
echo "Please check $lockfile if you think this is in error."
exit 1
else
echo "$0 appears to have completed for $1 without cleaning up its lockfile."
fi
fi
echo $$ > $lockfile
}

unlock (  ) {
rm -f $LOCK_DIR/balance.$1.lock
}

push_files (  ) {
local mode=$1 host=$2

if [ ! "$mode" -o ! -r "$EXCLUDE.$mode" ]; then
echo "$0 $$: mode unset for $host!"
return
fi

if [ ! "$host" ]; then
echo "$0 $$: host unset for push $mode!"
return
fi

lock $host

rsync --archive --rsh=ssh --delete --ignore-errors --whole-file \
--exclude-from="$EXCLUDE.$mode" / ${host}:${TARGET}

unlock $host
}

push_tier (  ) {
local mode=$1 host_list=$2

for host in $host_list; do
$SHELL -c "push_files $mode $host" &
done
}

export -f lock unlock push_files
export TARGET EXCLUDE LOCK_DIR PATH

[ "$FRONT_END" ] && push_tier front "$FRONT_END"
[ "$BACK_END" ] && push_tier back "$BACK_END"

#
# Fin.
#

This script (call it balance-push) will manage your rsync jobs for you, ensuring that servers don't "lap" themselves on each run. It will push the proper files to each group, depending on whatever you specify in the files in /usr/local/etc/. If it finds a server that hasn't finished its last run, it will continue to skip the server until it has finished (and issue you an email to that effect via cron's MAILTO feature).

The load on your master will likely go up a bit, depending on how many machines you're syncing to (as each server requires both an rsync and an ssh session.) But practically speaking, on a production network serving millions of hits a day, the load introduced to each individual web server is negligible.

Hack #42. Automated Snapshot-Style Incremental Backups with rsync

Use rsync to create fast, small, and safe snapshots of your filesystem

Here is a method for generating automatic rotating snapshot-style backups on a Linux server. Snapshot backups are a feature of some high-end industrial strength file servers; they create the illusion of multiple full (up to ownership/permission) backups per day without the space or processing overhead. All of the snapshots are read-only and are accessible directly by users as special system directories.

Since making a full copy of a large filesystem can be a time-consuming and expensive process, it is common to make full backups only once a week or once a month, and store only changes on the other days. This technique is called making "incremental" backups, and is supported by the venerable old dump and tar utilities, along with many others.

The standard GNU fileutils cp command comes with a -l flag that causes it to create (hard) links instead of copies (it doesn't hard-link directories, though, which is good; you might want to think about why that is). Another handy switch for the cp command is -a (archive), which causes it to recurse through directories and preserve file owners, timestamps, and access permissions.

Together, the combination cp -al makes what appears to be a full copy of a directory tree but is really just an illusion that takes almost no space. If we restrict operations on the copy to adding or removing (unlinking) files — i.e., never changing one in place — then the illusion of a full copy is complete. To the end-user, the only differences are that the illusion-copy takes almost no disk space and almost no time to generate.

We can combine rsync and cp -al to create what appear to be multiple full backups of a filesystem without taking multiple disks' worth of space, as in:

rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
cp -al backup.0 backup.1
rsync -a --delete source_directory/ backup.0/

If the above commands are run once every day, then backup.0, backup.1, backup.2, and backup.3 will appear to each be a full backup of source_directory/ as it appeared today, yesterday, two days ago, and three days ago, respectively -- complete, except that permissions and ownerships in old snapshots will get their most recent values (thanks to J.W. Schultz for pointing this out). In reality, the extra storage will be equal to the current size of source_directory/ plus the total size of the changes over the last three days -- exactly the same space that a full plus daily incremental backup with dump or tar would have taken.

This method is much better for network-based backups, since it's only necessary to do a full backup once, instead of once per week. Thereafter, only the changes need to be copied. Unfortunately, you can't rsync to a tape; you'll still need dump or tar for that.

If you have a spare machine, even a very low-end one, you can turn it into a dedicated backup server. Make it standalone, and keep it in a physically separate place — another room or even another building. Disable every single remote service on the backup server, and connect it only to a dedicated network interface on the source machine.

You can then perform backups from this machine using rsync over ssh (see [Hack #41]), and export the backups back to the original machine via read-only NFS. Then users can get to the snapshots themselves (without needing sysadmin intervention) and can't possibly change or delete them.

I'd consider this "pretty good" protection, but if you're (wisely) paranoid, or your job is on the line, build two backup servers. Then you can make sure that at least one of them is always offline.

Extensions: Hourly, Daily, and Weekly Snapshots

With a little bit of tweaking, you can make multiple-level rotating snapshots. On my system, for example, I keep the last four "hourly" snapshots (which are taken every four hours) as well as the last three "daily" snapshots (which are taken at midnight every day). You might also want to keep weekly or even monthly snapshots too, depending upon your needs and your available space.

I keep one script that runs every four hours to make and rotate hourly snapshots, and another script that runs once a day rotate the daily snapshots. There is no need to use rsync for the higher-level snapshots; just cp -al from the appropriate hourly one.

To make the automatic snapshots happen, I have added the following lines to root's crontab :

0 */4 * * * /usr/local/bin/make_snapshot.sh
0 13 * * * /usr/local/bin/daily_snapshot_rotate.sh

They cause make_snapshot.sh to be run every four hours on the hour and daily_snapshot_rotate.sh to be run every day at 13:00 (that is, 1:00 PM). Those scripts are included below.

Listing: make_snapshot.sh

#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# RCS info: $Id: ch03.xml,v 1.18 2003/01/09 14:39:36 sierra Exp ssherman $
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;


# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_Rw=/root/snapshot;
EXCLUDES=/usr/local/etc/backup_exclude;


# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting...";
exit; } fi

# attempt to remount the Rw mount point as Rw; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_Rw ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $SNAPSHOT_Rw readwrite";
exit;
}
fi;


# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_Rw/home/hourly.3 ] ; then \
$RM -rf $SNAPSHOT_Rw/home/hourly.3 ; \
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_Rw/home/hourly.2 ] ; then \
$MV $SNAPSHOT_Rw/home/hourly.2 $SNAPSHOT_Rw/home/hourly.3 ; \
fi;
if [ -d $SNAPSHOT_Rw/home/hourly.1 ] ; then \
$MV $SNAPSHOT_Rw/home/hourly.1 $SNAPSHOT_Rw/home/hourly.2 ; \
fi;

# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_Rw/home/hourly.0 ] ; then \
$CP -al $SNAPSHOT_Rw/home/hourly.0 $SNAPSHOT_Rw/home/hourly.1 ; \
fi;

# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first. If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC \
-va --delete --delete-excluded \
--exclude-from="$EXCLUDES" \
/home/ $SNAPSHOT_Rw/home/hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_Rw/home/hourly.0 ;

# and thats it for home.

# now remount the Rw snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_Rw ;
if (( $? )); then
{
$ECHO "snapshot: could not remount $SNAPSHOT_Rw readonly";
exit;
} fi;