Building a monitoring infrastructure is a complex undertaking. The system can potentially interact with every system in the environment, and its users range from the layman to the highly technical. Building the monitoring infrastructure well requires not only considerable systems know-how, but also a global perspective and good people skills. Most importantly, building monitoring systems also requires a light touch. The most important distinction between good monitoring systems and bad ones is the amount of impact they have on the network environment, in areas such as resource utilization, bandwidth utilization, and security. This fi rst chapter contains a collection of advice gleaned from mailing lists such as This e-mail address is being protected from spam bots, you need JavaScript enabled to view it , other systems administrators, and hard-won experience. My hope is that this chapter helps you to make some important design decisions up front, to avoid some common pitfalls, and to ensure that the monitoring system you build becomes a huge asset instead of a huge burden.

Good monitoring systems are not built one script at a time by administrators (admins) in separate silos. Admins create them methodically with the support of their management teams and a clear understanding of the environment—both procedural and computational—within which they operate.

Without a clear understanding of which systems are considered critical, the monitoring initiative is doomed to failure. It’s a simple question of context and usually plays out something like this:

Manager: “I need to be added to all the monitoring system alerts.”
Admin: “All of them?”
Manager: “Well yes, all of them.”
Admin: “Er, ok.”
The next day:
Manager: “My pager kept me up all night. What does this all mean?”
Admin: “Well, /var fi lled up on Server1, and the VPN tunnel to site5 was up and down.”
Manager: “Can’t you just notify me of the stuff that’s an actual problem?”
Admin: “Those are actual problems.”