The client layer is where the logs are generated. I often refer to these as the 'edge hosts', but that doesn't necessarily mean they are all at the network edge. The clients can be on a variety of networks and in various locations around the world.

A word of warning: In a large and heterogeneous environment, the client layer is where you're going to have to make the most compromises.

Many clients, especially older hosts and network gear, can't have their syslog software upgraded or replaced. The saving grace is that almost everything supports network syslog delivery. We'll take advantage of this to get the logs off the host and into our infrastructure.

Some things to consider:

• Do you have space for all those logs? You should profile the remote clients' log usage, over a week and over a month, to determine if you're going to be able to take them all and keep them as long as you'd like.
• Along with that daily volume number, consider the rate at which logs are written. Do you have a fast and furious period or are logs spread evenly throughout the day? The rate of arrival (and projected increases) will affect how you spread your syslog master hosts and what type of storage you'll need on them.
• Do you have admin access to upgrade clients or modify existing syslog client configurations? You'll need the access or the commitment of your Operations peers to get the configurations done.
• Where are your clients? Are they across WAN links, VPNs, or other network hops? Be aware and prepared to get firewall ACLs updated to allow traffic to pass. In our solution here, we use a local collector to capture UDP logs, then take advantage of TCP reliability across the miles.
• Do you have access to the clients to generate log entries (typically via logger or a similar utility)? This is needed as you move through each site to confirm that logs are flowing as they should.
• Have you considered conversion tools for non-syslog hosts, such as Microsoft Windows? There are both pay and free clients out there.

Now, what to send? I'm strongly in favor of sending everything that you can, provided you have the storage (which is cheap!) to hold it. That makes the client configurations straightforward, which speeds up the work and testing. That's a bonus if you have others helping you do the setup.

How to send it? If you can manage it, which means package, distribute, and configure the application in a repeatable way, I strongly suggest using Syslog-ng as a client. Using Syslog-ng gives you rich filtering at the client and TCP connectivity, which improves delivery reliability.

If you can't do this, add a remote logging line to syslog.conf (or similar config, depending on your device). From the syslog.conf man page:

       *.*                          @finlandia
       This rule would redirect all messages to a remote host 
       called finlandia.  This is useful especially in a cluster 
       of machines where all syslog messages will be stored on 
       only one machine.

This remote configuration will send your syslog packets over UDP to the remote host.

One other thing to remember, as you convert clients over to the new syslog infrastructure, you don't want newly deployed clients to be missed or require re-work. Work with your Operations peers to get the updated syslog configs built into your base configurations. That way, new hosts will be deployed and begin logging immediately.

Don't worry, we're only covering concepts now. At the end of this series, I'll provide a recipe with the order of operations needed to cook up your logging system.

Prior articles in this series:

• Series Overview
• Syslog Scheme

The first step in building our high-performance syslog delivery infrastructure is a review of how the thing works at its most basic level.

Traditionally, the syslog protocol on Unix-like hosts and network devices uses the UDP transport. UDP lacks any mechanism to ensure a connection is made and that packets are delivered. This alone makes standard syslog prone to quiet delivery failures on large networks, especially where network paths involve multiple router hops or tunneling. UDP may also be blocked at a router or firewall, preventing the syslog traffic from flowing.

Secondarily, the configuration of standard Unix syslogd offers little flexibility beyond facility and priority in directing delivery to log files or remote logging systems.

So, to solve these two problems, I chose (along with thousands of others) Syslog-ng as my logging and relaying application. Syslog-ng accepts traditional UDP syslog traffic, which is useful for a migration or for network devices whose syslogging facility can not be modified or replaced, as well as accepting/generating TCP syslog traffic which vastly improves delivery, and troubleshooting of delivery difficulties.

Syslog-ng also has rich set of listening, filtering, and relaying options that provide one with extreme flexibility in managing large volume, disparate log contents.

The following diagram illustrates a tiered collection system:

Given uptime and reliabilty requirements, this tiered system allows one to manipulate and replace components with minimum impact. Syslog traffic flows down the diagram, from the client (servers and network devices) to a collector layer, which may be geographically or otherwise centralized, to load-balancing, high-availability Layer-4 network gear, perhaps a Foundry switch or Netscaler/BigIP device.

Behind that, we have an aggregation and reporting tier where syslogs ultimately reside and may be archived, indexed, and consumed in other ways. And that's really the point of this whole thing, having the logs we need to troubleshoot and secure systems, as well as meet regulatory requirements.

References:

• OSI Model: Transport Layer
• Citrix Netscaler
• F5 BigIP
• Foundry switch

Prior articles in this series:

• Series Overview

System logs are vital to knowing what's going on with your systems. System logs capture a variety of system and application information and can help you ascertain the health and security of your systems. And these days, compliance requirements such as the Payment Card Industry Data Security Standard (PCI, for short) and the Sarbanes-Oxley Act (SOX) make logging and log retention required. Your particular business may also have other legal and regulatory requirements.

The next few postings here will show the way to building a high-performance, reliable, and secure logging infrastructure. The techniques here are not meant for a handful of hosts in a single location. We'll be looking at a multi-tier, world-wide system that can handle hundreds of millions of log entries per day.

We'll also talk about ensuring the integrity of your system logs to make the data therein useful and reliable, even for the legal and compliance world.

None of this is revolutionary or even particularly difficult, but I wanted to collect the techniques into one place, almost like a recipe, after having spent a lot of time assembling such a system over the months.

Scenario Details:

• Heterogeneous (Unix/Linux/Windows) hosts
• Heterogeneous (Cisco/Juniper/other) network devices
• Geographically dispersed data centers and points-of-presence (POP)
• Compartmentalized (firewalls) networks
• Requirements for PCI, SOX, and other monitoring
• Requirements for remote logging to aid in intrusion detection and forensics

Components:

• logging hosts with lots of disk capacity
• Syslog-NG
• Server load-balancing and Virtual IP network gear
• Standard software packaging and installation methods
• Scripts/tools to generate parallel requests
• Indexing tools to index and search logs
• Monitoring tools to read log streams

Index to the postings (links updated as we progress):

• The Scheme
• The Client
• The Transport, Part I
• The Relay
• The Transport, Part II
• Availability
• Storage
• Log Integrity
• Consuming The Logs
• Niceties
• The Recipe (checklists galore)

References:

• Syslog-NG
• Global server load-balancing
• Virtual IP address
• PCI DSS
• Sarbanes-Oxley Act