The client layer is where the logs are generated. I often refer to these as the 'edge hosts', but that doesn't necessarily mean they are all at the network edge. The clients can be on a variety of networks and in various locations around the world.

A word of warning: In a large and heterogeneous environment, the client layer is where you're going to have to make the most compromises.

Many clients, especially older hosts and network gear, can't have their syslog software upgraded or replaced. The saving grace is that almost everything supports network syslog delivery. We'll take advantage of this to get the logs off the host and into our infrastructure.

Some things to consider:

• Do you have space for all those logs? You should profile the remote clients' log usage, over a week and over a month, to determine if you're going to be able to take them all and keep them as long as you'd like.
• Along with that daily volume number, consider the rate at which logs are written. Do you have a fast and furious period or are logs spread evenly throughout the day? The rate of arrival (and projected increases) will affect how you spread your syslog master hosts and what type of storage you'll need on them.
• Do you have admin access to upgrade clients or modify existing syslog client configurations? You'll need the access or the commitment of your Operations peers to get the configurations done.
• Where are your clients? Are they across WAN links, VPNs, or other network hops? Be aware and prepared to get firewall ACLs updated to allow traffic to pass. In our solution here, we use a local collector to capture UDP logs, then take advantage of TCP reliability across the miles.
• Do you have access to the clients to generate log entries (typically via logger or a similar utility)? This is needed as you move through each site to confirm that logs are flowing as they should.
• Have you considered conversion tools for non-syslog hosts, such as Microsoft Windows? There are both pay and free clients out there.

Now, what to send? I'm strongly in favor of sending everything that you can, provided you have the storage (which is cheap!) to hold it. That makes the client configurations straightforward, which speeds up the work and testing. That's a bonus if you have others helping you do the setup.

How to send it? If you can manage it, which means package, distribute, and configure the application in a repeatable way, I strongly suggest using Syslog-ng as a client. Using Syslog-ng gives you rich filtering at the client and TCP connectivity, which improves delivery reliability.

If you can't do this, add a remote logging line to syslog.conf (or similar config, depending on your device). From the syslog.conf man page:

       *.*                          @finlandia
       This rule would redirect all messages to a remote host 
       called finlandia.  This is useful especially in a cluster 
       of machines where all syslog messages will be stored on 
       only one machine.

This remote configuration will send your syslog packets over UDP to the remote host.

One other thing to remember, as you convert clients over to the new syslog infrastructure, you don't want newly deployed clients to be missed or require re-work. Work with your Operations peers to get the updated syslog configs built into your base configurations. That way, new hosts will be deployed and begin logging immediately.

Don't worry, we're only covering concepts now. At the end of this series, I'll provide a recipe with the order of operations needed to cook up your logging system.

Prior articles in this series:

• Series Overview
• Syslog Scheme

Tags: syslog on technorati, delicious, netscape, google