The first step in building our high-performance syslog delivery infrastructure is a review of how the thing works at its most basic level.

Traditionally, the syslog protocol on Unix-like hosts and network devices uses the UDP transport. UDP lacks any mechanism to ensure a connection is made and that packets are delivered. This alone makes standard syslog prone to quiet delivery failures on large networks, especially where network paths involve multiple router hops or tunneling. UDP may also be blocked at a router or firewall, preventing the syslog traffic from flowing.

Secondarily, the configuration of standard Unix syslogd offers little flexibility beyond facility and priority in directing delivery to log files or remote logging systems.

So, to solve these two problems, I chose (along with thousands of others) Syslog-ng as my logging and relaying application. Syslog-ng accepts traditional UDP syslog traffic, which is useful for a migration or for network devices whose syslogging facility can not be modified or replaced, as well as accepting/generating TCP syslog traffic which vastly improves delivery, and troubleshooting of delivery difficulties.

Syslog-ng also has rich set of listening, filtering, and relaying options that provide one with extreme flexibility in managing large volume, disparate log contents.

The following diagram illustrates a tiered collection system:

Given uptime and reliabilty requirements, this tiered system allows one to manipulate and replace components with minimum impact. Syslog traffic flows down the diagram, from the client (servers and network devices) to a collector layer, which may be geographically or otherwise centralized, to load-balancing, high-availability Layer-4 network gear, perhaps a Foundry switch or Netscaler/BigIP device.

Behind that, we have an aggregation and reporting tier where syslogs ultimately reside and may be archived, indexed, and consumed in other ways. And that's really the point of this whole thing, having the logs we need to troubleshoot and secure systems, as well as meet regulatory requirements.

References:

• OSI Model: Transport Layer
• Citrix Netscaler
• F5 BigIP
• Foundry switch

Prior articles in this series:

• Series Overview

Tags: syslog on technorati, delicious, netscape, google