Linux Event Logging for Enterprise-Class Systems

Comparison of printk/syslog and Enterprise Event Logging

Note:
For sake of simplicity, the term "event" will be used to refer to both event log "events" and to printk and syslog messages.

Feature	Available with printk/syslog ?	Available with Enterprise Event Logging ?
Logging of textual messages	Yes	Yes
Logging of binary data, formatted after the event has been recorded in the log file	No - all "events" are logged as formatted text strings	Yes - binary data is formatted and displayed according to information specified by the developer and/or end user, either in a formatting template or with command line options. formatting is applied when the event record is read from the log
Logging of events to a few, well-known, destinations so that related events are stored together and keeping control of log file size(s) is practical.	Yes and No - can be stored in a few or in multiple destinations, as specified in syslog.conf	Yes - All events are stored in one of two available log files -- default is that all events are logged to /var/evlog/eventlog except those with facility=AUTHPRIV, which are stored in /var/evlog/privatelog
A standardized set of data (attributes) is recorded with each event by the logging system	Yes, the following: timestamp pid (optional) hostname (optional) program name (optional, but always "kernel" if logged with printk) The following is available but discarded before the message is recorded: facility severity (aka level)	Yes, the following standard attributes: timestamp (also assessable as "age") pid hostname (local, or if event is from a remote host, then the remote hostname) facility - standard, or developer-defined severity event type - developer-defined identifier uid - effective user id gid - effective group id pgrp - process group thread id processor id flags, including one that indicates that printk() or other logging function was called from interrupt context
Developer can define new Facility, for example, one identifying a specific device driver	No- all kernel events are facility of LOG_KERN. 16 other facilities are available in user-space	Yes - in addition to the ones defined for printk/syslog, the capability exists to define new facilities.
End user commands to extract, filter, and view only events of interest.	Limited by user's ability to concoct ad hoc text-parsing scripts	Yes - The evlview command, using the underlying query API in libevl, supports arbitrarily complex "filter" expressions for selecting events based on the values of their attributes. These attributes may be the standard attributes listed above, the textual message in a STRING-format record, or developer-defined attributes in BINARY-format records. Filter expressions can include numeric comparisons, string comparisons, and regular-expression matches.
End user can customize what event data is displayed and how it's formatted	No	Yes - end user can control which data within an event record is displayed and how it appears in several ways: with evlview command line options by specifying a canned, user-defined display format with the evlview -F option by modifying and recompiling the associated formatting template
Reliable, easy to configure asynchronous event notification sent to end-user and causing execution of a user-specified command line, when specific events are logged	Limited -- can write to named pipe, send e-mail, or display console message based solely on Facility and/or Severity	Yes - The evlnotify command, using the underlying query API in libevl, supports arbitrarily complex "filter" expressions for selecting events based on the values of their attributes. These attributes can be any of the standard attributes listed above. Along with the filter expression, the user specifies: Command line to execute after matching event is logged whether notification is sent for just the first matching event or for all matching events whether or not to restore the specified notification after a reboot The event notificaton mechanism uses real-time signaling for reliable notification, and also exists at the API level, which could be important for system-monitoring applications.
End user can determine which events to remove from the log, and which ones to keep, and manage growth of the log file's size.	Limited - logrotate "archives" entire message files and discards the oldest one (configurable by user)	Yes - The evlogmgr command, using the underlying query API in libevl, supports arbitrarily complex "filter" expressions for selecting events based on the values of their attributes. These attributes can be any of the standard attributes listed above. Events matching the filter are deleted. Events that are not deleted are moved towards the beginning of the log file to fill the freed space and the log file is truncated. The evlogmgr command can be scheduled as a cron job.
Discarding of duplicate records	Yes, fixed algorithm that's always enabled	Yes, configurable algorithm either based on event-count or time-interval or both. Can be enabled or disabled (all events are logged). The evlconfig command modifies the configuration.
Detection of event buffer overflow	No, messages are dropped without notice if ring buffer in kernel wraps on itself	Yes, count of dropped events is tracked and reported in an event record if kernel event buffer overflows.
Ability to specify "logging level" (severity) in the kernel	Yes	Yes
End user can screen-out events and prevent certain events from being logged	Limited - syslogd can drop events explicitly excluded in syslog.conf based solely on Facility and Severity; or, not identified at all in syslog.conf.	Yes - The evlconfig command, using the underlying query API in libevl, supports arbitrarily complex "filter" expressions for selecting events based on the values of their attributes. These attributes can be any of the standard attributes listed above. Events matching the filter (or "screen" is this case) are discarded and not logged.
Internationalization support - displaying of events in local language	No	Limited - Date and time are displayed based on LANG environment variable setting. Formatting templates support non-English languages for formatting strings applied to event records. More support is coming in 2004.
Resolution of kernel numeric addresses to their symbolic forms	Yes	Yes
Events can be optionally forwarded to a centralized host	Yes, but not secure; also, uses an "unreliable" connectionless protocol (UDP) and offers no other alternatives	Yes, offers both UDP and TCP. TCP connection is password authenticated. Centralized host only accepts events from hosts listed in its evlhost file. All hosts do not have to be the same architecture. Enhancements in 2004 include data encryption.