In the past, system logs were largely used as a troubleshooting tool. Over time, logging has evolved to play a much broader role. Logs have become a crucial part of system and network performance optimization, tracking of user actions, providing data for the investigation of suspicious activity, and enabling proactive monitoring. Easily accessible logs can ensure timely warnings on system problems before they cause irreparable damage.
But while the merits of maintaining log files are clear, the traditional storage and management of log files hasn’t made reviewing them something worth looking forward to. Few IT staff relish the thought of manually poring through log files to diagnose a system problem. It doesn’t help if it involves a breach of sensitive information since IT administrators will be under enormous pressure from management to get to the bottom of the matter fast.
Fortunately, there’s a way to ease and simplify this entire process — centralized log management.
Security Standards Demand It
Centralizing logging is implied or required by most IT security standards and regulations including PCI DSS, HIPAA, CoBIT, and ISO 27002. This has made log management one of the fastest growing segments of the tech industry with rising demand for log management guides such as "PHP Logging Basics -The Ultimate Guide to Logging".
Centralized logging however is not just about aggregating logs from various sources. After all, such enormous quantities of data can be counterintuitive by making problem solving harder. NIST’s Special Publication 800-92 may have been developed in 2006 but it remains one of the best guidelines in the practice of log management. It places logs in three categories based on the nature of the originating system — application logs, operating system logs, and security software logs. This categorization eases the analysis process in centralized log management.
The Place of Centralized Logging Tools
Altogether, the average IT administrator will have hundreds of log files they could place under centralized logging. Most modern systems are built with the capability to relay their log files to a centralized server. While you can perform centralized log management with default server tools, you are likely to get more value by using a purpose-built third-party application.
There are software-only and appliance-based log management solutions. Appliance options often deliver better depth and performance. However, keep an open mind. A solution that works near perfectly in one organization may yield mediocre results in another because no two technology environments are identical.
Evaluate different options and settle on the one that best secures and most efficiently collects log data from diverse sources.
Agentless vs .Agent-Based Tools
Centralized logging tools are either agentless or agent-based. Agent-based products have a special software (the agent) installed on monitored hosts. The agent program filters, aggregates and normalizes events before relaying the information to the central log management server. While this speeds up analysis on the central log server, the main drawback of this setup is the amount of time required to install the agent on each host.
Agentless products receive log data from hosts without the need for an agent to be installed on the host devices. An agentless product can be up and running much faster than an agent-based one. However, since there’s no log filtering at the host level, it takes significantly more time to analyze the logs on the central server.
Separating Legitimate Warnings from Noise
The best log management software not only aggregates data but also normalizes it, facilitates queries, and automatically generates alerts when an anomalous event is detected. Effective centralized log management is about striking the right balance between getting sufficient information to detect all significant security events and not overwhelming your networks and servers with unnecessary data.
Still, even with the best optimized log management tools, the majority of collected data will be noise. Where a log management tool demonstrates its versatility is in its ability to filter out this noise and zero in on actionable useful events. Actionable events must always trigger an instant alert and a quick investigation.
An event should be considered actionable when it’s a strong signal of malicious activity, mission critical application failure, excessively high sustained system activity or an unexpected sustained drop in activity. Alerts could be generated by an event in one system or from the real-time automated analysis of correlated events taking place in multiple systems.
Centralized logging can allow you to extract critical actionable events from an overwhelming volume of information. A good log management system will help you do just that.