This document provides an overview of the drivers for Log Management scalability and outlines the key requirements to consider as part of the evaluation process.
Whitepaper
Scalability in Log Management
Research 009-021609-02
ArcSight, Inc. Corporate Headquarters: 1-888-415-ARST 5 Results Way, Cupertino, CA 95014, USA EMEA Headquarters: +44 870 351 6510 www.arcsight.com info@arcsight.com Asia Pac Headquarters: 852 2166 8302Whitepaper: Scalability in Log Management
IntroductionIn the last few years, log management has become increasingly relevant to multiple groups within any organization. Audit teams leverage logs to automate compliance reporting and detect policy breaches. Security teams monitor log data to detect internal and external threats as well as for forensic investigations. Logs are also widely used by IT operations or helpdesk teams for faster troubleshooting and better adherence to service level agreements.
Companies typically begin their search for a log management solution with a given driver in mind, such as security threat monitoring. Yet, over time most companies will expand the scope of the original use case across more locations and devices and also grow into new use cases such as regulatory compliance or IT operations.
Other
Need to Aggregate, Analyze, Alert on and Archive All Network Log Data
Compliance and Prove Compliance with SOX, PCI, GLBA, HIPAA, etc.
Compliance with Industry Standards (COBIT, ISO,ITIL, NIC, PCOB, SANS)
IT Controls and Reporting to Prevent System Misuse, Forensics
Detection and Analysis of Security and Performance Incidents
To Assess IT Incidents and Minimize Downtime
0%10%20%30%40%50%60%70%80%90%Global 2000 Companies All Companies
This trend is validated by the SANS Institute's 2008 Log Management survey, which shows that most companies are leveraging logs for multiple use cases and it highlights the importance of evaluating the scalability of log management solutions up front. To that end, this whitepaper provides an overview of drivers for log management scalability and the resulting requirements that should be considered in any log management evaluation.
ArcSight 1Whitepaper: Scalability in Log Management
Dimensions of Log Management ScalabilityScalability in Log Management can be evaluated along five primary dimensions, which in turn point to specific requirements. Factors like EVENTadministrative and architectural scalability are addressed within these VOLUMEdimensions, as appropriate.
Event Volume ANALYSISASSETSAs organizations broaden the scope of their log management initiative, LOGevent volumes can increase significantly. Any large organization is likely to MANAGEMENTcollect thousands of events every second (equivalent to millions of logs each SCALABILITYday) just to support a single use case around security and network device monitoring. But consider what happens when regulatory drivers (PCI, SOX, HIPAA, GLBA, etc.) kick in? The organization must now collect more logs from existing devices (especially logs capturing user activity) and will also CAPACITYLOCATIONShave to expand collection to operating system, database, and application logs, which actually store the regulated data. Effectively, across use cases a large organization may easily require its log management investment to scale upwards of tens of thousands of incoming logs each second.
Expanding use cases can also drive the need for scale in outbound log event volumes. For example, many organizations invest in log management as a first step towards real-time use cases, such as fraud detection acquired through a SIEM solution. Alternatively, large distributed organizations may need to deploy log management in a tiered architecture depending on their organizational structure. In both cases the first tier of the log management architecture will need to support very high collection rates while simultaneously filtering and forwarding logs to the next tier or to a SIEM destination.
Scalability considerations:
. Can the solution easily expand to handle the total inbound and outbound event rates across devices and use cases (not just the immediate scope of deployment)? . Are there sufficient buffers to accommodate peak collection rates over the expected sustained event rates? . Is the solution able to support long-term event volumes (inbound and outbound) while leaving sufficient processing room for all the log analysis activity? Or conversely, what tradeoffs on analysis performance and capacity does scalability in collect... [download for more]
