One of the greatest challenges with implementing and supporting a cloud infrastructure is keeping track of what is happening with all of the pieces. An excellent tool for aggregating activity and event reports from a very wide variety of applications, infrastructure components, monitors, and more is Splunk. Most log aggregation and management tools are focused on a narrow segment of the cloud infrastructure, or, are focused on a narrow audience (such as PCI auditors). Not so with Splunk, which can do the security audit function, and still really excel at log management for operational purposes at the same time.
Splunk bills their product as a "search tool for IT". What they do is gather events and log files from across your Cloud and then index them in a database. They provide a variety of pre-built views into that database, reporting, graphing, and alerting capabilities, a comprehensive search language, command line interface, and even a browser plug-in that will pop up alerts while you are surfing the web.
Log data can be collected by an impressive suite of options. Probably the most common is to read existing log files where they are and then forward the data to a Splunk database. Sometimes you want to avoid all that disk i/o. With Splunk you can feed data directly to it via syslog, tcp, snmp traps, and a variety of other IPC mechanisms. You can even write your own custom data feed.
The Splunk database can be divided over several servers, but still searched from a single command. This lets you organize your data as best fits your cloud, straddling firewalled network zones, grouped for audit, access or other administrative purposes, or perhaps part of a global distribution with local data collection points around the world.
Search commands can be simple or sophisiticated, chained together, saved, shared, and even run on automatic intervals.
Splunk is not without competition. Two other major players in this space include XPLG and Loglogic. What sets Splunk apart is that it is customizable, [creative commons - free license] extensible, flexible, and (mostly) intuitive to administer. It doesn’t require specialized appliances. Instead it can be deployed and managed on your favorite platform by your favorite system administration team.
That power does come at a price. It takes some pretty good hardware (but not outrageous) to store and search the log database if you start scaling into the many Gigs of data per day. Splunk’s flexibility also means that it may take some effort to customize and deploy it within your particular cloud. (On the other hand you can engineer a close fit to your specific requirements.)
XPLG is also a software solution. It comes with some pretty fancy auto-discover tools, which means you may not have to know much about your infrastructure and applications (as long as they are reasonably standard) to start collecting log files. Their licensing is a little odd, and it appears that implementing and sharing extensions, plugins, or customizations may be more challenging as a result. Initial pricing quotes obtained from this vendor for a recent project I was working on were significantly higher than Splunk.
Loglogic sells a preconfigured appliance. It is a turnkey solution with a suite of agents you can deploy to collect data. Again the closed nature of the solution makes it hard to leverage ideas from the community, and more challenging to deploy customizations than the more open Splunk. It is probably a great solution for an organization looking for a quick plug and play that will collect most of the interesting logs in one swoop. Their product is not inexpensive, and it wouldn’t be practical to deploy Loglogic for small volumes of log aggregation (whereas Splunk, for small volumes, is free). Additionally, a complex environment, which might require several appliances, could drive the cost up very quickly.
Every now and then in the IT world you come across a company that appears to be "doing it right". Attractive website, excellent (and cool) product, sensible licensing, active and vibrant community, accomodating of the big guys AND the little guys, responsive and friendly support, clear road maps, and that je ne sais quoi magic. Splunk is one of them.