Artemis, Chukwa and Scribe

I'm going to talk about these three together because they are very similar and need to be compared against each other. All three are frameworks for log collection and analysis in a data center.

The first difference is in the latency of analysis. All three frameworks are used primarily for offline analysis, but Scribe also supports close-to-realtime routing of messages through various servers. (The other two have some modules that scan files periodically and are therefore intrinsically higher-latency.) It seems that in general, anything based on monitoring files might be higher-latency and less aware of message boundaries than a system like Scribe where the app actively pushes messages.

A second difference is in pieces of the analysis stack provided. The articles jointly identify four components needed for log analysis:

1. Some source of data (new instrumentation library in Scribe, or just collecting log files from existing libraries in Chukwa and Artemis).
   
2. A system for collecting the logs centrally.
3. A system for storing the logs.
4. Analysis tools on top of the collected logs.

Artemis provides all four components, although 1 just uses existing logs. Chukwa provides 1-3 but lets you build 4 rather than attempting to have as detailed a library as Artemis. Scribe really only provides 1 and 2, and in fact requires you to do the plumbing for 1 if you want to use external data sources. The interesting thing however is the commonality in how the articles suggest you do 3 and 4 -- they all talk about using Hadoop or Dryad if your scale is sufficiently large. Unsurprisingly, log analysis is a good use case of Dryad and MapReduce.