hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Anomaly Detection Framework with Chukwa" by EricYang
Date Thu, 21 May 2009 18:06:48 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by EricYang:

New page:
Describe Anomaly Detection Framework with Chukwa here.


== Introduction ==

Hadoop is a great computation platform for map reduce job, but trouble shooting faulty compute
node in the cluster is not an easy task.    Chukwa Anomaly Detection System, is a system for
detecting computer failure and misuse by monitoring system activity and classifying it as
either normal or anomalous. The classification is based on heuristics, rules, and patterns,
and will detect any type of misuse that falls out of normal system operation.

In order to determine what is failure, the system must be taught to recognize normal system
activity. This can be accomplished in several ways, most often with artificial intelligence
type techniques. Systems using neural networks have been used to great effect. Another method
is to define what normal usage of the system comprises using a strict mathematical model,
and flag any deviation from this as an system problem. This is known as strict anomaly detection.
 For the prototyping phase, Chukwa will use strict mathematical model as the skeleton.

== Design ==

A new processing pipeline has been introduced to post demux processor.  This enables Chukwa
to run ping/mr job based aggregation and anomaly detection framework.

== Implementation ==

View raw message