Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: Apache Wiki <wikidiffs@apache.org>
To: core-commits@hadoop.apache.org
Date: Thu, 21 May 2009 18:06:48 -0000
Message-ID: <20090521180648.22807.11312@eos.apache.org>
Subject: [Hadoop Wiki] Update of "Anomaly Detection Framework with Chukwa" by
 EricYang

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by EricYang:
http://wiki.apache.org/hadoop/Anomaly_Detection_Framework_with_Chukwa

New page:
Describe Anomaly Detection Framework with Chukwa here.

[[TableOfContents]]

== Introduction ==

Hadoop is a great computation platform for map reduce job, but trouble shooting faulty compute node in the cluster is not an easy task.    Chukwa Anomaly Detection System, is a system for detecting computer failure and misuse by monitoring system activity and classifying it as either normal or anomalous. The classification is based on heuristics, rules, and patterns, and will detect any type of misuse that falls out of normal system operation.

In order to determine what is failure, the system must be taught to recognize normal system activity. This can be accomplished in several ways, most often with artificial intelligence type techniques. Systems using neural networks have been used to great effect. Another method is to define what normal usage of the system comprises using a strict mathematical model, and flag any deviation from this as an system problem. This is known as strict anomaly detection.  For the prototyping phase, Chukwa will use strict mathematical model as the skeleton.

== Design ==

A new processing pipeline has been introduced to post demux processor.  This enables Chukwa to run ping/mr job based aggregation and anomaly detection framework.


== Implementation ==