hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "Anomaly Detection Framework with Chukwa" by JiaqiTan
Date Thu, 11 Jun 2009 16:44:50 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by JiaqiTan:
http://wiki.apache.org/hadoop/Anomaly_Detection_Framework_with_Chukwa

------------------------------------------------------------------------------
  
  == Implementation ==
  
- === Hadoop anomaly detection and behavioral visualization  ===
+ === Hadoop Anomaly Detection and Behavioral Visualization (Fingerpointing)  ===
  
  Current active developments for the Chukwa Anomaly Detection Framework are for detecting
anomalies in Hadoop based on the following tools/concepts from the CMU [http://www.ece.cmu.edu/~fingerpointing/
Fingerpointing project]:
  
@@ -45, +45 @@

  This visualization shows the detailed task-level progress of MapReduce jobs across nodes
in the cluster.
  
   1. ({{{FSMBuilder}}} MapReduce job, available soon) SALSA is used to extract state-machine
views of Hadoop's execution - uses post-Demux output; uses {{{JobData/JobHistory}}}
-  1. ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279]) State-machine data from
{{{FSMBuilder}}} is loaded into RDBMS using MDL 
+  1. State-machine data from {{{FSMBuilder}}} is loaded into RDBMS using MDL 
-  1. ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279]) Raw state-machine views
visualized using Swimlanes visualization HICC widget which reads data from RDBMS
+  1. Raw state-machine views visualized using Swimlanes visualization HICC widget which reads
data from RDBMS
  
  
  ==== MIROS (N x N heatmaps) Visualization ====
@@ -65, +65 @@

   1. Load anomaly data into RDBMS using MDL
   1. Visualization of heatmaps using HICC widget
  
+ == Usage ==
+ 
+ === Hadoop Anomaly Detection and Behavioral Visualization (Fingerpointing)  ===
+ 
+ ==== Swimlanes Visualization ====
+ 
+ This visualization is generated from the output of the Demux operation. The steps (mostly
envisioned to be automated) involved in generating the visualization are:
+ 
+  1. Generate state-machine views using {{{FSMBuilder}}} (Currently unavailable, pending
feature additions to the PostProcessor to support non-MDL tasks): Read post-Demux data (SequenceFiles
of ChukwaRecords of JobData data) as input, write state-machine view as SequenceFiles of ChukwaRecords
of states. (Unsupported at time of writing, but will be available soon)
+  1. Load states into database using MDL ([http://issues.apache.org/jira/browse/CHUKWA-279
CHUKWA-279]) 
+  1. Load Swimlanes widget in HICC ([http://issues.apache.org/jira/browse/CHUKWA-279 CHUKWA-279])

+ 

Mime
View raw message