mahout-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Mahout > Synthetic Control Data
Date Sat, 14 Aug 2010 22:32:00 GMT
Space: Apache Mahout (https://cwiki.apache.org/confluence/display/MAHOUT)
Page: Synthetic Control Data (https://cwiki.apache.org/confluence/display/MAHOUT/Synthetic+Control+Data)

Change Comment:
---------------------------------------------------------------------
Modified the format and verified executing the example with mahout-examples-0.4-SNAPSHOT.job.
Need to write more about Cluster Dumper

Edited by Joe Prasanna Kumar:
---------------------------------------------------------------------
h1. Introduction

The example will demonstrate clustering of control charts which exhibits a time series. [Control
charts |http://en.wikipedia.org/wiki/Control_chart] are tools used to determine whether or
not a manufacturing or business process is in a state of statistical control. Such control
charts are generated / simulated over equal time interval and available for use in UCI machine
learning database. The data is described [here |http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data.html].

h1. Problem description

A time series of control charts needs to be clustered into their close knit groups. The data
set we use is synthetic and so resembles real world information in an anonymized format. It
contains six different classes (Normal, Cyclic, Increasing trend, Decreasing trend, Upward
shift, Downward shift). With these trends occurring on the input data set, the Mahout clustering
algorithm will cluster the data into their corresponding class buckets. At the end of this
example, you'll get to learn how to perform clustering using Mahout. 

h1. Pre-Prep
Make sure you have the following covered before you work out the example.
# Input data set. Download it [here | http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data].

## Sample input data: 
Input consists of 600 rows and 60 columns. The rows from  1 - 100 contains Normal data. Rows
from 101 - 200 contains cyclic data and so on.. More info [here | http://archive.ics.uci.edu/ml/databases/synthetic_control/synthetic_control.data.html].
Sample of how the data looks is like below.
||_time||_time+x||_time+2x||..||_time+60x||
|28.7812|34.4632|31.3381|..|31.2834|
|24.8923|25.741|27.5532|..|32.8217|
..
..
|35.5351|41.7067|39.1705|48.3964|..|38.6103|
|24.2104|41.7679|45.2228|43.7762|..|48.8175|
..
..
# Setup Hadoop
## Assuming that you have installed Hadooop, start the daemons using $HADOOP_HOME/bin/start-all.sh.
If you have issues starting Hadoop, please reference the [Hadoop quick start guide | http://hadoop.apache.org/common/docs/current/quickstart.html]
## Copy the input to HDFS using $HADOOP_HOME/bin/hadoop fs \-put <PATH TO DATA> testdata
(HDFS input directory name should be testdata)
# Mahout Example job
Mahout's mahout-examples-$MAHOUT_VERSION.job does the actual clustering task and so it needs
to be created. This can be done as
## cd $MAHOUT_HOME
## mvn install. You will see BUILD SUCCESSFUL once all the corresponding tasks are through.
The job will be generated in $MAHOUT_HOME/examples/target/ and it's name will contain the
$MAHOUT_VERSION number. For example, when using Mahout 0.3 release, the job will be mahout-examples-0.3.job
This completes the pre-requisites for the clustering process using Mahout.

h1. Perform Clustering
With all the pre-work done, clustering the control data gets real simple. 
# Depending on which clustering technique to use, you can invoke the corresponding job as
below
## For [canopy |Canopy Clustering]:  
$HADOOP_HOME/bin/hadoop jar  $MAHOUT_HOME/examples/target/mahout-examples-$MAHOUT_VERSION.job
 org.apache.mahout.clustering.syntheticcontrol.canopy.Job
## For [kmeans |K-Means Clustering]:  
$HADOOP_HOME/bin/hadoop jar  $MAHOUT_HOME/examples/target/mahout-examples-$MAHOUT_VERSION.job
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
## For [fuzzykmeans |Fuzzy K-Means]:  
$HADOOP_HOME/bin/hadoop jar  $MAHOUT_HOME/examples/target/mahout-examples-$MAHOUT_VERSION.job
org.apache.mahout.clustering.syntheticcontrol.fuzzykmeans.Job
## For [dirichlet |Dirichlet Process Clustering]: 
$HADOOP_HOME/bin/hadoop jar  $MAHOUT_HOME/examples/target/mahout-examples-$MAHOUT_VERSION.job
org.apache.mahout.clustering.syntheticcontrol.dirichlet.Job
## For [meanshift |Mean Shift Clustering]: $HADOOP_HOME/bin/hadoop jar  $MAHOUT_HOME/examples/target/mahout-examples-$MAHOUT_VERSION.job
org.apache.mahout.clustering.syntheticcontrol.meanshift.Job
# Get the data out of HDFS{footnote}See [HDFS Shell | http://hadoop.apache.org/core/docs/current/hdfs_shell.html]{footnote}{footnote}The
output directory is cleared when a new run starts so the results must be retrieved before
a new run{footnote} and have a look{footnote}Dirichlet also prints data to console{footnote}
by following the below steps
## Use $HADOOP_HOME/bin/hadoop fs \-lsr output_ to view all outputs. 
## Use $HADOOP_HOME/bin/hadoop fs \-get output $MAHOUT_HOME/examples/output to copy them all
to your local machine and the output data points are in vector format. 
## Computed clusters are contained in _output/clusters-i_
## All result clustered points are placed into _output/clusteredPoints_
## you can run the ClusterDumper on them.

{display-footnotes}

Change your notification preferences: https://cwiki.apache.org/confluence/users/viewnotifications.action
   

Mime
View raw message