mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "McConnell, Christopher (GE Global Research)" <>
Subject KMeans Clustering Issues
Date Wed, 02 Feb 2011 19:34:45 GMT

I've begun to look into Mahout on top of Hadoop, specifically for large scale 
cluster analysis.

I am running into an issue however, attempting to run the, Path, Path, Path, DistanceMeasure, double, 
int, Boolean, Boolean) with the last (runSequential) false when the data is 
stored on HDFS.

I've seen multiple listings about this claiming a fix within the KMeansDriver 
by adding the job.setJarByClass() method call, however I am still getting the 
typical ClassNotFoundException: org.apache.mahout.math.Vector.

A quick overview, we've created a Map job to take our current dataset and 
convert it into the Sequence files required for the driver to be executed. We 
have then tried a few different ways of calling the - 
either within the same driver as the previous MR job or separately for a new 
JVM. Both of these tests were run through the Hadoop environment. Next, I've 
tried running a standalone Java application, setting up the configuration to 
read from HDFS, but not run within the Hadoop environment - this gives us the 
same ClassNotFoundException.

Our versions are Mahout 0.4, Hadoop 0.20.1+169.89 and Hadoop 0.20.2 (We have 
multiple clusters for testing).

I have done other tests with the KMeansDriver that did work, for example, 
utilizing the method within memory works fine. We can also run the clustering 
over MapReduce, if the job is launched through a java -jar command and data 
stored locally. Finally, I can execute the mahout binary with the kmeans 
argument (./mahout kmeans -c path -i path -x #) which also works fine, however 
we do not want to rely on creating multiple stages/running multiple (and 
separate) applications.

Any thoughts are appreciated.

Christopher McConnell
Computer Scientist
Advanced Computing Lab
Edison Engineering Development Program
GE Global Research

T  +1 518 387 5176

One Research Circle
Niskayuna, NY 12309

GE Imagination at Work

View raw message