mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Krumholz <alan_krumh...@yahoo.com.mx>
Subject using KmeansDriver with HDFS
Date Thu, 05 Sep 2013 01:58:44 GMT
Hi all,
I'm using mahout 0.7

I'm trying to use KmeansDriver (org.apache.mahout.clustering.kmeans.KMeansDriver) with HDFS
and I'm having some issues.
When I use it with my local file system everything seems to be working fine.
However, as soon as I change the Configuration object to use HDFS:
        Configuration conf = new Configuration();
        conf.addResource(new Path("C:\\hdp-win\\hadoop\\hadoop-1.1.0-SNAPSHOT\\conf\\core-site.xml"));
        conf.addResource(new Path("C:\\hdp-win\\hadoop\\hadoop-1.1.0-SNAPSHOT\\conf\\hdfs-site.xml"))


I run into problems

I was looking at the exception I get:


java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.RangeCheck(ArrayList.java:547)
    at java.util.ArrayList.get(ArrayList.java:322)
   at org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)


I pulled that code (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and
I think is trying to read a file from one of the paths I passed to the method but with a new
instance of the configuration object (not the configuration object I passed to the method
but one that doesn't have my HDFS configured)

 


205

  public void  [More ...] readFromSeqFiles(Configuration conf, Path path) throws IOException
{

206

    Configuration config = new Configuration();

207

    List<Cluster> clusters = Lists.newArrayList();

208

    for (ClusterWritable cw : new SequenceFileDirValueIterable<ClusterWritable>(path,
PathType.LIST,

209

        PathFilters.logsCRCFilter(), config)) {

210

      Cluster cluster = cw.getValue();

211

      cluster.configure(conf);

212

      clusters.add(cluster);

213

    }

214

    this.models = clusters;

215

    modelClass = models.get(0).getClass().getName();

216

    this.policy = readPolicy(path);

217

  }



any help would be really appreciated :)

Thanks!

Alan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message