mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robin Anil <robin.a...@gmail.com>
Subject Re: Mahout KMeans clustering results
Date Thu, 25 Feb 2010 08:54:52 GMT
yeah... Now you cant write RandomAccessSparseVectors or any impls

there is a class VectorWritable which does the serialization

use VectorWritable.class in sequence file constructor

for writing

VectorWritable writable = new VectorWritable();

for(each data) {
  Vector v  = new RandomAccessSparseVector(your data);

  writable.set(v);
  writer.write(key, writable);
}






On Thu, Feb 25, 2010 at 2:19 PM, Arshad Khan <khan.m.arshad@gmail.com>wrote:

> This may not be related to Mahout but I am seeing an NPE when creating a
> SequenceFile.Writer. The following code used to work fine with Mahout 0.2
> but now breaks with 0.3. I suspected that some dependencies may be causing
> that so updated the classpath with all deps for 0.3 but the exception still
> happens. The only diff from before is the use of RandomAccessSparseVector
> instead of SparseVector.
>
>        VectorWriter sfWriter;
>        Path path = new Path(outFile);
>        Configuration conf = new Configuration();
>        FileSystem fs = FileSystem.get(conf);
>
>        // NPE happens on following line
>        SequenceFile.Writer seqWriter = SequenceFile.createWriter(fs, conf,
> path,
>                                          LongWritable.class,
> RandomAccessSparseVector.class);
>
> Stack trace is follows:
> java.lang.NullPointerException
>        at
>
> org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
>        at
> org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:910)
>        at
>
> org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile.java:1074)
>        at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397)
>        at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:284)
>        at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:265)
>        at
> com.lilly.tma.lm.cluster.Analyzer.getSeqFileWriter(Analyzer.java:119)
>
> Any help is greatly appreciated.
>
> Thanks
>
> On Thu, Feb 25, 2010 at 2:27 PM, Arshad Khan <khan.m.arshad@gmail.com
> >wrote:
>
> > Thanks for the help and explanation. :)
> >
> >
> > On Thu, Feb 25, 2010 at 1:20 PM, Jake Mannix <jake.mannix@gmail.com
> >wrote:
> >
> >> And to clarify: you can use either one, but you should think of them
> like
> >> this:
> >> RandomAccessSparseVector is useful for vectors whose contents change
> >> a great deal (the moving centroids of a clustering algorithm, for
> >> example),
> >> and SequentialAccessSparseVector are useful (ie faster) in the case
> where
> >> they are built up, and then are essentially used in an immutable fashion
> >> (you repeatedly compute a lot of dot-products and add multiples of them
> >> onto other vectors [either DenseVectors or RandomAccessSparseVectors]).
> >>
> >>  -jake
> >>
> >> On Wed, Feb 24, 2010 at 7:45 PM, Robin Anil <robin.anil@gmail.com>
> wrote:
> >>
> >> > They are replaced by the two impls RandomAccessSparseVector or
> >> > SequentialAccessSparseVector
> >> >
> >> >
> >> > On Thu, Feb 25, 2010 at 9:10 AM, Arshad Khan <khan.m.arshad@gmail.com
> >> > >wrote:
> >> >
> >> > > Thanks for the quick reply.
> >> > >
> >> > > I have downloaded the latest 0.3 code. There seems to be significant
> >> > > changes
> >> > > in this version. For example, currently I am using
> >> > > org.apache.mahout.matrix.SparseVector class but in 0.3 I cannot find
> >> this
> >> > > class.
> >> > >
> >> > > What class it is replaced with?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Thu, Feb 25, 2010 at 10:12 AM, Ted Dunning <
> ted.dunning@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > There are known problems with that version of k-means.
> >> > > >
> >> > > > Try using the trunk version.  0.3 is very close and we are
> entering
> >> > code
> >> > > > freeze for that so you should be fine with the latest version.
> >> > > >
> >> > > > On Wed, Feb 24, 2010 at 5:46 PM, Arshad Khan <
> >> khan.m.arshad@gmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > Hello
> >> > > > >
> >> > > > > I am using Mahout 0.2 implementation of KMeans in one of
my Text
> >> > Mining
> >> > > > > project. I apply KMeans with a default K value of 4. It
seems
> that
> >> > > every
> >> > > > > time I repeat the clustering process on the same data set,
the
> >> > results
> >> > > > are
> >> > > > > different and difference (in terms of cluster size and
> membership)
> >> is
> >> > > > great
> >> > > > > from run to run. The initial set of centroid points are
chosen
> >> > randomly
> >> > > > > through RandomSeedGenerator. Is there a way to obtain more
> >> consistent
> >> > > > > results that do not differ so greatly? Or may be I am doing
> >> something
> >> > > > > wrong?
> >> > > > >
> >> > > > > Any help or idea is very much appreciated.
> >> > > > >
> >> > > > > Thanks and Regards
> >> > > > > Arshad
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Ted Dunning, CTO
> >> > > > DeepDyve
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message