mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Problem with Wikipedia demo
Date Tue, 01 Feb 2011 08:25:30 GMT
Yup, the Wikipedia Bayes classifier demo doesn't work. This is with a
recent checkout, minus the new changes to Vector.normalize(). Would
that matter?

I tested with chunk-0001.xml as training data and chunk-0002.xml as
test data. Countries.txt as the whatever it's called.

When I test against the same data I trained on (chunk-0001.xml), I get this:
 =======================================================
Summary
-------------------------------------------------------
Correctly Classified Instances          :          5       83.3333%
Incorrectly Classified Instances        :          1       16.6667%
Total Classified Instances              :          6

=======================================================
Confusion Matrix
-------------------------------------------------------
a      b      c         <--Classified as
0      1      0          |  1           a     = algeria
0      5      0          |  5           b     = united_states
0      0      0          |  0           c     = unknown
Default Category: unknown: 2


When I test with this on a different input set (chunk-0002.xml), I get this:

Running on hadoop, using HADOOP_HOME=/lucid/lance/open/hadoop-0.20.2
No HADOOP_CONF_DIR set, using /lucid/lance/open/hadoop-0.20.2/conf
11/01/31 23:49:16 INFO bayes.TestClassifier: Loading model from:
{basePath=../datasets/wikipedia_train1/, classifierType=bayes,
alpha_i=1.0, dataSource=hdfs, gramSize=1, verbose=false,
encoding=UTF-8, defaultCat=unknown,
testDirPath=../datasets/wikipedia_input2}
11/01/31 23:49:16 INFO bayes.TestClassifier: Testing Bayes Classifier
11/01/31 23:49:16 INFO io.SequenceFileModelReader:
file:/lucid/lance/open/datasets/wikipedia_train1/trainer-weights/Sigma_j/part-00000
11/01/31 23:49:16 INFO io.SequenceFileModelReader:
file:/lucid/lance/open/datasets/wikipedia_train1/trainer-weights/Sigma_k/part-00000
11/01/31 23:49:16 INFO io.SequenceFileModelReader:
file:/lucid/lance/open/datasets/wikipedia_train1/trainer-weights/Sigma_kSigma_j/part-00000
11/01/31 23:49:16 INFO io.SequenceFileModelReader: 24.58041375116976
11/01/31 23:49:16 INFO io.SequenceFileModelReader:
file:/lucid/lance/open/datasets/wikipedia_train1/trainer-thetaNormalizer/part-00000
11/01/31 23:49:16 INFO io.SequenceFileModelReader:
file:/lucid/lance/open/datasets/wikipedia_train1/trainer-tfIdf/trainer-tfIdf/part-00000
11/01/31 23:49:17 INFO datastore.InMemoryBayesDatastore: algeria
-30159.939567094563 78146.01904096449 -0.38594339081156
11/01/31 23:49:17 INFO datastore.InMemoryBayesDatastore: united_states
-78146.01904096449 78146.01904096449 -1.0
Exception in thread "main" java.lang.NullPointerException
  at org.apache.mahout.classifier.ConfusionMatrix.getCount(ConfusionMatrix.java:99)
  at org.apache.mahout.classifier.ConfusionMatrix.incrementCount(ConfusionMatrix.java:115)
  at org.apache.mahout.classifier.ConfusionMatrix.incrementCount(ConfusionMatrix.java:119)
  at org.apache.mahout.classifier.ConfusionMatrix.addInstance(ConfusionMatrix.java:87)
  at org.apache.mahout.classifier.ResultAnalyzer.addInstance(ResultAnalyzer.java:69)
  at org.apache.mahout.classifier.bayes.TestClassifier.classifySequential(TestClassifier.java:266)
  at org.apache.mahout.classifier.bayes.TestClassifier.main(TestClassifier.java:186)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
  at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
  at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:187)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message