mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robin Anil (JIRA)" <>
Subject [jira] Updated: (MAHOUT-60) Complementary Naive Bayes
Date Mon, 07 Jul 2008 04:23:31 GMT


Robin Anil updated MAHOUT-60:

    Attachment: MAHOUT-60.patch

  There are a lot of changes in this patch. Most of the Files have been renamed. The trainer
is now a bunch of *5 Map Reduce jobs*. The exact functionality of each job is as follows.
The trainer can support *any number of maps and any number of reduces*.  Also i am using Apache
Lang library commons-lang-2.4.jar ( which should be put in the classpath)

      //Read the features in each document normalized by length of each document
      CBayesFeatureDriver.runJob(input, output);

      //Calculate the TfIdf for each word in each label
      CBayesTfIdfDriver.runJob(input, output);

      //Calculate the Sums of weights for each label, for each feature and for each feature
and for each label
      CBayesWeightSummerDriver.runJob(input, output);

      //Calculate the W_ij = log(Theta) for each label, feature. This step actually generates
the complement class
      CBayesThetaDriver.runJob(input, output);
      //Calculate the normalization factor Sigma_W_ij for each complement class. 
      CBayesThetaNormalizerDriver.runJob(input, output);

  I have tested it on a 6 system cluster. On 20 newsgroups dataset, it takes around 4 minutes
to train. It just used to take  20-30 seconds when creating the CNB model in-memory. But the
design is based on the assumption that the datasets are going to be too huge to fit into memory.

There can be a lot of speed improvement if the Map-Reduce operations can be somehow chained.
So Instead of Map1 -> Reduce1 - > Map1 -> Reduce2....
if it is possible to do. Map1 -> Reduce1 - > Reduce2 -> Reduce3 ->... then we
could save a lot of time on IO. I am not sure if such a functionality exists in hadoop

 I will test it out on Dmoz or Wikipedia dataset (if i can preprocess it within a reasonable
amount of time)

  The other changes are that there is no longer a model file. The model is stored in multiple
part files in  the folders trainer-theta and trainer-thetaNormalizer

To Train
$bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar org.apache.mahout.examples.classifiers.cbayes.TrainTwentyNewsgroups
-t -i 20newsinput -o 20newsoutput
To Test
$bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar org.apache.mahout.examples.classifiers.cbayes.TestTwentyNewsgroups
-p 20newsoutput  -t work/20news-18828

Next Step, to make the Classifier and the Testing completely Map Reduce.

> Complementary Naive Bayes
> -------------------------
>                 Key: MAHOUT-60
>                 URL:
>             Project: Mahout
>          Issue Type: Sub-task
>          Components: Classification
>            Reporter: Robin Anil
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 0.1
>         Attachments: MAHOUT-60.patch, MAHOUT-60.patch, MAHOUT-60.patch
> The focus is to implement an improved text classifier based on this paper

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message