Mailing-List: contact mahout-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-dev@lucene.apache.org
Message-ID: <680384186.1215404611834.JavaMail.jira@brutus>
Date: Sun, 6 Jul 2008 21:23:31 -0700 (PDT)
From: "Robin Anil (JIRA)" <jira@apache.org>
To: mahout-dev@lucene.apache.org
Subject: [jira] Updated: (MAHOUT-60) Complementary Naive Bayes
In-Reply-To: <544303546.1212273045058.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAHOUT-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robin Anil updated MAHOUT-60:
-----------------------------

    Attachment: MAHOUT-60.patch

  There are a lot of changes in this patch. Most of the Files have been renamed. The trainer is now a bunch of *5 Map Reduce jobs*. The exact functionality of each job is as follows. The trainer can support *any number of maps and any number of reduces*.  Also i am using Apache Lang library commons-lang-2.4.jar ( which should be put in the classpath)


 {noformat} 
      //Read the features in each document normalized by length of each document
      CBayesFeatureDriver.runJob(input, output);
      

      //Calculate the TfIdf for each word in each label
      CBayesTfIdfDriver.runJob(input, output);
      

      //Calculate the Sums of weights for each label, for each feature and for each feature and for each label
      CBayesWeightSummerDriver.runJob(input, output);
      

      //Calculate the W_ij = log(Theta) for each label, feature. This step actually generates the complement class
      CBayesThetaDriver.runJob(input, output);
      
   
      //Calculate the normalization factor Sigma_W_ij for each complement class. 
      CBayesThetaNormalizerDriver.runJob(input, output);
{noformat} 

  I have tested it on a 6 system cluster. On 20 newsgroups dataset, it takes around 4 minutes to train. It just used to take  20-30 seconds when creating the CNB model in-memory. But the design is based on the assumption that the datasets are going to be too huge to fit into memory.

There can be a lot of speed improvement if the Map-Reduce operations can be somehow chained.
So Instead of Map1 -> Reduce1 - > Map1 -> Reduce2....
if it is possible to do. Map1 -> Reduce1 - > Reduce2 -> Reduce3 ->... then we could save a lot of time on IO. I am not sure if such a functionality exists in hadoop

 I will test it out on Dmoz or Wikipedia dataset (if i can preprocess it within a reasonable amount of time)

  The other changes are that there is no longer a model file. The model is stored in multiple part files in  the folders trainer-theta and trainer-thetaNormalizer

To Train
{noformat} 
$bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar org.apache.mahout.examples.classifiers.cbayes.TrainTwentyNewsgroups -t -i 20newsinput -o 20newsoutput
{noformat}
 
To Test
{noformat} 
$bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar org.apache.mahout.examples.classifiers.cbayes.TestTwentyNewsgroups -p 20newsoutput  -t work/20news-18828
{noformat} 

Next Step, to make the Classifier and the Testing completely Map Reduce.


> Complementary Naive Bayes
> -------------------------
>
>                 Key: MAHOUT-60
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-60
>             Project: Mahout
>          Issue Type: Sub-task
>          Components: Classification
>            Reporter: Robin Anil
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 0.1
>
>         Attachments: MAHOUT-60.patch, MAHOUT-60.patch, MAHOUT-60.patch
>
>
> The focus is to implement an improved text classifier based on this paper http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.