mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "XiaoboGu" <guxiaobo1...@gmail.com>
Subject RE: FW: NullPointerException after getBest() during training a AdaptiveLogisticRegression.
Date Fri, 24 Jun 2011 10:02:10 GMT
Hi,

After debugging the source , the NullPointerException is because of not seting CrossFolderLearner
with a auc



> -----Original Message-----
> From: XiaoboGu [mailto:guxiaobo1982@gmail.com]
> Sent: Friday, June 24, 2011 4:55 PM
> To: 'Ted Dunning'; 'dev@mahout.apache.org'
> Subject: RE: FW: NullPointerException after getBest() during training a
> AdaptiveLogisticRegression.
> 
> When I debug the patch, I found there are 3 threads suspended by line 126 of
> AdaptiveLogisticRegression.java,
> 
> I see there a inner Record buffer with length 500, the NullPointerException was thrown
when
> the inner buffer is full, is this a multi-threading related problem?
> 
> Regards,
> 
> Xiaobo Gu
> 
> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> Sent: Monday, June 06, 2011 6:23 PM
> To: dev@mahout.apache.org
> Cc: XiaoboGu
> Subject: Re: FW: NullPointerException after getBest() during training a
> AdaptiveLogisticRegression.
> 
> I am out of pocket for at least another week and will have a hard time finding time to
look at
> patches.
> 
> I should be able to do so on the time scale of weeks, however.  I am for this patch in
principle,
> but details matter a lot.
> On Sat, Jun 4, 2011 at 8:45 AM, Sean Owen <srowen@gmail.com> wrote:
> I don't know anything about this code; Ted does. Do you have a proposed
> patch that he could evaluate?
> 
> On Sat, Jun 4, 2011 at 4:34 PM, XiaoboGu <guxiaobo1982@gmail.com> wrote:
> 
> > Hi Sean,
> >        Can you help with this?
> >
> > Regards,
> >
> > Xiaobo Gu
> >
> > > -----Original Message-----
> > > From: XiaoboGu [mailto:guxiaobo1982@gmail.com]
> > > Sent: Sunday, May 29, 2011 4:23 PM
> > > To: 'Ted Dunning'
> > > Cc: 'dev@mahout.apache.org'
> > > Subject: NullPointerException after getBest() during training a
> > AdaptiveLogisticRegression.
> > >
> > > Hi,
> > >
> > >       The main process for MAHOUT-696 is as following, but it will always
> > cause a
> > > NullPointerException after the first call to getBest, can we continue
> > training
> > > AdaptiveLogisticRegressions after using getBest() to score some new lines
> > just as
> > > TrainLogistic does?
> > >
> > >
> > >
> > > double logPEstimate = 0;
> > >                       int k = 0;
> > >
> > >                       CsvRecordFactory csv = lmp.getCsvRecordFactory();
> > >                       model = lmp.createAdaptiveLogisticRegression();
> > >                       State<Wrapper, CrossFoldLearner> best = null;
> > >                       CrossFoldLearner learner = null;
> > >
> > >                       for (int pass = 0; pass < passes; pass++) {
> > >                               BufferedReader in = open(inputFile);
> > >
> > >                               // read variable names
> > >                               csv.firstLine(in.readLine());
> > >
> > >                               String line = in.readLine();
> > >
> > >                               while (line != null) {
> > >                                       // for each new line, get target
> > and predictors
> > >                                       Vector input = new
> > > RandomAccessSparseVector(lmp.getNumFeatures());
> > >                                       int targetValue =
> > csv.processLine(line, input);
> > >
> > >                                       // update model
> > >                                       model.train(targetValue, input);
> > >
> > >                                       k ++;
> > >
> > >                                       if (scores && (k % (skipscorenum
+
> > 1) == 0) ) {
> > >
> > >                                               best = model.getBest();
> > >                                               if (null != best) {
> > >                                                       learner =
> > best.getPayload().getLearner();
> > >                                               }
> > >                                               if (learner != null) {
> > >                                               // check performance while
> > this is still news
> > >                                               double logP =
> > learner.logLikelihood(targetValue, input);
> > >                                               if
> > (!Double.isInfinite(logP)) {
> > >                                                       if (k < 20) {
> > >
> > logPEstimate = (k * logPEstimate + logP)
> > >
> >     / (k + 1);
> > >                                                       } else {
> > >
> > logPEstimate = 0.95 * logPEstimate + 0.05
> > >
> >     * logP;
> > >                                                       }
> > >                                               }
> > >                                               double p =
> > learner.classifyScalar(input);
> > >
> > >
> > output.printf(Locale.ENGLISH,
> > >                                                               "%10d %2d
> > %10.2f %2.4f %10.4f %10.4f\n",
> > >                                                               k,
> > targetValue,
> > >
> > learner.percentCorrect(), p, logP,
> > >
> > logPEstimate);
> > >                                               }else{
> > >
> > output.printf(Locale.ENGLISH,
> > >
> > "%10d %2d %s\n", k, targetValue,
> > >
> > "AdaptiveLogisticRegression is not ready for
> > > scoring ... ");
> > >                                               }
> > >                                       }
> > >
> > >
> > >                                       line = in.readLine();
> > >                               }
> > >                               in.close();
> > >                       }
> > >
> > >
> > >
> > >
> > >        100  1 AdaptiveLogisticRegression is not ready for scoring ...
> > >        200  0 AdaptiveLogisticRegression is not ready for scoring ...
> > >        300  1 AdaptiveLogisticRegression is not ready for scoring ...
> > >        400  0 AdaptiveLogisticRegression is not ready for scoring ...
> > >        500  1 AdaptiveLogisticRegression is not ready for scoring ...
> > > Exception in thread "main" java.lang.IllegalStateException:
> > java.lang.NullPointerException
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.AdaptiveLogisticRegression.trainWithBufferedExamples(
> > > AdaptiveLogisticRegression.java:144)
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.AdaptiveLogisticRegression.train(AdaptiveLogisticRegress
> > > ion.java:117)
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.AdaptiveLogisticRegression.train(AdaptiveLogisticRegress
> > > ion.java:103)
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.TrainAdaptiveLogistic.main(TrainAdaptiveLogistic.java:7
> > > 2)
> > > Caused by: java.lang.NullPointerException
> > >       at
> > org.apache.mahout.classifier.sgd.CrossFoldLearner.train(CrossFoldLearner.java:134)
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.AdaptiveLogisticRegression$Wrapper.train(AdaptiveLogis
> > > ticRegression.java:411)
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.AdaptiveLogisticRegression$1.apply(AdaptiveLogisticReg
> > > ression.java:128)
> > >       at
> > >
> >
> org.apache.mahout.classifier.sgd.AdaptiveLogisticRegression$1.apply(AdaptiveLogisticReg
> > > ression.java:1)
> > >       at
> > org.apache.mahout.ep.EvolutionaryProcess$1.call(EvolutionaryProcess.java:146)
> > >       at
> > org.apache.mahout.ep.EvolutionaryProcess$1.call(EvolutionaryProcess.java:1)
> > >       at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > >       at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > >       at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > >       at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > >       at java.lang.Thread.run(Thread.java:662)
> >
> >



Mime
View raw message