mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Jordan <n...@influen.se>
Subject Re: Decision Forest/Partial Implementation TestForest Error
Date Sun, 16 Sep 2012 22:47:37 GMT
Let me ask this (in a slightly unrelated, but I believe semi-related)
another way.  When I run BuildForest with a large split size on a
large data set, I get task timeout unless I set mapred.task.timeout
pretty high.  It seems that I'm CPU bound while building the forest,
but nothing is updating the progress of the mapper.  Does this make
sense?  In what class is most of the CPU intensive action happening
when building a forest with partial implementation.  Everything I have
read seems to indicate that it is memory intensive, but I'm not seeing
that at all.

On Sat, Sep 8, 2012 at 1:54 PM, Nick Jordan <nick@influen.se> wrote:
> Actually, those are the command that work.  If I run the first command
> with: -Dmapred.max.split.size=21675370 (not the split size being 10x
> larger) that is when I get the failure running the TestForest job.
>
>
>
> On Sat, Sep 8, 2012 at 1:53 PM, Nick Jordan <nick@influen.se> wrote:
>> /usr/local/hadoop/hadoop-1.0.3/bin/hadoop jar
>> /usr/local/mahout/examples/target/mahout-examples-0.8-SNAPSHOT-job.jar
>> org.apache.mahout.classifier.df.mapreduce.BuildForest
>> -Dmapred.max.split.size=2167537 -oob -d
>> /securityWaitTime/wt_top_airports_2007-2012_learn.data -ds
>> /securityWaitTime/wt_top_airports_2007-2012.info -sl 5 -p -t 100
>>
>> /usr/local/hadoop/hadoop-1.0.3/bin/hadoop jar
>> /usr/local/mahout/examples/target/mahout-examples-0.8-SNAPSHOT-job.jar
>> org.apache.mahout.classifier.df.mapreduce.TestForest -i
>> /securityWaitTime/wt_top_airports_2007-2012_test.data -ds
>> /securityWaitTime/wt_top_airports_2007-2012.info -m ob -a -mr -o
>> predictions
>>
>> On Sat, Sep 8, 2012 at 5:05 AM, deneche abdelhakim <adeneche@gmail.com> wrote:
>>> Could you copy/paste the exact commands you used to run the training and
>>> the testing ?
>>>
>>> On Fri, Sep 7, 2012 at 11:10 PM, Nick Jordan <nick@influen.se> wrote:
>>>
>>>> Any thoughts here?
>>>>
>>>> On Thu, Sep 6, 2012 at 7:00 AM, Nick Jordan <nick@influen.se> wrote:
>>>> > Same problem with the sequential classifier.  My guess is that this
>>>> > "corruption" is happening because of that particular setting as it is
>>>> > the only thing that I'm changing, but I have no idea how to
>>>> > investigate further.
>>>> >
>>>> > Nick
>>>> >
>>>> > On Thu, Sep 6, 2012 at 2:22 AM, Abdelhakim Deneche <adeneche@gmail.com>
>>>> wrote:
>>>> >> Hi Nick,
>>>> >>
>>>> >> This is not a memory problem, the classifier tries to load the trained
>>>> forest but it's getting some unexpected values. This problem never occured
>>>> before! Could the forest files be corrupted ?
>>>> >>
>>>> >> Try training the forest once again, and this time use the sequential
>>>> classifier (don't use the -mr parameter) and see if the problem still
>>>> occurs.
>>>> >>
>>>> >>
>>>> >> On 5 sept. 2012, at 23:00, Nick Jordan <nick@influen.se> wrote:
>>>> >>
>>>> >>> Hello All,
>>>> >>>
>>>> >>> I'm playing around with decision forests using the partial
>>>> >>> implementation and my own data set.  I am getting an error with
>>>> >>> TestForest, but only for certain forests that I'm building with
>>>> >>> BuildForest.  Using the same descriptor and same build and test
data
>>>> >>> sets I get no error if I set mapred.max.split.size=1890528 which
is
>>>> >>> roughly 1/100th the size of the build data set.  I can build
the
>>>> >>> forest and test the remaining data and get the results with
no
>>>> >>> problem.  When I change the split size to 18905280, everything
still
>>>> >>> appears to work fine when building the forest, but when I then
try to
>>>> >>> test the remaining data I get the error below.
>>>> >>>
>>>> >>> I've dug around the code a little, but nothing stood out as
to why the
>>>> >>> array would go out of bounds at that specific value.  One solution
is
>>>> >>> to obviously not create partitions that large, but if it was
a problem
>>>> >>> with me running out of memory I would have expected an out of
memory
>>>> >>> error and not an index past the size the bounds of an array.
 I'd
>>>> >>> obviously prefer larger partitions and thus less of them and
can move
>>>> >>> running this job to something like EMR which should allow me
to have
>>>> >>> more memory, but I want to understand the nature of the error.
>>>> >>>
>>>> >>> For what it is worth I'm running this on hadoop-1.0.3 and
>>>> mahout-0.8-SNAPSHOT
>>>> >>>
>>>> >>> Thanks.
>>>> >>>
>>>> >>> --
>>>> >>>
>>>> >>> 12/09/05 17:52:09 INFO mapred.JobClient: Task Id :
>>>> >>> attempt_201209031756_0008_m_000000_0, Status : FAILED
>>>> >>> java.lang.ArrayIndexOutOfBoundsException: 946827879
>>>> >>>        at org.apache.mahout.classifier.df.node.Node.read(Node.java:58)
>>>> >>>        at
>>>> org.apache.mahout.classifier.df.DecisionForest.readFields(DecisionForest.java:197)
>>>> >>>        at
>>>> org.apache.mahout.classifier.df.DecisionForest.read(DecisionForest.java:203)
>>>> >>>        at
>>>> org.apache.mahout.classifier.df.DecisionForest.load(DecisionForest.java:225)
>>>> >>>        at
>>>> org.apache.mahout.classifier.df.mapreduce.Classifier$CMapper.setup(Classifier.java:212)
>>>> >>>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>>>> >>>        at
>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>> >>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>> >>>        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>> >>>        at java.security.AccessController.doPrivileged(Native
Method)
>>>> >>>        at javax.security.auth.Subject.doAs(Subject.java:416)
>>>> >>>        at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>> >>>        at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>

Mime
View raw message