mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: Fwd: PCA with ssvd leads to StackOverFlowError
Date Thu, 06 Mar 2014 15:36:16 GMT
If I'm right, then it will cause compile errors, but then, you just
fix those by replacing some Guava constructs with equivalent Java or
older Guava code. IIRC it is fairly trivial.

And in fact probably should not use Guava 12+ methods for this reason
even if compiling against 12+. And in fact I thought someone cleaned
that up...

On Thu, Mar 6, 2014 at 3:34 PM, Kevin Moulart <kevinmoulart@gmail.com> wrote:
> Ok so should I try and recompile and change the guava version to 11.0.2 in
> the pom ?
>
> Kévin Moulart
>
>
> 2014-03-06 16:26 GMT+01:00 Sean Owen <srowen@gmail.com>:
>
>> That's gonna be a Guava version problem. I have seen variants of this
>> for a while. Hadoop still uses 11.0.2 even in HEAD and you can often
>> get away with using a later version in a project like this, even
>> though code that executes on Hadoop will use an older Guava than you
>> compiled against. This is an example of that gotcha. I think it may be
>> necessary to force Mahout to use 11.0.2 and change this code.
>>
>> I am having deja vu like this has come up before too.
>>
>>
>>
>>
>>
>> On Thu, Mar 6, 2014 at 3:23 PM, Kevin Moulart <kevinmoulart@gmail.com>
>> wrote:
>> > Hi thanks very much it seems to have worked !
>> > Compiling with "mvn clean package -Dhadoop2.version=2.0.0-cdh4.6.0" works
>> > and I no longer have the error, but then when running tests that used to
>> > work with previous install like trainAdaptativeLogistic and then
>> > ValidateAdaptativeLogistic, the first works but the second yields an
>> error :
>> >
>> > bin/mahout validateAdaptiveLogistic --input
>> > /mnt/hdfs/user/myCompany/Echant/echant300k_wh.csv --model
>> > /mnt/hdfs/user/myCompany/Echant/Models/echnat.model --auc --scores
>> > --confusion.
>> > 14/03/06 15:53:42 WARN driver.MahoutDriver: No
>> > validateAdaptiveLogistic.props found on classpath, will use command-line
>> > arguments only
>> > Exception in thread "main" java.lang.NoSuchMethodError:
>> > com.google.common.collect.Queues.newArrayDeque()Ljava/util/ArrayDeque;
>> > at org.apache.mahout.math.stats.GroupTree$1.<init>(GroupTree.java:171)
>> >  at org.apache.mahout.math.stats.GroupTree.iterator(GroupTree.java:169)
>> > at org.apache.mahout.math.stats.GroupTree.access$300(GroupTree.java:14)
>> >  at org.apache.mahout.math.stats.GroupTree$2.iterator(GroupTree.java:317)
>> > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:105)
>> >  at org.apache.mahout.math.stats.TDigest.add(TDigest.java:88)
>> > at org.apache.mahout.math.stats.TDigest.add(TDigest.java:76)
>> >  at
>> >
>> org.apache.mahout.math.stats.OnlineSummarizer.add(OnlineSummarizer.java:57)
>> > at
>> >
>> org.apache.mahout.classifier.sgd.ValidateAdaptiveLogistic.mainToOutput(ValidateAdaptiveLogistic.java:107)
>> >  at
>> >
>> org.apache.mahout.classifier.sgd.ValidateAdaptiveLogistic.main(ValidateAdaptiveLogistic.java:63)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >  at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >  at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>> >  at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
>> > at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
>> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >  at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> >  at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> >
>> > I'll try some other tests to see what's working and what's not.
>> >
>> >
>> >
>> > 2014-03-06 15:58 GMT+01:00 Gokhan Capan <gkhncpn@gmail.com>:
>> >
>> >> Kevin,
>> >>
>> >>
>> >> From trunk, can you build mahout for hadoop2 using this command:
>> >>
>> >> mvn clean package -DskipTests=true
>> -Dhadoop2.version=<YOUR_HADOOP2_VERSION>
>> >>
>> >>
>> >> Then can you verify that you have the right hadoop jars with the
>> following
>> >> command:
>> >>
>> >> find . -name hadoop*.jar
>> >>
>> >>
>> >>
>> >> Gokhan
>> >>
>> >>
>> >> On Thu, Mar 6, 2014 at 3:26 PM, Kevin Moulart <kevinmoulart@gmail.com
>> >> >wrote:
>> >>
>> >> > Hi again, and thanks for the enthousiasm !
>> >> >
>> >> > I did compile the trunk with the hadoop2 profile and, althoug it
>> didn't
>> >> > work at first because of some Canopy tests not passing, when I skipped
>> >> the
>> >> > tests it compiled and when I tested it afterward it passed.
>> >> > I used the version I have isntalled, so I just added the line :
>> >> >     <hadoop2.version>2.0.0-cdh4.6.0</hadoop2.version>
>> >> > To the pom.xml and type :
>> >> > mvn -DskipTests clean install -Phadoop2
>> >> > Then :
>> >> > mvn test
>> >> >
>> >> > Then I tried it with these settings :
>> >> > export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
>> >> > export
>> >> >
>> >> >
>> >>
>> HADOOP_CLASSPATH=/home/myCompany/Downloads/mahout9/examples/target/mahout-examples-1.0-SNAPSHOT-job.jar
>> >> > export MAHOUT_HOME=/home/myCompany/Downloads/mahout9
>> >> >
>> >> > And the command gives me this :
>> >> > [root@node01 mahout9]# bin/mahout
>> >> > MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
>> >> > Running on hadoop, using
>> /opt/cloudera/parcels/CDH/lib/hadoop/bin/hadoop
>> >> > and HADOOP_CONF_DIR=/etc/hadoop/conf
>> >> > MAHOUT-JOB:
>> >> >
>> >> >
>> >>
>> /home/myCompany/Downloads/mahout9/examples/target/mahout-examples-1.0-SNAPSHOT-job.jar
>> >> > Exception in thread "main" java.lang.NoSuchMethodError:
>> >> > org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V
>> >> > at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:122)
>> >> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> > at
>> >> >
>> >> >
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >> >  at
>> >> >
>> >> >
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> > at java.lang.reflect.Method.invoke(Method.java:606)
>> >> >  at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> >> >
>> >> > I even tried with :
>> >> > export HADOOP_HOME=/.../hadoop,
>> >> > export HADOOP_HOME=/.../hadoop-0.20-mapreduce
>> >> > export HADOOP_HOME=/.../hadoop-mapreduce
>> >> >
>> >> > And it still gives me the same result.
>> >> >
>> >> > And recompiling with  <hadoop2.version>2.0.0</hadoop2.version>
or
>> >> >  <hadoop2.version>2.0.0-mr1-cdh4.6.0</hadoop2.version>
didn't work.
>> >> >
>> >> > Any idea ?
>> >> >
>> >> >
>> >> >
>> >> > 2014-03-05 22:42 GMT+01:00 Andrew Musselman <
>> andrew.musselman@gmail.com
>> >> >:
>> >> >
>> >> > > I mean "balance the risk aversion against the value of new features"
>> >> duh.
>> >> > >
>> >> > >
>> >> > > On Wed, Mar 5, 2014 at 1:39 PM, Andrew Musselman <
>> >> > > andrew.musselman@gmail.com
>> >> > > > wrote:
>> >> > >
>> >> > > > Yeah, for sure; balancing clients' risk aversion to technical
>> >> features
>> >> > is
>> >> > > > why we often recommend vendor solutions.
>> >> > > >
>> >> > > > Having a little button to choose a newer version of a component
in
>> >> the
>> >> > > > Manager UI (even with a confirmation dialog that said "Are
you
>> sure?
>> >> > Are
>> >> > > > you crazy?") would be more palatable to some teams than installing
>> >> > > > tarballs, is what I'm getting at.
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Mar 5, 2014 at 1:30 PM, Sean Owen <srowen@gmail.com>
>> wrote:
>> >> > > >
>> >> > > >> You can always install whatever version of anything on
your
>> cluster
>> >> > > >> that you want. It may or may not work, but often happens
to, at
>> >> least
>> >> > > >> for whatever you need it to do.
>> >> > > >>
>> >> > > >> It's just the same as it is without a packaged distribution
--
>> dump
>> >> > > >> new tarballs and cross your fingers. Nothing is weird
or
>> different
>> >> > > >> about the setup or layout. That is the "here be dragons"
>> solution,
>> >> > > >> already
>> >> > > >>
>> >> > > >> You go with support from a packaged distribution when
you want a
>> >> "here
>> >> > > >> be no dragons" solution. Everything else is by definition
already
>> >> > > >> something you can and should do yourself outside of a
packaged
>> >> > > >> distribution. And really -- you freely can, and it's
not hard, if
>> >> you
>> >> > > >> know what you are doing.
>> >> > > >>
>> >> > > >> On Wed, Mar 5, 2014 at 9:15 PM, Andrew Musselman
>> >> > > >> <andrew.musselman@gmail.com> wrote:
>> >> > > >> > Feels like just yesterday :)
>> >> > > >> >
>> >> > > >> > Consider this a feature request to have more flexible
component
>> >> > > >> versioning,
>> >> > > >> > even with a caveat/"here be dragons" warning.  I
know that
>> >> > complicates
>> >> > > >> > things but people do use your releases a long time.
 I
>> personally
>> >> > > >> wished I
>> >> > > >> > could upgrade Pig on CDH 4 for new features but
there was no
>> >> simple
>> >> > > way
>> >> > > >> on
>> >> > > >> > a managed cluster.
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > On Wed, Mar 5, 2014 at 12:12 PM, Sean Owen <srowen@gmail.com>
>> >> > wrote:
>> >> > > >> >
>> >> > > >> >> I don't understand this -- CDH always bundles
the latest
>> release.
>> >> > > >> >>
>> >> > > >> >> You know that CDH4 was released in July 2012,
right? So it
>> >> included
>> >> > > >> >> 0.7 + patches. CDH5 includes 0.8 because 0.9
was released
>> about a
>> >> > > >> >> month after it began beta 2.
>> >> > > >> >>
>> >> > > >> >> CDH follows semantic versioning and won't introduce
changes
>> that
>> >> > are
>> >> > > >> >> not backwards-compatible in a minor version
update. 0.x
>> releases
>> >> of
>> >> > > >> >> Mahout act like major version changes -- not
backwards
>> >> compatible.
>> >> > So
>> >> > > >> >> 4.x will always be 0.7 and 5.x will always be
0.8.
>> >> > > >> >>
>> >> > > >> >> On Wed, Mar 5, 2014 at 5:34 PM, Dmitriy Lyubimov
<
>> >> > dlieu.7@gmail.com>
>> >> > > >> >> wrote:
>> >> > > >> >> > On Wed, Mar 5, 2014 at 9:08 AM, Sean Owen
<srowen@gmail.com
>> >
>> >> > > wrote:
>> >> > > >> >> >
>> >> > > >> >> >> I don't follow what here makes you
say they are "cut down"
>> >> > > releases?
>> >> > > >> >> >>
>> >> > > >> >> >
>> >> > > >> >> > meaning it seems to be pretty much 2 releases
behind the
>> >> > official.
>> >> > > >> But i
>> >> > > >> >> > definitely don't follow CDH developments
in this department,
>> >> you
>> >> > > >> seem in
>> >> > > >> >> a
>> >> > > >> >> > better position to explain the existing
patchlevel there so
>> I
>> >> > defer
>> >> > > >> to
>> >> > > >> >> you
>> >> > > >> >> > to explain why this patchlevel is not there.
>> >> > > >> >> >
>> >> > > >> >> > -d
>> >> > > >> >>
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kévin Moulart
>> >> > GSM France : +33 7 81 06 10 10
>> >> > GSM Belgique : +32 473 85 23 85
>> >> > Téléphone fixe : +32 2 771 88 45
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Kévin Moulart
>> > GSM France : +33 7 81 06 10 10
>> > GSM Belgique : +32 473 85 23 85
>> > Téléphone fixe : +32 2 771 88 45
>>

Mime
View raw message