crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkw...@gmail.com>
Subject Re: Test failure on the master branch
Date Wed, 15 Oct 2014 01:25:08 GMT
One thing I noticed is that is when testing we don't force IPv4.  Zookeeper
does not handle IPv6 well and can actually significantly slow down tests
trying to resolve the localhost.  This doesn't affect everyone because your
/etc/hosts could help it resolve faster but does make machine setup
important.  We could try adding an argline like this to surefire/failsafe
to enforce IPv4.

<argLine>-Xmx512m -XX:MaxPermSize=128m
-Djava.net.preferIPv4Stack=true</argLine>

On Tue, Oct 14, 2014 at 9:28 AM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:

> Thanks J!
>
> On Tue, Oct 14, 2014 at 9:57 AM, Josh Wills <jwills@cloudera.com> wrote:
>
> > I'm still rocking 1.7. Will give 1.8 a whirl this evening.
> >
> > J
> >
> > On Tue, Oct 14, 2014 at 6:53 AM, Danushka Menikkumbura <
> > danushka.menikkumbura@gmail.com> wrote:
> >
> > > I am on Ubuntu 14.04.1 LTS and Java 1.8.0_20.
> > >
> > > BTW maybe it is a result of the commit
> > > 3f98411364cec32a0a8c6681dfaabd43caa4dd60?.
> > >
> > > Thanks,
> > > Danushka
> > >
> > >
> > >
> > > On Tue, Oct 14, 2014 at 9:44 AM, Josh Wills <jwills@cloudera.com>
> wrote:
> > >
> > > > No, master is compatible w/Hadoop 1; the error you're seeing is
> caused
> > by
> > > > the HBase testing code being flaky. I've experienced the flaky
> > > > HFileTargetIT test on my machine before, but never on a regular
> basis.
> > > Can
> > > > you give me the basics of your setup-- OS, java version, etc.?
> > > >
> > > > On Tue, Oct 14, 2014 at 6:38 AM, Danushka Menikkumbura <
> > > > danushka.menikkumbura@gmail.com> wrote:
> > > >
> > > > > No I don't.
> > > > >
> > > > > That means the master branch is not compatible with Hadoop 1?
> > > > >
> > > > > Thanks,
> > > > > Danushka
> > > > >
> > > > > On Tue, Oct 14, 2014 at 9:26 AM, Josh Wills <jwills@cloudera.com>
> > > wrote:
> > > > >
> > > > > > Hrm, okay. Do you get it if you use the -Dcrunch.platform=2
> option?
> > > > > >
> > > > > > On Tue, Oct 14, 2014 at 6:23 AM, Danushka Menikkumbura <
> > > > > > danushka.menikkumbura@gmail.com> wrote:
> > > > > >
> > > > > > > Yes. I get it every time I try to build HBase module.
> > > > > > >
> > > > > > > FYI : org.apache.crunch.io.hbase.HFileTargetIT is the test
in
> > > error.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Danushka
> > > > > > >
> > > > > > > On Tue, Oct 14, 2014 at 9:15 AM, Josh Wills <
> jwills@cloudera.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > That can happen intermittently if the local HBase
cluster
> gets
> > > hung
> > > > > > up--
> > > > > > > do
> > > > > > > > you get it regularly (i.e., every time you run?)
> > > > > > > >
> > > > > > > > J
> > > > > > > >
> > > > > > > > On Tue, Oct 14, 2014 at 6:08 AM, Danushka Menikkumbura
<
> > > > > > > > danushka.menikkumbura@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I am getting the following test failure while
building
> > Crunch.
> > > > Have
> > > > > > you
> > > > > > > > got
> > > > > > > > > an idea as to what may be the issue here?.
> > > > > > > > >
> > > > > > > > > 34787 [Thread-2057] INFO
> > > > > > > > >
> > > > > > >
> > > org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob
> > > > > -
> > > > > > > > > Job status available at: http://localhost:8080/
> > > > > > > > > 44548 [M:0;danushka:38318.oldLogCleaner] ERROR
> > > > > > > > > org.apache.hadoop.hbase.client.HConnectionManager
 -
> > Connection
> > > > not
> > > > > > > found
> > > > > > > > > in the list, can't delete it (connection
> > > > > > > > >
> > > key=HConnectionKey{properties={hbase.zookeeper.quorum=localhost,
> > > > > > > > > hbase.rpc.timeout=60000,
> > > > hbase.zookeeper.property.clientPort=57963,
> > > > > > > > > zookeeper.znode.parent=/hbase,
> > hbase.client.retries.number=350,
> > > > > > > > > hbase.client.pause=100}, username='danushka'}).
May be the
> > key
> > > > was
> > > > > > > > > modified?
> > > > > > > > > java.lang.Exception
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:493)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:429)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276)
> > > > > > > > > at org.apache.hadoop.hbase.Chore.run(Chore.java:94)
> > > > > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > > > > > Tests run: 1, Failures: 0, Errors: 0, Skipped:
0, Time
> > elapsed:
> > > > > > 21.625
> > > > > > > > sec
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Danushka
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Director of Data Science
> > > > > > > > Cloudera <http://www.cloudera.com>
> > > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Director of Data Science
> > > > > > Cloudera <http://www.cloudera.com>
> > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Director of Data Science
> > > > Cloudera <http://www.cloudera.com>
> > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > >
> > >
> >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message