crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Test failure on the master branch
Date Wed, 15 Oct 2014 04:17:50 GMT
I can replicate the original error- looks like dfs.data.dir permissions
aren't set properly in test mode for some reason. Will investigate and
figure out a patch.

On Tuesday, October 14, 2014, Micah Whitacre <mkwhit@gmail.com> wrote:

> One thing I noticed is that is when testing we don't force IPv4.  Zookeeper
> does not handle IPv6 well and can actually significantly slow down tests
> trying to resolve the localhost.  This doesn't affect everyone because your
> /etc/hosts could help it resolve faster but does make machine setup
> important.  We could try adding an argline like this to surefire/failsafe
> to enforce IPv4.
>
> <argLine>-Xmx512m -XX:MaxPermSize=128m
> -Djava.net.preferIPv4Stack=true</argLine>
>
> On Tue, Oct 14, 2014 at 9:28 AM, Danushka Menikkumbura <
> danushka.menikkumbura@gmail.com <javascript:;>> wrote:
>
> > Thanks J!
> >
> > On Tue, Oct 14, 2014 at 9:57 AM, Josh Wills <jwills@cloudera.com
> <javascript:;>> wrote:
> >
> > > I'm still rocking 1.7. Will give 1.8 a whirl this evening.
> > >
> > > J
> > >
> > > On Tue, Oct 14, 2014 at 6:53 AM, Danushka Menikkumbura <
> > > danushka.menikkumbura@gmail.com <javascript:;>> wrote:
> > >
> > > > I am on Ubuntu 14.04.1 LTS and Java 1.8.0_20.
> > > >
> > > > BTW maybe it is a result of the commit
> > > > 3f98411364cec32a0a8c6681dfaabd43caa4dd60?.
> > > >
> > > > Thanks,
> > > > Danushka
> > > >
> > > >
> > > >
> > > > On Tue, Oct 14, 2014 at 9:44 AM, Josh Wills <jwills@cloudera.com
> <javascript:;>>
> > wrote:
> > > >
> > > > > No, master is compatible w/Hadoop 1; the error you're seeing is
> > caused
> > > by
> > > > > the HBase testing code being flaky. I've experienced the flaky
> > > > > HFileTargetIT test on my machine before, but never on a regular
> > basis.
> > > > Can
> > > > > you give me the basics of your setup-- OS, java version, etc.?
> > > > >
> > > > > On Tue, Oct 14, 2014 at 6:38 AM, Danushka Menikkumbura <
> > > > > danushka.menikkumbura@gmail.com <javascript:;>> wrote:
> > > > >
> > > > > > No I don't.
> > > > > >
> > > > > > That means the master branch is not compatible with Hadoop 1?
> > > > > >
> > > > > > Thanks,
> > > > > > Danushka
> > > > > >
> > > > > > On Tue, Oct 14, 2014 at 9:26 AM, Josh Wills <jwills@cloudera.com
> <javascript:;>>
> > > > wrote:
> > > > > >
> > > > > > > Hrm, okay. Do you get it if you use the -Dcrunch.platform=2
> > option?
> > > > > > >
> > > > > > > On Tue, Oct 14, 2014 at 6:23 AM, Danushka Menikkumbura
<
> > > > > > > danushka.menikkumbura@gmail.com <javascript:;>>
wrote:
> > > > > > >
> > > > > > > > Yes. I get it every time I try to build HBase module.
> > > > > > > >
> > > > > > > > FYI : org.apache.crunch.io.hbase.HFileTargetIT is
the test in
> > > > error.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Danushka
> > > > > > > >
> > > > > > > > On Tue, Oct 14, 2014 at 9:15 AM, Josh Wills <
> > jwills@cloudera.com <javascript:;>
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > That can happen intermittently if the local HBase
cluster
> > gets
> > > > hung
> > > > > > > up--
> > > > > > > > do
> > > > > > > > > you get it regularly (i.e., every time you run?)
> > > > > > > > >
> > > > > > > > > J
> > > > > > > > >
> > > > > > > > > On Tue, Oct 14, 2014 at 6:08 AM, Danushka Menikkumbura
<
> > > > > > > > > danushka.menikkumbura@gmail.com <javascript:;>>
wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I am getting the following test failure
while building
> > > Crunch.
> > > > > Have
> > > > > > > you
> > > > > > > > > got
> > > > > > > > > > an idea as to what may be the issue here?.
> > > > > > > > > >
> > > > > > > > > > 34787 [Thread-2057] INFO
> > > > > > > > > >
> > > > > > > >
> > > > org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob
> > > > > > -
> > > > > > > > > > Job status available at: http://localhost:8080/
> > > > > > > > > > 44548 [M:0;danushka:38318.oldLogCleaner]
ERROR
> > > > > > > > > > org.apache.hadoop.hbase.client.HConnectionManager
 -
> > > Connection
> > > > > not
> > > > > > > > found
> > > > > > > > > > in the list, can't delete it (connection
> > > > > > > > > >
> > > > key=HConnectionKey{properties={hbase.zookeeper.quorum=localhost,
> > > > > > > > > > hbase.rpc.timeout=60000,
> > > > > hbase.zookeeper.property.clientPort=57963,
> > > > > > > > > > zookeeper.znode.parent=/hbase,
> > > hbase.client.retries.number=350,
> > > > > > > > > > hbase.client.pause=100}, username='danushka'}).
May be
> the
> > > key
> > > > > was
> > > > > > > > > > modified?
> > > > > > > > > > java.lang.Exception
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:493)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:429)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276)
> > > > > > > > > > at org.apache.hadoop.hbase.Chore.run(Chore.java:94)
> > > > > > > > > > at java.lang.Thread.run(Thread.java:745)
> > > > > > > > > > Tests run: 1, Failures: 0, Errors: 0, Skipped:
0, Time
> > > elapsed:
> > > > > > > 21.625
> > > > > > > > > sec
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Danushka
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Director of Data Science
> > > > > > > > > Cloudera <http://www.cloudera.com>
> > > > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Director of Data Science
> > > > > > > Cloudera <http://www.cloudera.com>
> > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Director of Data Science
> > > > > Cloudera <http://www.cloudera.com>
> > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Director of Data Science
> > > Cloudera <http://www.cloudera.com>
> > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> > >
> >
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message