crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Test failure on the master branch
Date Wed, 15 Oct 2014 12:41:32 GMT
Yeah, I'm right: we needed the 5711 hack in there ~ a million years ago to
get the WordCountHBaseIT to work:
https://www.google.com/search?q=crunch+hbase-5711

I wonder why it's back now.

On Wed, Oct 15, 2014 at 5:33 AM, Josh Wills <jwills@cloudera.com> wrote:

> So applying the bit mask workaround from HBASE-5711 fixes the
> HFileTargetIT test, but I have this deja vu-like feeling that we ran into
> something like this before, but the Internet at this hotel isn't quite
> solid enough for me to figure out when it was. Will keep poking around.
>
> J
>
> On Tue, Oct 14, 2014 at 9:17 PM, Josh Wills <jwills@cloudera.com> wrote:
>
>> I can replicate the original error- looks like dfs.data.dir permissions
>> aren't set properly in test mode for some reason. Will investigate and
>> figure out a patch.
>>
>>
>> On Tuesday, October 14, 2014, Micah Whitacre <mkwhit@gmail.com> wrote:
>>
>>> One thing I noticed is that is when testing we don't force IPv4.
>>> Zookeeper
>>> does not handle IPv6 well and can actually significantly slow down tests
>>> trying to resolve the localhost.  This doesn't affect everyone because
>>> your
>>> /etc/hosts could help it resolve faster but does make machine setup
>>> important.  We could try adding an argline like this to surefire/failsafe
>>> to enforce IPv4.
>>>
>>> <argLine>-Xmx512m -XX:MaxPermSize=128m
>>> -Djava.net.preferIPv4Stack=true</argLine>
>>>
>>> On Tue, Oct 14, 2014 at 9:28 AM, Danushka Menikkumbura <
>>> danushka.menikkumbura@gmail.com> wrote:
>>>
>>> > Thanks J!
>>> >
>>> > On Tue, Oct 14, 2014 at 9:57 AM, Josh Wills <jwills@cloudera.com>
>>> wrote:
>>> >
>>> > > I'm still rocking 1.7. Will give 1.8 a whirl this evening.
>>> > >
>>> > > J
>>> > >
>>> > > On Tue, Oct 14, 2014 at 6:53 AM, Danushka Menikkumbura <
>>> > > danushka.menikkumbura@gmail.com> wrote:
>>> > >
>>> > > > I am on Ubuntu 14.04.1 LTS and Java 1.8.0_20.
>>> > > >
>>> > > > BTW maybe it is a result of the commit
>>> > > > 3f98411364cec32a0a8c6681dfaabd43caa4dd60?.
>>> > > >
>>> > > > Thanks,
>>> > > > Danushka
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Tue, Oct 14, 2014 at 9:44 AM, Josh Wills <jwills@cloudera.com>
>>> > wrote:
>>> > > >
>>> > > > > No, master is compatible w/Hadoop 1; the error you're seeing
is
>>> > caused
>>> > > by
>>> > > > > the HBase testing code being flaky. I've experienced the
flaky
>>> > > > > HFileTargetIT test on my machine before, but never on a regular
>>> > basis.
>>> > > > Can
>>> > > > > you give me the basics of your setup-- OS, java version,
etc.?
>>> > > > >
>>> > > > > On Tue, Oct 14, 2014 at 6:38 AM, Danushka Menikkumbura <
>>> > > > > danushka.menikkumbura@gmail.com> wrote:
>>> > > > >
>>> > > > > > No I don't.
>>> > > > > >
>>> > > > > > That means the master branch is not compatible with
Hadoop 1?
>>> > > > > >
>>> > > > > > Thanks,
>>> > > > > > Danushka
>>> > > > > >
>>> > > > > > On Tue, Oct 14, 2014 at 9:26 AM, Josh Wills <
>>> jwills@cloudera.com>
>>> > > > wrote:
>>> > > > > >
>>> > > > > > > Hrm, okay. Do you get it if you use the -Dcrunch.platform=2
>>> > option?
>>> > > > > > >
>>> > > > > > > On Tue, Oct 14, 2014 at 6:23 AM, Danushka Menikkumbura
<
>>> > > > > > > danushka.menikkumbura@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > > > Yes. I get it every time I try to build HBase
module.
>>> > > > > > > >
>>> > > > > > > > FYI : org.apache.crunch.io.hbase.HFileTargetIT
is the test
>>> in
>>> > > > error.
>>> > > > > > > >
>>> > > > > > > > Thanks,
>>> > > > > > > > Danushka
>>> > > > > > > >
>>> > > > > > > > On Tue, Oct 14, 2014 at 9:15 AM, Josh Wills
<
>>> > jwills@cloudera.com
>>> > > >
>>> > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > That can happen intermittently if the
local HBase cluster
>>> > gets
>>> > > > hung
>>> > > > > > > up--
>>> > > > > > > > do
>>> > > > > > > > > you get it regularly (i.e., every time
you run?)
>>> > > > > > > > >
>>> > > > > > > > > J
>>> > > > > > > > >
>>> > > > > > > > > On Tue, Oct 14, 2014 at 6:08 AM, Danushka
Menikkumbura <
>>> > > > > > > > > danushka.menikkumbura@gmail.com> wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Hi,
>>> > > > > > > > > >
>>> > > > > > > > > > I am getting the following test
failure while building
>>> > > Crunch.
>>> > > > > Have
>>> > > > > > > you
>>> > > > > > > > > got
>>> > > > > > > > > > an idea as to what may be the issue
here?.
>>> > > > > > > > > >
>>> > > > > > > > > > 34787 [Thread-2057] INFO
>>> > > > > > > > > >
>>> > > > > > > >
>>> > > >
>>> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob
>>> > > > > > -
>>> > > > > > > > > > Job status available at: http://localhost:8080/
>>> > > > > > > > > > 44548 [M:0;danushka:38318.oldLogCleaner]
ERROR
>>> > > > > > > > > > org.apache.hadoop.hbase.client.HConnectionManager
 -
>>> > > Connection
>>> > > > > not
>>> > > > > > > > found
>>> > > > > > > > > > in the list, can't delete it (connection
>>> > > > > > > > > >
>>> > > > key=HConnectionKey{properties={hbase.zookeeper.quorum=localhost,
>>> > > > > > > > > > hbase.rpc.timeout=60000,
>>> > > > > hbase.zookeeper.property.clientPort=57963,
>>> > > > > > > > > > zookeeper.znode.parent=/hbase,
>>> > > hbase.client.retries.number=350,
>>> > > > > > > > > > hbase.client.pause=100}, username='danushka'}).
May be
>>> the
>>> > > key
>>> > > > > was
>>> > > > > > > > > > modified?
>>> > > > > > > > > > java.lang.Exception
>>> > > > > > > > > > at
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:493)
>>> > > > > > > > > > at
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:429)
>>> > > > > > > > > > at
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141)
>>> > > > > > > > > > at
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276)
>>> > > > > > > > > > at org.apache.hadoop.hbase.Chore.run(Chore.java:94)
>>> > > > > > > > > > at java.lang.Thread.run(Thread.java:745)
>>> > > > > > > > > > Tests run: 1, Failures: 0, Errors:
0, Skipped: 0, Time
>>> > > elapsed:
>>> > > > > > > 21.625
>>> > > > > > > > > sec
>>> > > > > > > > > >
>>> > > > > > > > > > Thanks,
>>> > > > > > > > > > Danushka
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > --
>>> > > > > > > > > Director of Data Science
>>> > > > > > > > > Cloudera <http://www.cloudera.com>
>>> > > > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > --
>>> > > > > > > Director of Data Science
>>> > > > > > > Cloudera <http://www.cloudera.com>
>>> > > > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Director of Data Science
>>> > > > > Cloudera <http://www.cloudera.com>
>>> > > > > Twitter: @josh_wills <http://twitter.com/josh_wills>
>>> > > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Director of Data Science
>>> > > Cloudera <http://www.cloudera.com>
>>> > > Twitter: @josh_wills <http://twitter.com/josh_wills>
>>> > >
>>> >
>>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message