hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruben Quintero <rfq_...@yahoo.com>
Subject Re: hbase -0.90.x upgrade - zookeeper exception in mapreduce job
Date Tue, 12 Apr 2011 22:09:05 GMT
I'm running into the same issue, but did some poking around and it seems that 
Zookeeper connections are being left open by an HBase internal.

Basically, I'm running a mapreduce job within another program, and noticed in 
the logs that every time the job is run, a connection is open, but I never see 
it closed again. The connection is opened within the job.submit().

I looked closer and checked the jstack after running it for just under an hour, 
and sure enough there are a ton of Zookeeper threads just sitting there. Here's  
a pastebin link: http://pastebin.com/MccEuvrc

I'm running 0.90.0 right now.

- Ruben






________________________________
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Sent: Tue, April 12, 2011 4:23:05 PM
Subject: Re: hbase -0.90.x upgrade - zookeeper exception in mapreduce job

It's more in the vain of
https://issues.apache.org/jira/browse/HBASE-3755 and
https://issues.apache.org/jira/browse/HBASE-3771

Basically 0.90 has a regression regarding the handling of zookeeper
connections that make it that you have to be super careful not to have
more than 30 per machine (each new Configuration is one new ZK
connection). Upping your zookeeper max connection config should get
rid of your issues since you only get it occasionally.

J-D

On Tue, Apr 12, 2011 at 7:59 AM, Venkatesh <vramanathan00@aol.com> wrote:
>
>  I get this occasionally..(not all the time)..Upgrading from 0.20.6 to 0.90.2
> Is this issue same as this JIRA
> https://issues.apache.org/jira/browse/HBASE-3578
>
> I'm using HBaseConfiguration.create() & setting that in job
> thx
> v
>
>
>  2011-04-12 02:13:06,870 ERROR Timer-0 
>org.apache.hadoop.hbase.mapreduce.TableInputFormat - 
>org.apache.hadoop.hbase.ZooKeeperConnectionException: 
>org.apache.hadoop.hbase.ZooKeeperConnectionException: 
>org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = 
>ConnectionLoss for /hbase        at 
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1000)
>
>        at 
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
>
>        at 
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
>
>        at 
>org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
>        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>        at 
>org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91)
>
>        at 
>org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>        at 
>org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>        at 
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:882)
>        at 
>org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>        at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
>        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:448)
>
>
>
>



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message