hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: Zookeeper connection error on mapreduce HBase writes
Date Sun, 27 Mar 2011 08:03:27 GMT
This stack trace really starts getting familiar. Check next email
threads, it is likely caused by zookeeper connection starvation.

Of course moving htable to setup method is the right call. But i would
also like to point out that if you are using hbase 0.90, each new
instance of Htable(conf) will create a new unshared hbase connection
in your case which needs relinquishing. Since you never closed one, it
would seem you ran out of max allowed # of zookeeper connections from
the same ip almost instantly.

Also, TableInputFormat opens and doesn't close hbase connection on the
front end, so it turns out it leaks one connection per MR run in the
front end. But since it seems you are recycling jvm per job, it
wouldn't be an issue in your case.

On Fri, Mar 25, 2011 at 1:19 PM, Jonathan Bender
<jonathan.bender@gmail.com> wrote:
> I actually created another Configuration object (cfg) within the map()
> method itself, so it still worked.  Now I have a much better idea of how the
> Mapper is called.
>
> Moving the HTable object configuration to the setup() method was the right
> call.  Thanks!
>
> On Fri, Mar 25, 2011 at 12:01 PM, Buttler, David <buttler1@llnl.gov> wrote:
>
>> I would suggest that you have each mapper have its own HTable, rather than
>> having a static HTable in the outer class.  Configure it from the setup
>> method of the mapper.
>>
>> Hmm..., I am not exactly sure how the configuration from your HTable is
>> passed to the mapper in the first place.  You are configuring it in the
>> DomainTableTransform which is not run on when instantiating the individual
>> mappers (hence it is a public static class).  I don't think your code
>> actually works at all, even for a little bit.
>>
>> Dave
>>
>>
>> -----Original Message-----
>> From: Jonathan Bender [mailto:jonathan.bender@gmail.com]
>> Sent: Friday, March 25, 2011 11:16 AM
>> To: user@hbase.apache.org
>> Subject: Zookeeper connection error on mapreduce HBase writes
>>
>> Hello all,
>>
>> I wrote a routine that scans an HBase table, and writes to another table
>> from within the map function using HTable.put().  When I run the job, it
>> works fine for the first few rows but ZooKeeper starts having issues
>> opening
>> up a connection after a while.
>>
>> Am I just overloading the ZK server by opening up a new HTable connection
>> during each map() task, or is there something else wrong with my
>> configuration?  Any other suggestions for reading and writing tables
>> directly via MapReduce?
>>
>> Here's the syslog from my tasktracker node: http://pastebin.com/VzUs3TJP
>>
>> And the error log: http://pastebin.com/9gBRyR4e
>>
>> And here's the MapReduce code I am running: http://pastebin.com/ir3yWaR1
>>
>> Excerpt of the error:
>>
>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase
>>  at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
>>  at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
>> at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
>>  at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>>  at
>>
>> com.mydomain.emir.mapreduce.DomainTableTransform$myMapper.map(DomainTableTransform.java:82)
>> at
>>
>> com.mydomain.emir.mapreduce.DomainTableTransform$myMapper.map(DomainTableTransform.java:1)
>>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
>>  at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>  at
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>> at org.apache.hadoop.mapred.Child.main(Child.java:234)
>> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase
>> at
>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
>>  at
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:986)
>> ... 15 more
>>
>

Mime
View raw message