hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hegner, Travis" <THeg...@trilliumit.com>
Subject Client Connection to Zookeeper times out during mapreduce
Date Thu, 15 Jul 2010 14:14:54 GMT
Hi All!

I apologize in advance for what is probably a rookie mistake, but it's been a while since
I've done a map reduce with Hbase as input or output. I'm assuming that this is a cluster
config problem because even the built-in "rowcounter" mapreduce job is failing. The hbase
shell is operating normally, and I can even access Hbase from thrift.

Cluster:
1 master: NN, JT, ZK, HM
5 slaves: DN, TT, RS

When attempting to run a mapreduce job with Hbase as a source, my map tasks are failing with
the following logged exceptions:

2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_000000_0/work
2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection,
connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@80fa6f
2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
is false
2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server
localhost/127.0.0.1:2181
2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x0
to sun.nio.ch.SelectionKeyImpl@1d85f79
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown
input
java.nio.channels.ClosedChannelException
                at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
                at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
                at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown
output
java.nio.channels.ClosedChannelException
                at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
                at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
                at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:22,532 WARN org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to
create /hbase -- check quorum servers, currently=localhost:2181
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
for /hbase
                at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
                at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
                at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
                at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808)
                at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405)
                at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432)
                at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634)
                at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601)
                at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134)
                at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112)
                at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:87)
                at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
                at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
                at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:573)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
                at org.apache.hadoop.mapred.Child.main(Child.java:170)
2010-07-15 09:16:22,689 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server
localhost/0:0:0:0:0:0:0:1:2181
2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x0
to sun.nio.ch.SelectionKeyImpl@17ce4e7
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown
input
java.nio.channels.ClosedChannelException
                at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
                at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
                at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:22,690 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown
output
java.nio.channels.ClosedChannelException
                at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
                at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
                at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:24,056 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server
localhost/127.0.0.1:2181
2010-07-15 09:16:24,056 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x0
to sun.nio.ch.SelectionKeyImpl@982589
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown
input
java.nio.channels.ClosedChannelException
                at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
                at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
                at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-07-15 09:16:24,057 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown
output
java.nio.channels.ClosedChannelException
                at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
                at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
                at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)

Where I'm confused is that the log says it's connecting to the quorum at 'localhost', but
the quorum is configured only on the master node. All DNS, rDNS is configured properly. The
hadoop-env.sh file has the zookeeper.jar in the classpath on all of the slaves.

Can anyone see what I'm missing?

Thanks,
Travis Hegner
http://www.travishegner.com/


________________________________
The information contained in this communication is confidential and is intended only for the
use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited
and may be unlawful. If you have received this communication in error, you should know that
you are bound to confidentiality, and should please immediately notify the sender or our IT
Department at 866.459.4599.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message