Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9DE2A9E4E for ; Fri, 10 Feb 2012 22:48:17 +0000 (UTC) Received: (qmail 44516 invoked by uid 500); 10 Feb 2012 22:48:15 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 44425 invoked by uid 500); 10 Feb 2012 22:48:15 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 44417 invoked by uid 99); 10 Feb 2012 22:48:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Feb 2012 22:48:14 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.210.169 as permitted sender) Received: from [209.85.210.169] (HELO mail-iy0-f169.google.com) (209.85.210.169) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Feb 2012 22:48:08 +0000 Received: by iagz16 with SMTP id z16so6042026iag.14 for ; Fri, 10 Feb 2012 14:47:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=mMwiMv7o7rDKXYfqIqmLD2qY9rDgQF5qlMtlQ4xUzp0=; b=JiAiJMIC8RkHp+34QADQJN0vKWA99GcKWeLMmApc2Eq0tr8bollMMWolUT4p+3qBor A1/8/6fhoE6+A4xTR2FufMDlIDk/fW58MMCCQlmn337eRqvDP89JaCE1dcAeUHvh/riZ GhETCt9wNIXLTXQN9Q7pGnu3UFXb4ROH2Kc5E= MIME-Version: 1.0 Received: by 10.50.203.66 with SMTP id ko2mr14573424igc.7.1328914067614; Fri, 10 Feb 2012 14:47:47 -0800 (PST) Sender: jdcryans@gmail.com Received: by 10.231.229.6 with HTTP; Fri, 10 Feb 2012 14:47:47 -0800 (PST) In-Reply-To: References: Date: Fri, 10 Feb 2012 14:47:47 -0800 X-Google-Sender-Auth: XuxUATCHAyZ5y8x-KXegQWMqD94 Message-ID: Subject: Re: Thrift on regionservers - won't connect From: Jean-Daniel Cryans To: user@hbase.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable So is the master alive? On Fri, Feb 10, 2012 at 12:21 PM, Tomas Tillery wr= ote: > I stand corrected - The region servers were still connected because they > had remained connected. Restarting them resulted in them also receiving a > Connection refused response. Error log follows. > > 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Opening socket connection to > server master/192.168.1.1:2181 > 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Socket connection establishe= d > to master/192.168.1.1:2181, initiating session > 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Session establishment comple= te > on server master/192.168.1.1:2181, sessionid =3D 0x13568dc6dfe0005, > negotiated timeout =3D 40000 > 12/02/10 20:12:24 INFO ipc.HBaseRpcMetrics: Initializing RPC Metrics with > hostName=3DHRegionServer, port=3D60020 > 12/02/10 20:12:24 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=3Dmaster:2181 sessionTimeout=3D180000 watcher=3Dregionserve= r:60020 > 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Opening socket connection to > server master/192.168.1.1:2181 > 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Socket connection establishe= d > to master/192.168.1.1:2181, initiating session > 12/02/10 20:12:24 INFO zookeeper.ClientCnxn: Session establishment comple= te > on server master/192.168.1.1:2181, sessionid =3D 0x13568dc6dfe000c, > negotiated timeout =3D 40000 > 12/02/10 20:12:24 INFO regionserver.ShutdownHook: Installed shutdown hook > thread: Shutdownhook:regionserver60020 > 12/02/10 20:12:24 INFO regionserver.MemStoreFlusher: > globalMemStoreLimit=3D395.0m, globalMemStoreLimitLowMark=3D345.6m, > maxHeap=3D987.5m > 12/02/10 20:12:24 INFO regionserver.HRegionServer: Runs every 10000000ms > 12/02/10 20:12:24 INFO regionserver.HRegionServer: Attempting connect to > Master server at master:60000 > 12/02/10 20:13:24 WARN regionserver.HRegionServer: Unable to connect to > master. Retrying. Error was: > java.net.ConnectException: Connection refused > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method= ) > =A0 =A0 =A0 =A0at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > =A0 =A0 =A0 =A0at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.jav= a:206) > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:40= 8) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseCl= ient.java:328) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:88= 3) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) > =A0 =A0 =A0 =A0at $Proxy6.getProtocolVersion(Unknown Source) > =A0 =A0 =A0 =A0at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.= java:419) > =A0 =A0 =A0 =A0at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.= java:393) > =A0 =A0 =A0 =A0at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.= java:444) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServe= r.java:1477) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionS= erver.java:1530) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegi= onServer.java:1514) > =A0 =A0 =A0 =A0at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java= :573) > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619) > > On Thu, Feb 9, 2012 at 2:48 PM, Tomas Tillery wro= te: > >> That appears to be the case. Here's an excerpt from the regionserver log >> on the same machine: >> >> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Attempting connect to >> Master server at master:60000 >> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Connected to master a= t >> master:60000 >> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Telling master at >> master:60000 that we are up >> 12/01/24 22:11:44 INFO regionserver.HRegionServer: Master passed us >> address to use. Was=3Dregion-1:60020, Now=3Dregion-1:60020 >> >> >> On Thu, Feb 9, 2012 at 2:43 PM, Jean-Daniel Cryans = wrote: >> >>> 60010 is the web ui port by default, 60000 is the port the master uses >>> for communication. Clients (like thrift) and region servers don't rely >>> on a configured port because the master already posts it in zookeeper >>> along with its address (that's how your thrift server found it). >>> >>> If your region servers are using master:60000 and they don't have any >>> problem connecting to a port that doesn't respond, we have a bigger >>> issue here :) >>> >>> J-D >>> >>> On Thu, Feb 9, 2012 at 11:38 AM, Tomas Tillery >>> wrote: >>> > No, it isn't - as far as I know, it should be aiming for 60010, rathe= r >>> than >>> > 60000 (though HBase is also bound to 60000, there's no telnet respons= e >>> > there). There's probably something obvious I'm not managing to see, b= ut >>> I >>> > can't find a way to redirect it to the proper port. >>> > Starting thrift with this install is a matter of invoking >>> > /etc/init.d/hadoop-hbase-thrift. Since it isn't invoked through the >>> shell >>> > script, I'm not sure where to specify the port to connect to. >>> > >>> > On Thu, Feb 9, 2012 at 2:27 PM, Jean-Daniel Cryans >> >wrote: >>> > >>> >> Connection refused is a java exception that usually means that eithe= r >>> >> there's nothing listening on the port or it's firewall'd. Is >>> >> "master:60000" really what you're expecting to see here? Should be >>> >> easy to "telnet master 60000" from that machien to really verify tha= t >>> >> it works as expected. >>> >> >>> >> J-D >>> >> >>> >> On Thu, Feb 9, 2012 at 11:19 AM, Tomas Tillery >> > >>> >> wrote: >>> >> > Hello, all >>> >> > >>> >> > It's my understanding that it's best practice to colocate thrift o= n >>> each >>> >> of >>> >> > the regionservers, and use load balancing to spread queries betwee= n >>> them. >>> >> > However, I can't seem to get thrift to bind from the regionservers >>> (only >>> >> > from the hmaster). >>> >> > >>> >> > The machines have connectivity between them; regionservers connect >>> and >>> >> > communicate with the hmaster without any problems. >>> >> > >>> >> > I'm running Hadoop 0.20.2-cdh3u2, HBase 0.90.4-cdh3u2. >>> >> > >>> >> > Thrift can run succesfully from the HMaster, but not from any of t= he >>> >> > datanode/regionservers >>> >> > >>> >> > What appears to be the relevant content from >>> >> > hbase-hbase-thrift-region-1.out below: >>> >> > >>> >> > 12/02/09 19:09:17 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 0 of 10 failed; retrying after sleep of 1000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:18 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 1 of 10 failed; retrying after sleep of 1000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:19 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 2 of 10 failed; retrying after sleep of 1000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:20 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 3 of 10 failed; retrying after sleep of 2000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:22 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 4 of 10 failed; retrying after sleep of 2000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:24 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 5 of 10 failed; retrying after sleep of 4000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:28 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 6 of 10 failed; retrying after sleep of 4000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:32 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 7 of 10 failed; retrying after sleep of 8000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:40 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 8 of 10 failed; retrying after sleep of 16000 >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > 12/02/09 19:09:56 INFO >>> >> client.HConnectionManager$HConnectionImplementation: >>> >> > getMaster attempt 9 of 10 failed; no more retrying. >>> >> > java.net.ConnectException: Connection refused >>> >> > =A0 =A0 =A0 =A0at sun.nio.ch.SocketChannelImpl.checkConnect(Native= Method) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.j= ava:206) >>> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.net.NetUtils.connect(NetUtils.= java:408) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBase= Client.java:328) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:= 883) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) >>> >> > =A0 =A0 =A0 =A0at $Proxy4.getProtocolVersion(Unknown Source) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) >>> >> > =A0 =A0 =A0 =A0at >>> >> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:360) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> > Exception in thread "main" >>> >> > org.apache.hadoop.hbase.MasterNotRunningException: master:60000 >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa= tion.getMaster(HConnectionManager.java:396) >>> >> > =A0 =A0 =A0 =A0at >>> >> > org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:9= 4) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:194) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> >> >>> org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.(ThriftS= erver.java:188) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:87= 0) >>> >> > =A0 =A0 =A0 =A0at >>> >> > >>> org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927) >>> >> >>> >> >>