hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh: No such file or directory
Date Thu, 01 Dec 2011 17:58:08 GMT
So since I don't see the rest of the log I'll have to assume that the
region server was never able to connect to the master. Connection
refused could be a firewall, start the master and then try to telnet
from the other machines to master:60000.

J-D

On Thu, Dec 1, 2011 at 6:45 AM, Vamshi Krishna <vamshi2105@gmail.com> wrote:
> I found in the logs of region server machines, i found this error (on both
> regionserver machines)
>
> 2011-11-30 14:43:42,447 INFO org.apache.hadoop.ipc.HbaseRPC: Server at
> hbase-master/10.0.1.54:60020 could not be reached after 1 tries, giving up.
> *2011-11-30 14:44:37,762* WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> java.net.ConnectException: Connection refused
>    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>    at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>    at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>    at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>    at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>    at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>    at $Proxy5.getProtocolVersion(Unknown Source)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>    at java.lang.Thread.run(Thread.java:662)
>  2011-11-30 14:44:40,768 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
> Master server at hbase-master:60000
> *2011-11-30 14:45:40,847* WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
> master. Retrying. Error was:
> java.net.ConnectException: Connection refused
>    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>    at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>    at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>    at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>    at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
>    at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
>    at $Proxy5.getProtocolVersion(Unknown Source)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
>    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>    at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>    at java.lang.Thread.run(Thread.java:662)
>
>
> and the same error is observed in the whole log repeatedly. After seeing it
> what  i understand is that some how master started HRegionServer daemons on
> the machines but from then onwards the RegionServer machines are not able
> to communicate with master. If we observe it is trying to communicate with
> master for evry one minute.
>
> But i am not understanding where to check and modify the things.. please
> help. i feel all connections are OK.
>
> On Thu, Dec 1, 2011 at 12:28 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> stop-hbase.sh only tells the master to stop, which in turn will tell
>> the region servers to stop. If they are still running, it might be
>> because of an error. Look at their logs to figure what's going on.
>>
>> J-D
>>
>> On Tue, Nov 29, 2011 at 10:46 PM, Vamshi Krishna <vamshi2105@gmail.com>
>> wrote:
>> > hey soryy for posting multiple times.
>> > J-D, As you said, i refered to my regionserver log, there i found
>> >              Could not resolve the DNS name of vamshikrishna-desktop
>> > so i added an alias ' vamshikrishna-desktop ' to its corresponding IP
>> > address in /etc/hosts.  So, from then master is able to run HRegionServer
>> > daemon in the regionserver machines also.
>> >
>> > But the ONLY problem now is when i stop hbase on my master node by
>> running
>> > bin/stop-hbase.sh, all hbase daemons are stopping on matser node but NOT
>> on
>> > regionserver nodes.The HRegionServer daemon is still running on the other
>> > regionserver machines.
>> > I think the HRegionServer daemons on all the regionserver machines should
>> > also get stopped, when we execute bin/sto-hbase.sh on master node, right?
>> > please help..
>> >
>> > On Wed, Nov 30, 2011 at 12:38 AM, Jean-Daniel Cryans <
>> jdcryans@apache.org>wrote:
>> >
>> >> You posted this in two threads, please refrain from doing this in the
>> >> future.
>> >>
>> >> From what I can read in there, it tried to get the master address
>> >> that's supposed to be in zookeeper but it failed because it was
>> >> missing and then died. The way it's handled is a bit ugly but the
>> >> effect is the same, take a look at the master log to see what's going
>> >> on.
>> >>
>> >> J-D
>> >>
>> >> On Mon, Nov 28, 2011 at 10:33 PM, Vamshi Krishna <vamshi2105@gmail.com>
>> >> wrote:
>> >> > Hi Lars,
>> >> > i am not using cygwin, i am using 3 ubuntu-10.04 machines.
>> >> > Finally that problem i mentioned got resolved i.e now i can see the
>> >> > following after i run bin/start-hbase.sh on my master machine,
>> >> >
>> >> > hbase-master: starting zookeeper, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-zookeeper-vamshikrishna-desktop.out
>> >> > starting master, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/logs/hbase-hduser-master-vamshikrishna-desktop.out
>> >> >  hbase-regionserver2: starting regionserver, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.out
>> >> > hbase-regionserver1: starting regionserver, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.out
>> >> >
>> >> > hbase-master: starting regionserver, logging to
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshikrishna-desktop.out
>> >> >
>> >> >
>> >> >
>> >> > But still , if i do jps on both region server machines, HRegionServer
>> >> > daemon is not seen.And i checked the logs on both the regionservere
>> >> > machines, i.e
>> >> >
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-vamshi-laptop.log
>> >> > ---(hbase-regionserver1)
>> >> > and
>> >> >
>> >>
>> /home/hduser/Documents/HBASE_SOFTWARE/hbase-0.90.4/bin/../logs/hbase-hduser-regionserver-arun-desktop.log
>> >> > ---(hbase-regionserver2)
>> >> >
>> >> > i found the same in both .it is like this,   (here
>> vamshikrishna-desktop
>> >> is
>> >> > the master and namenode machine)
>> >> >
>> >> >
>> >> >
>> >> > 2011-11-29 11:24:36,714 ERROR org.apache.hadoop.hbase.HServerAddress:
>> >> Could
>> >> > not resolve the DNS name of vamshikrishna-desktop
>> >> > 2011-11-29 11:24:36,716 FATAL
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region
>> >> server
>> >> > serverName=vamshi-laptop,60020,1322546076276, load=(requests=0,
>> >> regions=0,
>> >> > usedHeap=0, maxHeap=0): Unhandled exception: hostname can't be null
>> >> > java.lang.IllegalArgumentException: hostname can't be null
>> >> >    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:121)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108)
>> >> >    at
>> >> org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:63)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMasterAddress(HRegionServer.java:1484)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1457)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>> >> >    at java.lang.Thread.run(Thread.java:662)
>> >> > 2011-11-29 11:24:36,719 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
>> >> > exception: hostname can't be null
>> >> > 2011-11-29 11:24:36,719 INFO org.apache.hadoop.ipc.HBaseServer:
>> Stopping
>> >> > server on 60020
>> >> > 2011-11-29 11:24:36,749 INFO
>> >> > org.apache.hadoop.hbase.regionserver.StoreFile: Allocating
>> LruBlockCache
>> >> > with maximum size 199.4m
>> >> > 2011-11-29 11:24:36,775 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server
>> at:
>> >> > vamshi-laptop,60020,1322546076276
>> >> > 2011-11-29 11:24:36,775 DEBUG
>> >> > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog
>> tracker
>> >> > org.apache.hadoop.hbase.catalog.CatalogTracker@da2cef
>> >> > 2011-11-29 11:24:36,776 INFO
>> org.apache.hadoop.hbase.regionserver.Leases:
>> >> > regionserver60020 closing leases
>> >> > 2011-11-29 11:24:36,776 INFO
>> org.apache.hadoop.hbase.regionserver.Leases:
>> >> > regionserver60020 closed leases
>> >> > 2011-11-29 11:24:36,776 INFO
>> >> >
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >> > Closed zookeeper sessionid=0x133ede31f3c0004
>> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ClientCnxn:
>> EventThread
>> >> > shut down
>> >> > 2011-11-29 11:24:36,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> >> > 0x133ede31f3c0004 closed
>> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ClientCnxn:
>> EventThread
>> >> > shut down
>> >> > 2011-11-29 11:24:36,789 INFO org.apache.zookeeper.ZooKeeper: Session:
>> >> > 0x133ede31f3c0005 closed
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> starting;
>> >> > hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-14,5,main]
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> >> hook
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> shutdown
>> >> > hook thread.
>> >> > 2011-11-29 11:24:36,792 INFO
>> >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> >> finished.
>> >> >
>> >> > what is the cause for problem here, please help.
>> >> >
>> >> > On Mon, Nov 28, 2011 at 7:28 PM, Lars George <lars.george@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> Did you add the list of servers to the regionservers file in the
>> >> >> $HBASE_HOME/conf/ dir? Are you using Cygwin? Or what else is your
>> >> >> environment?
>> >> >>
>> >> >> Lars
>> >> >>
>> >> >> On Nov 26, 2011, at 7:37 AM, Vamshi Krishna wrote:
>> >> >>
>> >> >> > Hi i am running hbase on 3 machines, on one node master and
>> >> regionserver,
>> >> >> > on other two nodes regionservers. i ran bin/start-hbase.sh
on
>> master,
>> >> all
>> >> >> > the hbase daemons running on master node(HMaster, HQuorumpeer,
>> >> >> > HRegionServer), But on the other regionserver machines i could
not
>> >> find
>> >> >> any
>> >> >> > hbase daemons running. According to my knowledge the HRegionserver
>> >> >> daemons
>> >> >> > should run on other machines automatically when i run
>> >> bin/start-hbase.sh
>> >> >> on
>> >> >> > master node, right? But its not happening so.
>> >> >> > More over its printing errors on my master node when i run
>> >> >> > bin/start-hbase.sh
>> >> >> >
>> >> >> > hbase-regionserver1: bash: line 0: cd: {HBASE_HOME}/bin/..:
No such
>> >> file
>> >> >> or
>> >> >> > directory
>> >> >> > hbase-regionserver1: bash: {HBASE_HOME}/bin/hbase-daemon.sh:
No
>> such
>> >> file
>> >> >> > or directory
>> >> >> > hbase-regionserver2: bash: line 0: cd: {HBASE_HOME}/bin/..:
No such
>> >> file
>> >> >> or
>> >> >> > directory
>> >> >> > hbase-regionserver2: bash: {HBASE_HOME}/bin/hbase-daemon.sh:
No
>> such
>> >> file
>> >> >> > or directory  ,
>> >> >> >
>> >> >> > but i could find out {HBASE_HOME}/bin/hbase-daemon.sh on both
>> >> machines.
>> >> >> > Infact the path of the {HBASE_HOME} folder on each of the
>> respective
>> >> >> > machines machines is same. i.e
>> >> >> > /home/hduser/Documents/HBASE_SOFTWRAE/hbase-0.90.4
>> >> >> >
>> >> >> > please can anybody help?
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > *Regards*
>> >> >> > *
>> >> >> > Vamshi Krishna
>> >> >> > *
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > *Regards*
>> >> > *
>> >> > Vamshi Krishna
>> >> > *
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > *Regards*
>> > *
>> > Vamshi Krishna
>> > *
>>
>
>
>
> --
> *Regards*
> *
> Vamshi Krishna
> *

Mime
View raw message