hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Leffel" <daniel.lef...@gmail.com>
Subject Re: HBase Master only listening to Local Connections on 60000?
Date Fri, 02 May 2008 17:02:01 GMT
Hi All,
Wanted to put the answer out there in case someone else was having the same
problem.

In looking at the /etc/hosts file, it's important that the hostname NOT map
to 127.0.0.1 there. Instead, map it to the real ip address that you want
regions servers to connect to the master on.

Cheers!

Danny

On Thu, May 1, 2008 at 12:51 PM, Daniel Leffel <daniel.leffel@gmail.com>
wrote:

> Also, in looking at the master, it clearly thinks it's ip address is
> 127.0.0.1, although it is accepting remote connections on port 60000 now.
>
>
> On Thu, May 1, 2008 at 12:47 PM, Daniel Leffel <daniel.leffel@gmail.com>
> wrote:
>
>> Switching to all IPs (both in hbase-site.xml and regionservers file)
>> causes the local region server and master to start up fine. The remote ones
>> start throwing the following exception (debug enabled):
>>
>> 2008-05-01 14:45:32,284 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>> Initializing RPC Metrics with hostName=60020, port=60020
>> 2008-05-01 14:45:32,326 DEBUG org.apache.hadoop.hbase.HRegionServer:
>> Telling master at 10.10.1.114:60000 that we are up
>> 2008-05-01 14:45:32,376 WARN org.apache.hadoop.hbase.HRegionServer: error
>> telling master we are up
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>>         at org.apache.hadoop.hbase.Leases.createLease(Leases.java:147)
>>         at
>> org.apache.hadoop.hbase.HMaster.regionServerStartup(HMaster.java:1318)
>>         at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:585)
>>         at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
>>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
>>
>>         at org.apache.hadoop.ipc.Client.call(Client.java:512)
>>         at
>> org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:210)
>>         at $Proxy0.regionServerStartup(Unknown Source)
>>         at
>> org.apache.hadoop.hbase.HRegionServer.reportForDuty(HRegionServer.java:1112)
>>         at
>> org.apache.hadoop.hbase.HRegionServer.run(HRegionServer.java:712)
>>         at java.lang.Thread.run(Thread.java:595)
>>
>>
>>
>>
>> On Thu, May 1, 2008 at 10:13 AM, stack <stack@duboce.net> wrote:
>>
>>> Daniel:
>>>
>>> The line out of RpcMetrics reporting port for hostname looks like a
>>> little bug we should fix up in hadoop core (I took a quick look; looks like
>>> the right args are being passed -- problem must be further up the stack or
>>> perhaps your hadoop is older?).
>>>
>>> Regards the next line reporting localhost, thats odd.  Is your networking
>>> setup properly?  What does the host report for IP when you query using the
>>> hostname you've inserted as the value for hbase.master?  What if you use IP
>>> instead?
>>>
>>> Here is what my logs looks like:
>>>
>>> 2008-05-01 04:15:34,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>> Initializing RPC Metrics with hostName=60000, port=60000
>>> 2008-05-01 04:15:34,361 INFO org.apache.hadoop.hbase.HMaster: HMaster
>>> initialized on XXX.XXX.44.139:60000
>>>
>>> Here is my hbase.master:
>>>
>>> aa0-XXX-XXX.at.our.domain:60000
>>>
>>> St.Ack
>>>
>>> P.S. And on offlining of tables, we're still working on it..... should
>>> have a candidate up in the next day or so w/ a fix included.
>>>
>>>
>>>
>>> Daniel Leffel wrote:
>>>
>>>> Actually, I did find something curious in the master log with debug
>>>> turned
>>>> on:
>>>>
>>>> 2008-05-01 11:05:56,407 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>>> Initializing RPC Metrics with hostName=60000, port=60000
>>>> 2008-05-01 11:05:56,482 INFO org.apache.hadoop.hbase.HMaster: HMaster
>>>> initialized on 127.0.0.1:60000
>>>>
>>>> hostname 60000? but then the next line shows localhost?
>>>>
>>>> On Thu, May 1, 2008 at 9:32 AM, Daniel Leffel <daniel.leffel@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>> Not sure what's going on with this and thought I'd ask. I can't find
>>>>> any
>>>>> rationale for this on the wiki or in the logs.
>>>>>
>>>>> I've begun to set up our production environment which is a multi-node
>>>>> setup. (To-date our development environment was a single node in
>>>>> psudeo-distributed mode).
>>>>>
>>>>> The HBase master is listening on port 60000 only for connections that
>>>>> are
>>>>> local and is refusing remote connections. All the other services on the
>>>>> box
>>>>> (60010,600020 and 600030) listen both locally and remotely. There's no
>>>>> firewall on the box.
>>>>>
>>>>> In the hbase-site.xml, I've specificed hbase.master to be
>>>>> [hostname]:60000
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message