hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: hbase standalone cannot start master, cannot assign requested address at port 60000
Date Thu, 16 Sep 2010 20:51:03 GMT
Hey,

Ok the picture is all clear.

So HBase is a minimally configured system... You dont want to specify
the bind address in your config file, because usually you have 1 file
that you distribute to dozens or even potentially hundreds of systems.
 So specifying configuration for 1 system is just not really the way
to go with clustered software.

So what does hbase do?  We need to know the node's identity so when we
register ourselves we know what our IP is, and that IP goes into the
META table.  So we grab the hostname (as per 'hostname' on most
systems).  Then reverse DNS it, use that IP to bind to.

In this case, the problem is your hostname is reversing to the
external IP which your host doesnt actually have an interface to.  If
you want to run internal network services behind a NAT you will need
to have local IPs and hostnames and not reuse your external name/IP as
internal hostnames.

So, change your hostname to 'myhost' and make sure it DNS reverses to
10.0.0.2 (your real IP) and you should be off to the races.

-ryan

On Thu, Sep 16, 2010 at 1:06 PM, Michael Scott <mjscottuic@gmail.com> wrote:
> Thanks again.  This changes the behavior, but it does not yet fix my
> problem.  The hbase.rootdir property forces the hbase master to stay alive
> for a little while, so I had a moment of short-lived euphoria when Hmaster
> appeared in the jps list, but this only lasts while it tries to connect to
> localhost:9000 (which is not open), and it still doesn't open port 60000 and
> it still thinks it is named my-static-ip.com (i.e., same error message as
> before).  The removal of localhost.localdomain from /etc/hosts made no
> difference one way or the other.  I still am looking for a way to try to
> have hbase bind to localhost:6000 instead of my-static-ip.com:6000.  I will
> also try to see why localhost:9000 is not open (though that appears later in
> the log file, so I don't think it is causing the failure to open 60000).
> Thanks for the help so far, I will post again with further info.
>
> Michael
>
> On Thu, Sep 16, 2010 at 12:53 PM, N.N. Gesli <nngesli@gmail.com> wrote:
>
>> I have this in hbase-site.xml:
>>
>>  <property>
>>    <name>hbase.rootdir</name>
>>    <value>hdfs://localhost:9000/hbase</value>
>>    <description>The directory shared by region servers.
>>    Should be fully-qualified to include the filesystem to use.
>>    E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR
>>    </description>
>>  </property>
>>
>>  <property>
>>    <name>hbase.cluster.distributed</name>
>>    <value>true</value>
>>    <description>For psuedo-distributed, you want to set this to true.
>>    false means that HBase tries to put Master + RegionServers in one
>> process.
>>    Pseudo-distributed = seperate processes/pids</description>
>>  </property>
>>
>>  <property>
>>    <name>hbase.regionserver.hlog.replication</name>
>>    <value>1</value>
>>    <description>For HBase to offer good data durability, we roll logs if
>>    filesystem replication falls below a certain amount.  In
>> psuedo-distributed
>>    mode, you normally only have the local filesystem or 1 HDFS DataNode, so
>> you
>>    don't want to roll logs constantly.</description>
>>  </property>
>>
>>  <property>
>>    <name>hbase.tmp.dir</name>
>>    <value>/tmp/hbase-testing</value>
>>    <description>Temporary directory on the local filesystem.</description>
>>  </property>
>>
>> I also hase Hadoop conf directory in HBASE_CLASSPATH (hbase-env.sh).
>>
>> I just tried etc/hosts with "127.0.0.1               localhost.localdomain
>> localhost" line. I got the same error I was getting before. I switched it
>> back to "127.0.0.1       localhost" and it worked. In between those
>> changes,
>> I stopped hbase, hadoop and killed still running region server. I hope that
>> helps.
>>
>> N.Gesli
>>
>>
>>
>>
>> On Thu, Sep 16, 2010 at 7:04 AM, Michael Scott <mjscottuic@gmail.com>
>> wrote:
>>
>> > This sounds promising, I have one quick question about your steps:  where
>> > in
>> > the Hbase config *site*.xml did you make the change back to localhost?
>>  My
>> > hbase master is using the public IP address (97.86.88.18), and I don't
>> > think
>> > I've told it to.  I want to convince hbase to get rid of the line in the
>> > log
>> > file that says something like:
>> >
>> > 2010-09-16 09:59:21,727 INFO org.apache.hadoop.hbase.master.HMaster: My
>> > address is 97-86-88-18.static.aldl.mi.charter.com:60000
>> >
>> > (Note that my /etc/hosts has only the one line
>> > 127.0.0.1               localhost.localdomain localhost
>> > since I'm not running ipv6, but somehow hbase knows that the interface is
>> a
>> > comcast static address.  I can use /etc/hosts to change that to the
>> > registered domain name for 97-86-88-18, but this doesn't help.)
>> >
>> > To reply to Ryan's question, my ifconfig gives:
>> >
>> > eth0      Link encap:Ethernet  HWaddr 00:24:E8:01:DA:B8
>> >          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
>> >          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>> >          RX packets:319475 errors:0 dropped:0 overruns:0 frame:0
>> >          TX packets:290698 errors:0 dropped:0 overruns:0 carrier:0
>> >          collisions:0 txqueuelen:1000
>> >          RX bytes:108186958 (103.1 MiB)  TX bytes:187845633 (179.1 MiB)
>> >          Interrupt:28 Base address:0xa000
>> >
>> > lo        Link encap:Local Loopback
>> >          inet addr:127.0.0.1  Mask:255.0.0.0
>> >          UP LOOPBACK RUNNING  MTU:16436  Metric:1
>> >          RX packets:370795 errors:0 dropped:0 overruns:0 frame:0
>> >          TX packets:370795 errors:0 dropped:0 overruns:0 carrier:0
>> >          collisions:0 txqueuelen:0
>> >          RX bytes:108117402 (103.1 MiB)  TX bytes:108117402 (103.1 MiB)
>> >
>> > Thanks a bunch!
>> >
>> > Michael
>> >
>> > On Thu, Sep 16, 2010 at 1:12 AM, N.N. Gesli <nngesli@gmail.com> wrote:
>> >
>> > > Hi Michael,
>> > >
>> > > I was having a similar problem and following this thread for any
>> > > suggestions. I tried everything suggested and more.
>> > >
>> > > I was trying to run Hadoop/Hbase pseudo distributed version on my Mac.
>> I
>> > > initially started with Hadoop 21.0 and Hbase 0.89 versions. I had
>> exactly
>> > > the same error that you were getting. Then switched to Hadoop 20.2 and
>> > > Hbase
>> > > 20.6 - still HMaster was not starting. Then finally it worked. Below
>> are
>> > my
>> > > steps to success :)
>> > >
>> > > * stopped hbase
>> > > * stopped hadoop
>> > > * run jps; RegionServer was still running; killed it manually
>> > > * in tmp directory (where hadoop namenode and *.pid files are stored) I
>> > > removed everything related to hadoop and hbase, including the
>> > directories.
>> > > (I had no data in Hadoop, so I could do this)
>> > > * changed the ports back to default 600**
>> > > * changed back Hadoop and Hbase configurations to "localhost" in
>> > *site*.xml
>> > > and regionservers. (Only I will be using this - no remote connection)
>> > > * changed back my /etc/hosts to the original version. It looks like
>> this:
>> > > 127.0.0.1    localhost
>> > > ::1             localhost
>> > > fe80::1%lo0    localhost
>> > > * reformatted the Hadoop namenode
>> > > * started Hadoop
>> > > * started HBase and it worked :)
>> > >
>> > > Let me know if you want to know any specific configuration.
>> > >
>> > > N.Gesli
>> > >
>> > > On Wed, Sep 15, 2010 at 10:41 PM, Ryan Rawson <ryanobjc@gmail.com>
>> > wrote:
>> > >
>> > > > What is your ifconfig output looking like?
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Sep 15, 2010 at 10:07 PM, Michael Scott <
>> mjscottuic@gmail.com>
>> > > > wrote:
>> > > > > Thanks for the continued advice.  I am still confused by the
>> > different
>> > > > > behaviors of hadoop and hbase. As I said before, I can't get
hbase
>> to
>> > > > work
>> > > > > on any of the ports that hadoop works on, so I guess hadoop and
>> hbase
>> > > are
>> > > > > using different interfaces.  Why is this, and can't I ask hbase
to
>> > use
>> > > > the
>> > > > > interface that hadoop uses?  What interfaces are hadoop and
hbase
>> > > using?
>> > > > >
>> > > > > Also (and maybe this is the wrong forum for this question), how
can
>> I
>> > > get
>> > > > my
>> > > > > OS to allow me to open 60000 using the IP address?  I have
>> > temporarily
>> > > > > disabled selinux and iptables, as I thought that this would simply
>> > > allow
>> > > > all
>> > > > > port connections. Still, this works just fine:
>> > > > > bash-4.0$ nc -l  60000 > /tmp/nc.out
>> > > > >
>> > > > > but this does not:
>> > > > > bash-4.0$ nc -l 97.86.88.18 60000 > /tmp/nc.out
>> > > > > (returns "nc: Cannot assign requested address"; I get the same
>> error
>> > > for
>> > > > the
>> > > > > hostname instead of the IP address, and for 10.0.0.1, but 10.0.0.0
>> is
>> > > > > allowed)
>> > > > >
>> > > > > I am trying to get hbase running for a socorro server, which
will
>> > > running
>> > > > > locally.  I don't know if that matters.
>> > > > >
>> > > > > Thanks,
>> > > > > Michael
>> > > > >
>> > > > > On Wed, Sep 15, 2010 at 6:04 PM, Ryan Rawson <ryanobjc@gmail.com>
>> > > wrote:
>> > > > >
>> > > > >> Hey,
>> > > > >>
>> > > > >> If you bind to localhost you wont actually be reachable by
anyone!
>> > > > >>
>> > > > >> The question is why is your OS disallowing binds to a specific
>> > > > >> interface/port combo?
>> > > > >>
>> > > > >> HBase does not really run in a blended/multihomed environment...
>> > > > >> meaning if you have multiple interfaces, you have to choose
one
>> that
>> > > > >> we work over.  This is because we need to know a singular
>> canonical
>> > > > >> IP/name for any given server because we put that info up
inside
>> > > > >> ZooKeeper and META tables.  So it's not just an artificial
>> > constraint,
>> > > > >> but exists for cluster management needs.
>> > > > >>
>> > > > >> Having said that, we do work on multihomed machines, eg:
ec2, you
>> > > > >> might bind hbase to the internal interface taking advantage
of the
>> > > > >> unmetered/faster network. Also better for security as well.
>> > > > >>
>> > > > >> Let us know if you need more background on how we use the
network
>> > and
>> > > > why.
>> > > > >> -ryan
>> > > > >>
>> > > > >> On Wed, Sep 15, 2010 at 10:18 AM, Michael Scott <
>> > mjscottuic@gmail.com
>> > > >
>> > > > >> wrote:
>> > > > >> > Hi again,
>> > > > >> >
>> > > > >> > I think the hbase server master is not starting because
it is
>> > > > attempting
>> > > > >> to
>> > > > >> > open port 60000 on its public IP address, rather than
using
>> > > localhost.
>> > > >  I
>> > > > >> > cannot seem to figure out how to force it (well, configure
it)
>> to
>> > > > attempt
>> > > > >> to
>> > > > >> > bind to localhost:60000 instead.  As far as I can see,
 this is
>> > set
>> > > in
>> > > > >> the
>> > > > >> > file:
>> > > > >> >
>> > > > >> > org/apache/hadoop/hbase/master/HMaster.java
>> > > > >> >
>> > > > >> > I don't know much about java, so I'd prefer not to edit
the
>> source
>> > > if
>> > > > >> there
>> > > > >> > is an option, but I will if necessary.  Can someone
please point
>> > me
>> > > to
>> > > > >> the
>> > > > >> > way to change this setting?  Any help would be greatly
>> > appreciated.
>> > > > >> >
>> > > > >> > Thanks,
>> > > > >> > Michael
>> > > > >> >
>> > > > >> > On Wed, Sep 15, 2010 at 12:42 AM, Michael Scott <
>> > > mjscottuic@gmail.com
>> > > > >> >wrote:
>> > > > >> >
>> > > > >> >> Hi again,
>> > > > >> >>
>> > > > >> >> IPV6 was enabled.  I shut it off, rebooted to be
sure, verified
>> > it
>> > > > was
>> > > > >> >> still off, and encountered the same problem once
again.
>> > > > >> >>
>> > > > >> >> I also tried to open port 60000 by hand with a small
php file.
>>  I
>> > > can
>> > > > do
>> > > > >> >> this (as any user) for localhost.  I can NOT do
this (not even
>> as
>> > > > root)
>> > > > >> for
>> > > > >> >> the IP address which matches the fully qualified
domain name,
>> > which
>> > > > is
>> > > > >> what
>> > > > >> >> hbase is trying to use.  Is there some way for
me to configure
>> > > hbase
>> > > > to
>> > > > >> use
>> > > > >> >> localhost instead of the fully qualified domain
name for the
>> > > master?
>> > > >  I
>> > > > >> >> would have thought this was done by default, or
that there
>> would
>> > be
>> > > > an
>> > > > >> >> obvious line in some conf file, but I can't find
it.
>> > > > >> >>
>> > > > >> >> Thanks again,
>> > > > >> >>
>> > > > >> >> Michael
>> > > > >> >>
>> > > > >> >> On Tue, Sep 14, 2010 at 12:23 PM, Todd Lipcon <
>> todd@cloudera.com
>> > >
>> > > > >> wrote:
>> > > > >> >>
>> > > > >> >>> Hi Michael,
>> > > > >> >>>
>> > > > >> >>> It might be related to IPV6. Do you have IPV6
enabled on this
>> > > > machine?
>> > > > >> >>>
>> > > > >> >>> Check out this hadoop JIRA that might be related
for some
>> tips:
>> > > > >> >>> https://issues.apache.org/jira/browse/HADOOP-6056
>> > > > >> >>>
>> > > > >> >>> <https://issues.apache.org/jira/browse/HADOOP-6056>-Todd
>> > > > >> >>>
>> > > > >> >>> On Tue, Sep 14, 2010 at 10:17 AM, Michael Scott
<
>> > > > mjscottuic@gmail.com
>> > > > >> >>> >wrote:
>> > > > >> >>>
>> > > > >> >>> > That's correct.  I tried a number of different
ports to see
>> if
>> > > > there
>> > > > >> was
>> > > > >> >>> > something weird, and then I shut down the
hadoop server and
>> > > tried
>> > > > to
>> > > > >> >>> > connect
>> > > > >> >>> > to 50010 (which of course should have been
free at that
>> point)
>> > > but
>> > > > >> got
>> > > > >> >>> the
>> > > > >> >>> > same "cannot assign to requested address"
error.  If I start
>> > > > hadoop,
>> > > > >> >>> > netstat
>> > > > >> >>> > shows a process listening on 50010.
>> > > > >> >>> >
>> > > > >> >>> > I am going to try this on a different OS,
I am wondering if
>> > FC11
>> > > > is
>> > > > >> my
>> > > > >> >>> > problem.
>> > > > >> >>> >
>> > > > >> >>> > Michael
>> > > > >> >>> >
>> > > > >> >>> > On Tue, Sep 14, 2010 at 11:41 AM, Stack
<stack@duboce.net>
>> > > wrote:
>> > > > >> >>> >
>> > > > >> >>> > > On Tue, Sep 14, 2010 at 9:33 AM, Michael
Scott <
>> > > > >> mjscottuic@gmail.com>
>> > > > >> >>> > > wrote:
>> > > > >> >>> > > > I don't see why hadoop binds
>> > > > >> >>> > > > to a port but hbase does not
(I even tried starting
>> hbase
>> > > with
>> > > > >> >>> hadoop
>> > > > >> >>> > off
>> > > > >> >>> > > > and binding to 50010, which hadoop
uses).
>> > > > >> >>> > > >
>> > > > >> >>> > >
>> > > > >> >>> > > Using 50010 worked for hadoop but
not for hbase?  (Odd.
>>  We
>> > > > hadoop
>> > > > >> >>> > > their mechanism essentially).
>> > > > >> >>> > >
>> > > > >> >>> > > St.Ack
>> > > > >> >>> > >
>> > > > >> >>> >
>> > > > >> >>>
>> > > > >> >>>
>> > > > >> >>>
>> > > > >> >>> --
>> > > > >> >>> Todd Lipcon
>> > > > >> >>> Software Engineer, Cloudera
>> > > > >> >>>
>> > > > >> >>
>> > > > >> >>
>> > > > >> >
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>

Mime
View raw message