Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAKYcnAvmnF43s+_LtecmZZZb8fmM2ju0SpyFqiG4t24Hxy26Fw@mail.gmail.com>
References: <HK2PR02MB1058500CF07B0C03760B11DAA37D0@HK2PR02MB1058.apcprd02.prod.outlook.com>
 <CAKYcnAu2bcvJcYH26-gXtCNDcQHCcsUb6+Bp5i5yQeBZ1j=0GA@mail.gmail.com>
 <HK2PR02MB10581EF0B8FA720AB0DDED7FA37C0@HK2PR02MB1058.apcprd02.prod.outlook.com>
 <CAKYcnAvmnF43s+_LtecmZZZb8fmM2ju0SpyFqiG4t24Hxy26Fw@mail.gmail.com>
From: Yu Li <carp84@gmail.com>
Date: Thu, 19 Jan 2017 23:43:15 +0800
Message-ID: <CAM7-19LSurAbsVGYKN1Urzr5R7gVVJg-VhWQ9KFK9d-5C8aTjA@mail.gmail.com>
Subject: Re: hbase has problems with two hostname
To: Hbase-User <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a1140f66646a58b05467464e9
archived-at: Thu, 19 Jan 2017 15:43:29 -0000

--001a1140f66646a58b05467464e9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

If you have two network cards and one hostname binding on each, please make
sure to set the hostname you'd like to use in HBase in /etc/hosts on each
nodes, including HMaster and every RS.

Hope this helps.

Best Regards,
Yu

On 17 January 2017 at 23:47, Dima Spivak <dimaspivak@apache.org> wrote:

> Is there any other DNS server running that might be confusing reverse
> lookup? What happens if you run `host YOUR_RS_IP_ADDRESS`?
>
> And what kind of machines are you using in your deployment?
>
> Cheers,
>
> On Mon, Jan 16, 2017 at 11:34 PM C R <cuirong198610@hotmail.com> wrote:
>
> > Thanks,
> >
> >
> >
> >
> >
> > I deployed my HBase very simply, which has one Master and three
> > regionservers.
> >
> >
> >
> >
> >
> > [hbase@bjsh19-16-30 conf]$ more regionservers
> >
> > bjsh19-16-33.qbos.com
> >
> > bjsh19-16-34.qbos.com
> >
> > bjsh19-16-35.qbos.com
> >
> > [hbase@bjsh19-16-30 conf]$ more hbase-site.xml
> >
> >
> >
> > ...
> >
> >
> >
> > <configuration>
> >
> > <property>
> >
> >         <name>zookeeper.znode.parent</name>
> >
> >                 <value>/hbase117</value>
> >
> >                 </property>
> >
> > <property>
> >
> >   <name>hbase.rootdir</name>
> >
> >         <value>hdfs://bidc/hbase117</value>
> >
> >                   </property>
> >
> > <property>
> >
> >   <name>hbase.zookeeper.quorum</name>
> >
> >         <value>bjsh19-16-30.qbos.com,bjsh19-16-31.qbos.com,
> > bjsh19-16-32.qbos.com</value>
> >
> >                   </property>
> >
> > <property>
> >
> >   <name>hbase.cluster.distributed</name>
> >
> >         <value>true</value>
> >
> >                   </property>
> >
> > <property>
> >
> >   <name>hbase.zookeeper.property.clientPort</name>
> >
> >         <value>2181</value>
> >
> >                   </property>
> >
> > </configuration>
> >
> >
> >
> >
> >
> > The special place is the file /etc/hosts with one ip mapping to two
> > hostnames on all nodes,so it will have the message:
> >
> >
> >
> > ...
> >
> >
> >
> > the server that tried to transition was wjsa-tsl05,16020,1484623636195
> not
> > the expected bjsh19-16-34.qbos.com,16020,1484623636195
> >
> >
> >
> > ...
> >
> >
> >
> >
> >
> > ________________________________
> >
> > =E5=8F=91=E4=BB=B6=E4=BA=BA: Dima Spivak <dimaspivak@apache.org>
> >
> > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2017=E5=B9=B41=E6=9C=8817=E6=97=
=A5 4:50
> >
> > =E6=94=B6=E4=BB=B6=E4=BA=BA: user@hbase.apache.org
> >
> > =E4=B8=BB=E9=A2=98: Re: hbase has problems with two hostname
> >
> >
> >
> > Hi C R,
> >
> >
> >
> > Like many Hadoop-like services, HBase is pretty temperamental about
> >
> > requiring forward and reverse DNS to work properly. FWIW, the
> configuration
> >
> > file where you can populate RegionServers doesn't tend to matter as lon=
g
> as
> >
> > the hbase-site.xml file is populated correctly (it's just used to start
> >
> > daemons from one place).
> >
> >
> >
> > If you pass along more details about how exactly you're deploying HBase=
,
> we
> >
> > might be able to give more advice.
> >
> >
> >
> > On Mon, Jan 16, 2017 at 8:00 PM C R <cuirong198610@hotmail.com> wrote:
> >
> >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > more /etc/hosts
> >
> > >
> >
> > >
> >
> > > ...
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > 10.19.16.31  bjsh19-16-31.qbos.com    wjsa-tsl02
> >
> > >
> >
> > >
> >
> > > ...
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > There will have six regionservers listed in web console, but
> >
> > >
> >
> > > only three in the configuration file,  metadata tables also are not
> > online
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > Hmaster will be dead after a while.
> >
> > >
> >
> > >
> >
> > > what should I do?
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > snapshot:
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:45:24,394 INFO
> >
> > >  [MASTER_SERVER_OPERATIONS-bjsh19-16-30:16000-0]
> > master.AssignmentManager:
> >
> > > Assigning
> > hbase:namespace,,1484623643279.30fab746cb3b6ceadcbda421459204b9.
> >
> > > to bjsh19-16-34.qbos.com,16020,1484623636195
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:45:24,395 INFO  [bjsh19-16-30:16000.activeMasterManager=
]
> >
> > > master.AssignmentManager: Joined the cluster in 23ms, failover=3Dtrue
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:50:24,314 FATAL [bjsh19-16-30:16000.activeMasterManager=
]
> >
> > > master.HMaster: Failed to become active master
> >
> > >
> >
> > >
> >
> > > java.io.IOException: Timedout 300000ms waiting for namespace table to
> be
> >
> > > assigned
> >
> > >
> >
> > >
> >
> > >         at
> >
> > >
> > org.apache.hadoop.hbase.master.TableNamespaceManager.
> start(TableNamespaceManager.java:104)
> >
> > >
> >
> > >
> >
> > >         at
> >
> > > org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:986=
)
> >
> > >
> >
> > >
> >
> > >         at
> >
> > >
> > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitializati
> on(HMaster.java:780)
> >
> > >
> >
> > >
> >
> > >         at
> >
> > > org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:183)
> >
> > >
> >
> > >
> >
> > >         at
> > org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1652)
> >
> > >
> >
> > >
> >
> > >         at java.lang.Thread.run(Thread.java:745)
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:50:24,315 FATAL [bjsh19-16-30:16000.activeMasterManager=
]
> >
> > > master.HMaster: Master server abort: loaded coprocessors are: []
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:50:24,316 FATAL [bjsh19-16-30:16000.activeMasterManager=
]
> >
> > > master.HMaster: Unhandled exception. Starting shutdown.
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > ...
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:27:17,926 INFO  [regionserver/
> >
> > > bjsh19-16-34.qbos.com/10.19.16.34:16020] regionserver.HRegionServer:
> >
> > > Serving as wjsa-tsl05,16020,1
> >
> > >
> >
> > >
> >
> > > 484623636195, RpcServer on bjsh19-16-34.qbos.com/10.19.16.34:16020,
> >
> > > sessionid=3D0x154563e43e30179
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:27:17,934 INFO  [regionserver/
> >
> > > bjsh19-16-34.qbos.com/10.19.16.34:16020]
> > quotas.RegionServerQuotaManager:
> >
> > > Quota support disabled
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:27:23,966 INFO
> >
> > >  [PriorityRpcServer.handler=3D14,queue=3D0,port=3D16020]
> >
> > > regionserver.RSRpcServices: Open hbase:namespace,,148462364327
> >
> > >
> >
> > >
> >
> > > 9.30fab746cb3b6ceadcbda421459204b9.
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:27:24,008 WARN  [RS_OPEN_REGION-bjsh19-16-34:16020-0]
> >
> > > zookeeper.ZKAssign: regionserver:16020-0x154563e43e30179,
> > quorum=3Dbjsh19-16
> >
> > >
> >
> > >
> >
> > > -30:2181,bjsh19-16-31:2181,bjsh19-16-32:2181, baseZNode=3D/hbase115ne=
w
> >
> > > Attempt to transition the unassigned node for
> 30fab746cb3b6ceadcbda421459
> >
> > >
> >
> > >
> >
> > > 204b9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the
> server
> >
> > > that tried to transition was wjsa-tsl05,16020,1484623636195 not the
> >
> > >
> >
> > >
> >
> > > expected bjsh19-16-34.qbos.com,16020,1484623636195
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:27:24,008 WARN  [RS_OPEN_REGION-bjsh19-16-34:16020-0]
> >
> > > coordination.ZkOpenRegionCoordination: Failed transition from OFFLINE
> to
> > O
> >
> > >
> >
> > >
> >
> > > PENING for region=3D30fab746cb3b6ceadcbda421459204b9
> >
> > >
> >
> > >
> >
> > > 2017-01-17 11:27:24,008 WARN  [RS_OPEN_REGION-bjsh19-16-34:16020-0]
> >
> > > handler.OpenRegionHandler: Region was hijacked? Opening cancelled for
> > enco
> >
> > >
> >
> > >
> >
> > > dedName=3D30fab746cb3b6ceadcbda421459204b9
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
>

--001a1140f66646a58b05467464e9--