hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Date Fri, 10 Apr 2015 21:32:32 GMT
Sridhar,

The logs from solix4 are very interesting:

solix 4:
2015-04-08 22:48:21,308 INFO  [regionserver60020-SendThread(
solix1.abcsupply.com:2181)] zookeeper.ClientCnxn: Session establishment
complete on server solix1.abcsupply.
com/10.1.252.86:2181, sessionid = 0x14c9c32c6fd0004, negotiated timeout =
40000
...
2015-04-08 22:48:21,530 INFO  [regionserver60020]
regionserver.HRegionServer: reportForDuty to
master=solix1.abcsupply.com,60000,1428551050776
with port=60020, startcode=1428551291142
2015-04-08 22:48:21,777 WARN  [regionserver60020]
regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused

Which mean there is a live master since we can get a znode from ZooKeeper
(coincidentally zk connected to solix1 ZK server) and that znode  is
telling us that the master in solix1 but we cannot connect to the master.

Have you verified if iptables or any other firewall rule is blocking the
connectivity between the RSs and the master?

thanks,
esteban.









--
Cloudera, Inc.


On Fri, Apr 10, 2015 at 12:29 AM, sridhararao mutluri <drmsr6@hotmail.com>
wrote:

> Hi Team,
>
> I uploaded al lregion logs in below address.The Hregions are up on three
> nodes and i saw erros "connection refused"
>
> https://gist.github.com/drmsr6/7e40958db1ea364e3541
>
> Thanks,
> Sridhar
>
>
> > From: busbey@cloudera.com
> > Date: Fri, 10 Apr 2015 02:02:07 -0500
> > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of
> Hbase
> > To: drmsr6@hotmail.com
> > CC: user@hbase.apache.org; esteban@cloudera.com
> >
> > the master log shows that no region servers ever come online. can you
> > upload the region server logs?
> >
> > On Fri, Apr 10, 2015 at 1:15 AM, sridhararao mutluri <drmsr6@hotmail.com
> >
> > wrote:
> >
> > > Team,
> > >
> > > Please find master log url and suggest any solution:
> > >
> > > https://gist.github.com/drmsr6/a25de9e60bb6229f796e
> > >
> > > Thanks,
> > > Sridhar
> > >
> > >
> > > ------------------------------
> > > Date: Thu, 9 Apr 2015 07:46:40 -0500
> > > Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of
> Hbase
> > > From: busbey@cloudera.com
> > > To: drmsr6@hotmail.com
> > > CC: user@hbase.apache.org; esteban@cloudera.com
> > >
> > >
> > > Please post a link to updated master logs.
> > >
> > > --
> > > Sean
> > > On Apr 9, 2015 5:50 AM, "sridhararao mutluri" <drmsr6@hotmail.com>
> wrote:
> > >
> > > Hi Esteban,
> > >
> > > We increased maximum files and with reboot ulimit -n shows across all
> > > clusters
> > >
> > > [hadoop@solix1 bin]$ ulimit -n
> > > 65535
> > >
> > > But unluckily we are getting same error now also:
> > >
> > > hbase(main):001:0> create 'buses', 'vi'
> > >
> > >
> > > *ERROR: java.io.IOException: Table Namespace Manager not ready yet, try
> > > again later*
> > > at
> > >
> org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179)
> > > at
> > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735)
> > > at
> > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774)
> > > at
> > >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> > > at
> > >
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > at
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Do we downgrade JDK from 1.8 to 1.7 and validate.Please suggest.
> > >
> > > Thanks,
> > > Sridhar
> > >
> > >
> > >
> > > ------------------------------
> > > From: drmsr6@hotmail.com
> > > To: esteban@cloudera.com
> > > Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of
> Hbase
> > > Date: Thu, 9 Apr 2015 06:44:10 +0530
> > >
> > > Hi Esteban,
> > >
> > > Thanks much for your speedy suggestion.I will implement and will come
> back
> > > to you.
> > >
> > > Regards,
> > >
> > > Dr.Sridhar
> > >
> > >
> > > ------------------------------
> > > From: esteban@cloudera.com
> > > Date: Wed, 8 Apr 2015 18:06:21 -0700
> > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of
> Hbase
> > > To: drmsr6@hotmail.com
> > >
> > >
> > > what i meant to say is that you should try to change the ulimit to 64k
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Wed, Apr 8, 2015 at 6:06 PM, Esteban Gutierrez <
> esteban@cloudera.com>
> > > wrote:
> > >
> > >
> > >
> > > cool... so the HBase master started, but you might want to bump your
> > > ulimits -n by 60 times, 1024 of max open files is too low of hbase.
> > >
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Wed, Apr 8, 2015 at 5:55 PM, sridhararao mutluri <
> drmsr6@hotmail.com>
> > > wrote:
> > >
> > > Hi Esteban,
> > >
> > > Please find the link:
> > >
> > > https://gist.github.com/drmsr6/bcf439f09cac95b59eca
> > >
> > >
> > > I am waiting for final inputs.:)
> > >
> > > Thanks,
> > > Dr.Sridhar,
> > > 919573453136
> > >
> > >
> > > ------------------------------
> > > From: esteban@cloudera.com
> > > Date: Wed, 8 Apr 2015 11:54:04 -0700
> > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of
> Hbase
> > > To: drmsr6@hotmail.com
> > >
> > >
> > >
> > > Sridhar,
> > >
> > > what is the URL to the log file?
> > >
> > > thanks
> > >
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Tue, Apr 7, 2015 at 6:35 PM, sridhararao mutluri <
> drmsr6@hotmail.com>
> > > wrote:
> > >
> > > Hi Esteban,
> > >
> > > I pasted logs in github.com:
> > >
> > >
> > >
> > > Thanks,
> > > Sridhar
> > >
> > >
> > >
> > > > From: esteban@cloudera.com
> > > > Date: Tue, 7 Apr 2015 08:44:55 -0700
> > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of
> > > Hbase
> > > > To: user@hbase.apache.org
> > >
> > > >
> > > > Sridhar,
> > > >
> > > > What do you see in the HBase Master logs? The exception you are
> getting
> > > > from the HBase Master is just a side effect and not the real cause?
> Is it
> > > > possible for you to upload the HBase Master logs to a site like
> > > pastebin.com
> > > > or gist.github.com so we can look at?
> > > >
> > > > cheers,
> > > > esteban.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cloudera, Inc.
> > > >
> > > >
> > > > On Tue, Apr 7, 2015 at 8:03 AM, sridhararao mutluri <
> drmsr6@hotmail.com>
> > > > wrote:
> > > >
> > > > > Team,
> > > > > The port the HBase Master should bind to 60000
> > > > > Thanks,Sridhar> From: serega.sheypak@gmail.com
> > > > > > Date: Tue, 7 Apr 2015 16:40:54 +0200
> > > > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6
> HA:Issues of
> > > > > Hbase
> > > > > > To: user@hbase.apache.org
> > > > > > CC: busbey@cloudera.com
> > > > > >
> > > > > > <property>
> > > > > > <name>hbase.master</name>
> > > > > > <value>hdfs://cluster1:60000</value>
> > > > > > </property>
> > > > > > what is it?
> > > > > >
> > > > > > 2015-04-07 16:34 GMT+02:00 sridhararao mutluri <
> drmsr6@hotmail.com>:
> > > > > >
> > > > > > > Hi,
> > > > > > > This is my hbase-site.xml:
> > > > > > > <configuration> <property> <name>hbase.master</name>
> > > > > > > <value>hdfs://cluster1:60000</value> </property>
> > > > > > > <property> <name>hbase.rootdir</name>
> > > > > > > <value>hdfs://mycluster/hbase</value> </property>
> > > > > <property>
> > > > > > > <name>hbase.cluster.distributed</name>
> > > > > > > <value>true</value> </property> <property>
> > > > > > > <name>hbase.zookeeper.property.clientPort</name>
> > > > > > > <value>2181</value> </property> <property>
> > > > > > > <name>hbase.zookeeper.quorum</name>
> > > > > > > <value>cluster1,cluster2,cluster3</value> </property>
> > > > > > > <property> <name>hbase.zookeeper.property.dataDir</name>
> > > > > > > <value>/hadoop/hdfs/zookeeper/data/zk1</value>
> > > > > > > </property></configuration>
> > > > > > > Thanks,Sridhar
> > > > > > > > Date: Tue, 7 Apr 2015 07:09:44 -0700
> > > > > > > > Subject: Re: Hbase 0.98 Distributed Mode with hadoop
2.6
> > > HA:Issues of
> > > > > > > Hbase
> > > > > > > > From: yuzhihong@gmail.com
> > > > > > > > To: user@hbase.apache.org
> > > > > > > > CC: busbey@cloudera.com
> > > > > > > >
> > > > > > > > bq. <property> <name>hbase.rootdir</name>
> > > > > > > > <value>hdfs://mycluster/hbase</value>
</property>
> > > > > <property>
> > > > > > > >
> > > > > > > > Looks like there is a property missing at the end
of the
> line.
> > > > > > > >
> > > > > > > > You showed snippet from shell output. Have you checked
master
> > > log ?
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri
<
> > > > > drmsr6@hotmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Team,
> > > > > > > > > I am trying to use hbase 0.98 distributed mode
with zk
> 3.4.6 &
> > > > > hadoop
> > > > > > > ha
> > > > > > > > > 2.6.(JDK 1.8)
> > > > > > > > > I am having following issue and little help in
google pages
> > > also
> > > > > > > > > I tried to start zk first after clearing zk data
dir and
> tried
> > > to
> > > > > start
> > > > > > > > > master first and rs later and no luck
> > > > > > > > > I used mycluster/hbase in hbase-site.xml and
no luck to
> > > me.tried
> > > > > to put
> > > > > > > > > hdfs-site.xml/core-site.xml in $Hbase_home/conf
also.
> > > > > > > > > I noticed all hadoop.*jars in $HBASE_HOME/lib
are 2.2 of
> hadoop
> > > > > where
> > > > > > > as
> > > > > > > > > we are using 2.6 and tried to copy those hadoop
jars ..but
> no
> > > luck.
> > > > > > > > > A New new error is coming:
> > > > > > > > >
> > > > > > > > > hbase(main):002:0> create 'cars', 'vi'
> > > > > > > > > ERROR: java.io.IOException: Table Namespace Manager
not
> ready
> > > yet,
> > > > > try
> > > > > > > > > again later at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179)
> > > > > > > > > at
> > > > > > > > >
> > > > >
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735)
> > > > > > > > > at
> > > > > > >
> > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> > > > > > > > > at
> > > > > > > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
> > > > > > > > > at
> > > > > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
> > > > > > > > > at
> > > > > > > > >
> > > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > > > > > > > > at java.lang.Thread.run(Thread.java:745)ZK are
running on 3
> > > > > > > servers.I
> > > > > > > > > tired to stop hbase and then stopped zk in 3
clusters and
> > > cleared
> > > > > zk
> > > > > > > data
> > > > > > > > > and started fresh.
> > > > > > > > > My bash_profile on classpath:export
> > > > > > > > >
> CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export
> > > > > > > > > CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:.
> > > > > > > > > my hbase_site.xml
> > > > > > > > > <property> <name>hbase.rootdir</name>
> > > > > > > > > <value>hdfs://mycluster/hbase</value>
</property>
> > > > > > > <property>
> > > > > > > > > and hadoop core-site is same.
> > > > > > > > > Any incompatibility between JDK1.8 with hbase
or hadoop
> 2.6?
> > > > > > > > > Please suggest any solution.
> > > > > > > > > Thanks,Sridhar
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > Sean
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message