hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: HBase master WebUI is not responding
Date Fri, 04 Sep 2015 15:41:20 GMT
Can you do the following ?

run hbck command to see if any inconsistency is reported
check region server log on test-rs3 to see if hbase:meta is successfully
opened

Cheers

On Fri, Sep 4, 2015 at 8:20 AM, Akmal Abbasov <akmal.abbasov@icloud.com>
wrote:

> Hi Ted,
>
> No, I’ve restarted master node and now there is no exceptions, but still
> webui is not responding
> 2015-09-04 14:54:57,926 DEBUG [FifoRpcScheduler.handler1-thread-26]
> master.AssignmentManager: based on AM, current
> region=hbase:meta,,1.1588230740 is on server=test-rs3,60020,1441378286999
> server being checked: test-rs5,60020,1441261824016
> What else could be wrong?
>
> Thanks.
>
>
> On 03 Sep 2015, at 16:20, Ted Yu <yuzhihong@gmail.com> wrote:
>
> Did the exception lead to hbase:meta assignment failure ?
>
> Please check region server log on 10.10.8.53 <http://10.10.8.53:60000/>
>
> On Thu, Sep 3, 2015 at 7:15 AM, Akmal Abbasov <akmal.abbasov@icloud.com>
> wrote:
>
>> Hi Ted,
>> I’m getting a lot of
>> 2015-09-03 14:08:20,651 WARN  [1865504699@qtp-838397579-13]
>> client.HConnectionManager$HConnectionImplementation: Checking master
>> connection
>> com.google.protobuf.ServiceException: java.net.SocketTimeoutException:
>> Call to test-m/10.10.8.53:60000 failed because
>> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
>> channel to be ready for read. ch :
>> java.nio.channels.SocketChannel[connected local=/10.10.8.53:41243
>> remote=test-m/10.10.8.53:60000]
>>         at
>> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
>>         at
>> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
>>         at
>> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:42561)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(HConnectionManager.java:1508)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(HConnectionManager.java:2146)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(HConnectionManager.java:1859)
>>         at
>> org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3320)
>>         at
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:113)
>>         at
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
>>         at
>> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3347)
>>         at
>> org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptorsByNamespace(HBaseAdmin.java:2281)
>>         at
>> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__catalogTables(MasterStatusTmplImpl.java:465)
>>         at
>> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:274)
>>         at
>> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:386)
>>         at
>> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:376)
>>         at
>> org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:95)
>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>         at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>         at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>         at
>> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
>>         at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>         at
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1122)
>>         at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>         at
>> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>>         at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>         at java.io.FilterInputStream.read(FilterInputStream.java:133)
>>         at
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$PingInputStream.read(RpcClient.java:558)
>>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>>         at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>>         at java.io.DataInputStream.readInt(DataInputStream.java:387)
>>         at
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076)
>>         at
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
>>
>> Thanks
>>
>> > On 03 Sep 2015, at 15:50, Ted Yu <yuzhihong@gmail.com> wrote:
>> >
>> > Can you log into the master node and check master log ?
>> >
>> > Pastebin snippet of master log if you need more help.
>> >
>> > Cheers
>> >
>> >
>> >> On Sep 3, 2015, at 6:24 AM, Akmal Abbasov <akmal.abbasov@icloud.com>
>> wrote:
>> >>
>> >> Hi,
>> >> I’m having problems with accessing HBase master webui. when I try to
>> access it, it returns redirect page /master-status, and then it is freezes.
>> >> I’ve tried to wait for a long time, but still it didn’t work.
>> >> I am using hbase hbase-0.98.7-hadoop2, hadoop hadoop-2.5.1
>> >> Any suggestions?
>> >> Thanks.
>>
>>
>
>

Mime
View raw message