hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: hbase thrift connection drops / refuses after warning
Date Fri, 18 Sep 2015 23:35:02 GMT
Can it be https://issues.apache.org/jira/browse/HBASE-14196?

Do you idle your Thrift server for more than 10 min sometimes?

-Vlad

On Fri, Sep 18, 2015 at 4:23 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:

> The thrift deamon just stalls? You don't see this on 0.98.6 -- do you ever
> see retries exhausted exceptions on 0.98?
>
> On Wed, Sep 2, 2015 at 5:17 PM, Abraham Tom <work2much@gmail.com> wrote:
>
> > Trying to see if anybody knows why this is happening and how to fix
> > we are on hbase 1.1.1
> >
> > We use javascript to connect to hbase via thrift
> > every once in a while we get the following warning
> >
> > WARN  [thrift-worker-3] thrift.ThriftServerRunner$HBaseHandler: Can't get
> > the location
> > This warning disconnects thrift from responding to any more requests
> >
> > full output from thrift log
> > thrift.ThriftServerRunner$HBaseHandler: Can't get the location
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
> > location
> >         at
> >
> >
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:309)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> >         at
> >
> org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.ReversedClientScanner.<init>(ReversedClientScanner.java:56)
> >         at
> > org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:800)
> >         at
> >
> >
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1451)
> >         at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
> >         at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:497)
> >         at
> >
> >
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)
> >         at com.sun.proxy.$Proxy11.scannerOpenWithScan(Unknown Source)
> >         at
> >
> >
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609)
> >         at
> >
> >
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593)
> >         at
> > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> >         at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> >         at
> >
> >
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> >         at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> >         at
> > org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >         at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.io.IOException: hconnection-0x538516ab closed
> >         at
> >
> >
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146)
> >         at
> >
> >
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
> >         ... 24 more
> >
> > jps shows that thrift is still operational, but the only way to clear
> this
> > up is to restart thrift.
> > Thrift is invoked as hbase-daemon.sh start thrift -hsha -f
> > hbase master and region look fine and hbase shell can still be invoked
> and
> > queried.
> >
> >
> > That invocation works on 0.98.6 instance and our javascript npm code base
> > has not changed
> >
> >
> >
> >
> > --
> > Abraham Tom
> > Email:   work2much@gmail.com
> > Phone:  415-515-3621
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message