hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Keller <brya...@gmail.com>
Subject Re: HBaseClient.call() hang
Date Tue, 18 Dec 2012 22:25:30 GMT
Yes, that is what I would expect, but the client is stuck in the Object.wait() call and doesn't
get notified until the RPC timeout passes.

On Dec 18, 2012, at 12:27 PM, "Mesika, Asaf" <asaf.mesika@gmail.com> wrote:

> One thing I don't get:
> If the RS went down, then the RPC connection should have been reset, thus causing the
client to interrupt, right? It shouldn't be a matter of timeout at all.
> On Dec 17, 2012, at 7:18 PM, Bryan Keller wrote:
>> It seems there was a cascading effect. The regionservers were busy with scanning
a table, which resulted in some long GC's. The GC's were long enough to trigger the Zookeeper
timeout on at least one regionserver, which resulted in the regionserver shutting itself down.
This then caused the Object.wait() call which got stuck, and only exited after the very long
RPC timeout.
>> I have done a fair amount of work optimizing the GCs, and I increased the regionserver
timeouts, which should help with the regionserver shutdowns. But if a regionserver does shut
down for some other reason, this will still result in the Object.wait() hang.
>> One approach might be to have the regionservers send back a keep-alive, or progress,
message during a scan, and that message would reset the RPC timer. The regionserver could
do this every x number of rows processed server-side. Then the RPC timeout could be something
more sensible rather than being set to the longest time it takes to scan a region.
>> HBASE-5416 looks useful, it will make scans faster, but the problem I'm encountering
will still be present, but perhaps I could set the RPC timeout a bit lower. HBASE-6313 might
fix the hang, in which case I could live with the longer RPC timeout setting.
>> On Dec 14, 2012, at 9:49 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> Bryan:
>>> bq. My only thought would be to forego using filters
>>> Please keep using filters.
>>> I and Sergey are working on HBASE-5416: Improve performance of scans with
>>> some kind of filters
>>> This feature allows you to specify one column family as being essential.
>>> The other column family is only returned to client when essential column
>>> family matches. I wonder if this may be of help to you.
>>> You mentioned regionserver going down or being busy. I assume it was not
>>> often that regionserver(s) went down. For busy region server, did you try
>>> jstack'ing regionserver process ?
>>> Thanks
>>> On Fri, Dec 14, 2012 at 2:59 PM, Bryan Keller <bryanck@gmail.com> wrote:
>>>> I have encountered a problem with HBaseClient.call() hanging. This occurs
>>>> when one of my regionservers goes down while performing a table scan.
>>>> What exacerbates this problem is that the scan I am performing uses
>>>> filters, and the region size of the table is large (4gb). Because of this,
>>>> it can take several minutes for a row to be returned when calling
>>>> scanner.next(). Apparently there is no keep alive message being sent back
>>>> to the scanner while the region server is busy, so I had to increase the
>>>> hbase.rpc.timeout value to a large number (60 min), otherwise the next()
>>>> call will timeout waiting for the regionserver to send something back.
>>>> The result is that this HBaseClient.call() hang is made much worse,
>>>> because it won't time out for 60 minutes.
>>>> I have a couple of questions:
>>>> 1. Any thoughts on why the HBaseClient.call() is getting stuck? I noticed
>>>> that call.wait() is not using any timeout so it will wait indefinitely
>>>> until interrupted externally
>>>> 2. Is there a solution where I do not need to set hbase.rpc.timeout to a
>>>> very large number? My only thought would be to forego using filters and do
>>>> the filtering client side, which seems pretty inefficient
>>>> Here is a stack dump of the thread that was hung:
>>>> Thread 10609: (state = BLOCKED)
>>>> - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>>>> - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
>>>> -
>>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(org.apache.hadoop.io.Writable,
>>>> java.net.InetSocketAddress, java.lang.Class,
>>>> org.apache.hadoop.hbase.security.User, int) @bci=51, line=904 (Interpreted
>>>> frame)
>>>> -
>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(java.lang.Object,
>>>> java.lang.reflect.Method, java.lang.Object[]) @bci=52, line=150
>>>> (Interpreted frame)
>>>> - $Proxy12.next(long, int) @bci=26 (Interpreted frame)
>>>> - org.apache.hadoop.hbase.client.ScannerCallable.call() @bci=72, line=92
>>>> (Interpreted frame)
>>>> - org.apache.hadoop.hbase.client.ScannerCallable.call() @bci=1, line=42
>>>> (Interpreted frame)
>>>> -
>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(org.apache.hadoop.hbase.client.ServerCallable)
>>>> @bci=36, line=1325 (Interpreted frame)
>>>> - org.apache.hadoop.hbase.client.HTable$ClientScanner.next() @bci=117,
>>>> line=1299 (Compiled frame)
>>>> - org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue()
>>>> @bci=41, line=150 (Interpreted frame)
>>>> - org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue()
>>>> @bci=4, line=142 (Interpreted frame)
>>>> - org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue()
>>>> @bci=4, line=458 (Interpreted frame)
>>>> - org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue() @bci=4,
>>>> line=76 (Interpreted frame)
>>>> -
>>>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue()
>>>> @bci=4, line=85 (Interpreted frame)
>>>> -
>>>> org.apache.hadoop.mapreduce.Mapper.run(org.apache.hadoop.mapreduce.Mapper$Context)
>>>> @bci=6, line=139 (Interpreted frame)
>>>> -
>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(org.apache.hadoop.mapred.JobConf,
>>>> org.apache.hadoop.mapreduce.split.JobSplit$TaskSplitIndex,
>>>> org.apache.hadoop.mapred.TaskUmbilicalProtocol,
>>>> org.apache.hadoop.mapred.Task$TaskReporter) @bci=201, line=645 (Interpreted
>>>> frame)
>>>> - org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf,
>>>> org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=100, line=325
>>>> (Interpreted frame)
>>>> - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=268 (Interpreted
>>>> frame)
>>>> -
>>>> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
>>>> java.security.AccessControlContext) @bci=0 (Interpreted frame)
>>>> - javax.security.auth.Subject.doAs(javax.security.auth.Subject,
>>>> java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted
>>>> frame)
>>>> -
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
>>>> @bci=14, line=1332 (Interpreted frame)
>>>> - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=776,
>>>> line=262 (Interpreted frame)

View raw message