hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: client call scan on some region hang
Date Tue, 06 Jan 2015 12:59:40 GMT
Can you pastebin region server log ?

When the scan is being performed, can you get jstack and pastebin it ?

0.94.15 was an old release, any chance of upgrade ?

Thanks



> On Jan 6, 2015, at 2:34 AM, Bi,hongyu—mike <boylook@gmail.com> wrote:
> 
> sorry , forgot to attach the version: 0.94.15;
> 
> and i call compact (as well as many times flush region) from hbase shell
> didn't take effect, no compaction happened;
> 
> 2015-01-06 18:26 GMT+08:00 Bi,hongyu—mike <boylook@gmail.com>:
> 
>> scan debug log:
>> 15/01/06 18:20:56 DEBUG client.ClientScanner: Creating scanner over T
>> starting at key 'Rowx'
>> 15/01/06 18:20:56 DEBUG client.ClientScanner: Advancing internal scanner
>> to startKey at 'Rowx'
>> 15/01/06 18:20:56 DEBUG client.MetaScanner: Scanning .META. starting at
>> row=XXXX for max=10 rows using
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@427b7b5d
>> 15/01/06 18:20:56 DEBUG
>> client.HConnectionManager$HConnectionImplementation: Cached location for
>> <THAT REGION> is RS_IP:60020
>> ......
>> 15/01/06 18:21:16 DEBUG zookeeper.ClientCnxn: Got ping response for
>> sessionid: 0x3499df682b076cf after 0ms
>> 15/01/06 18:21:36 DEBUG zookeeper.ClientCnxn: Got ping response for
>> sessionid: 0x3499df682b076cf after 0ms
>> 15/01/06 18:21:56 DEBUG zookeeper.ClientCnxn: Got ping response for
>> sessionid: 0x3499df682b076cf after 0ms
>> 15/01/06 18:21:56 DEBUG zookeeper.ClientCnxn: Reading reply
>> sessionid:0x3499df682b076cf, packet:: clientPath:null serverPath:null
>> finished:false header:: 9,4  replyHeader:: 9,21519728740,-101  request::
>> '/hbase/table/T,F  response::
>> 15/01/06 18:21:56 DEBUG
>> client.HConnectionManager$HConnectionImplementation: Removed <THAT REGION>
>> for tableName=T from cache because of Rowx
>> 15/01/06 18:21:56 DEBUG
>> client.HConnectionManager$HConnectionImplementation: Cached location for
>> <THAT REGION> is RS_IP:60020
>> 15/01/06 18:21:56 DEBUG client.ClientScanner: Advancing internal scanner
>> to startKey at 'Rowx'
>> 
>> 2015-01-06 18:09 GMT+08:00 Bi,hongyu—mike <boylook@gmail.com>:
>> 
>>> write traffic is ok:
>>> 2015-01-06 17:46:01,127 WARN org.apache.hadoop.hbase.ipc.SecureServer:
>>> (responseTooSlow): {"processingtimems":68,"call":"multi(Region=Rx of 149
>>> actions and first row key= Rowx), rpc version=1, client version=29,
>>> methodsFingerPrint=-1105746420","client":"IP:port}
>>> 
>>> scan on that region slow:
>>> 015-01-06 16:23:25,087 ERROR
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting on
>>> region Rx, call next(8002464006782223710, 1, 0), rpc version=1, client
>>> version=29, methodsFingerPrint=-1771721648 from 10.201.202.31:31285
>>> after 87821 ms, since caller disconnected
>>>        at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:438);
>>> 
>>> hbase hfile -r 'Rx' -p     can produce the result
>>> 
>>> 2015-01-06 18:03 GMT+08:00 Bi,hongyu—mike <boylook@gmail.com>:
>>> 
>>>> Hi  all,
>>>> 
>>>> There's one region which can take write request but scan;
>>>> If I scan on that region I'll get scanner lease timeout(60s by
>>>> default),while I can scan other region of  the same table and get the
>>>> result less than 10ms(our slow rpc threadhold is 10ms);
>>>> 
>>>> hbck report OK, and I use "hbase hfile" tool to check that region's
>>>> storefile and the region ,which all extract the result;
>>>> 
>>>> so I don't have any idea on it...
>>>> any help will be appreciate, many thanks!
>> 

Mime
View raw message