hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: RS not processing any requests
Date Thu, 06 Sep 2012 15:04:22 GMT
We can throttle our replication load.  How do you do that? ;-)

~Jeff

On 9/5/2012 6:09 PM, Himanshu Vashishtha wrote:
> It usually happens in a long running setup (at least for me). Can you
> throttle your load?
>
> Replication is evolving; I'd say update if you can (or backport the jiras?).
>
> Himanshu
>
>
> On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <jeffw@qualtrics.com> wrote:
>> hmm.  So if we are on 0.92 what suggestion would you have to prevent the
>> problem?
>>
>> ~Jeff
>>
>>
>> On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote:
>>> Number of PRI handlers are governed by
>>> "hbase.regionserver.metahandler.count"; default is 10.
>>>
>>> Increasing their number will not solve it, but will delay its
>>> occurring (i don't know about your load etc).
>>>
>>> Another related jira is HBase-6550.
>>>
>>> Some more context for your use case:
>>>
>>> http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements
>>>
>>>
>>> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <jeffw@qualtrics.com> wrote:
>>>> It looks like that is problem we are having.  We are on 0.92 so we don't
>>>> get
>>>> the patch.  But one solution seems to be increasing the privileged
>>>> handlers.
>>>> How do we increase the number of privilege handlers?
>>>>
>>>>
>>>> ~Jeff
>>>>
>>>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote:
>>>>> Your RS priority handlers are blocked on meta lookup, so it becomes
>>>>> unresponsive. Looks like you hitting
>>>>> https://issues.apache.org/jira/browse/HBASE-6165
>>>>> You running HBase replication? just confirming.
>>>>>
>>>>> Himanshu
>>>>>
>>>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <stack@duboce.net> wrote:
>>>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook
>>>>>> <nathanielc@qualtrics.com>
>>>>>> wrote:
>>>>>>> We ran a jstack on the both the RS process and the hbase shell
process
>>>>>>> trying to do the scan.
>>>>>>>
>>>>>>> Jstack log for RS:
>>>>>>> http://pastebin.com/9Y9t5ERE
>>>>>>>
>>>>>> What JVM (I don't know what (20.10-b01 mixed mode) is).
>>>>>>
>>>>>> I see a bunch of this:
>>>>>>
>>>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>>>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry
>>>>>> [0x000000004ab0f000]
>>>>>>       java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>            at .....
>>>>>>
>>>>>> But when I go to look for other instances of the object monitor,
I
>>>>>> don't find any.  I see this for each instance of BLOCKED (Or at least,
>>>>>> the two or three I checked).
>>>>>>
>>>>>> Whats your OS?
>>>>>>
>>>>>> St.Ack
>>>>
>>>> --
>>>> Jeff Whiting
>>>> Qualtrics Senior Software Engineer
>>>> jeffw@qualtrics.com
>>>>
>>>>
>>>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> jeffw@qualtrics.com
>>
>>
>>

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com




Mime
View raw message