hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: RS not processing any requests
Date Thu, 06 Sep 2012 15:04:40 GMT
Great this is good news!

~Jeff

On 9/5/2012 6:49 PM, Ted Yu wrote:
> Backport has been done in HBASE-6724
>
> Cheers
>
> On Wed, Sep 5, 2012 at 5:09 PM, Himanshu Vashishtha <hvashish@cs.ualberta.ca 
> <mailto:hvashish@cs.ualberta.ca>> wrote:
>
>     It usually happens in a long running setup (at least for me). Can you
>     throttle your load?
>
>     Replication is evolving; I'd say update if you can (or backport the jiras?).
>
>     Himanshu
>
>
>     On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <jeffw@qualtrics.com
>     <mailto:jeffw@qualtrics.com>> wrote:
>     > hmm.  So if we are on 0.92 what suggestion would you have to prevent the
>     > problem?
>     >
>     > ~Jeff
>     >
>     >
>     > On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote:
>     >>
>     >> Number of PRI handlers are governed by
>     >> "hbase.regionserver.metahandler.count"; default is 10.
>     >>
>     >> Increasing their number will not solve it, but will delay its
>     >> occurring (i don't know about your load etc).
>     >>
>     >> Another related jira is HBase-6550.
>     >>
>     >> Some more context for your use case:
>     >>
>     >>
>     http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements
>     >>
>     >>
>     >> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <jeffw@qualtrics.com
>     <mailto:jeffw@qualtrics.com>> wrote:
>     >>>
>     >>> It looks like that is problem we are having.  We are on 0.92 so we don't
>     >>> get
>     >>> the patch.  But one solution seems to be increasing the privileged
>     >>> handlers.
>     >>> How do we increase the number of privilege handlers?
>     >>>
>     >>>
>     >>> ~Jeff
>     >>>
>     >>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote:
>     >>>>
>     >>>> Your RS priority handlers are blocked on meta lookup, so it becomes
>     >>>> unresponsive. Looks like you hitting
>     >>>> https://issues.apache.org/jira/browse/HBASE-6165
>     >>>> You running HBase replication? just confirming.
>     >>>>
>     >>>> Himanshu
>     >>>>
>     >>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <stack@duboce.net <mailto:stack@duboce.net>>
wrote:
>     >>>>>
>     >>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook
>     >>>>> <nathanielc@qualtrics.com <mailto:nathanielc@qualtrics.com>>
>     >>>>> wrote:
>     >>>>>>
>     >>>>>> We ran a jstack on the both the RS process and the hbase
shell process
>     >>>>>> trying to do the scan.
>     >>>>>>
>     >>>>>> Jstack log for RS:
>     >>>>>> http://pastebin.com/9Y9t5ERE
>     >>>>>>
>     >>>>> What JVM (I don't know what (20.10-b01 mixed mode) is).
>     >>>>>
>     >>>>> I see a bunch of this:
>     >>>>>
>     >>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10
>     >>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry
>     >>>>> [0x000000004ab0f000]
>     >>>>>      java.lang.Thread.State: BLOCKED (on object monitor)
>     >>>>>           at .....
>     >>>>>
>     >>>>> But when I go to look for other instances of the object monitor,
I
>     >>>>> don't find any.  I see this for each instance of BLOCKED (Or
at least,
>     >>>>> the two or three I checked).
>     >>>>>
>     >>>>> Whats your OS?
>     >>>>>
>     >>>>> St.Ack
>     >>>
>     >>>
>     >>> --
>     >>> Jeff Whiting
>     >>> Qualtrics Senior Software Engineer
>     >>> jeffw@qualtrics.com <mailto:jeffw@qualtrics.com>
>     >>>
>     >>>
>     >>>
>     >
>     > --
>     > Jeff Whiting
>     > Qualtrics Senior Software Engineer
>     > jeffw@qualtrics.com <mailto:jeffw@qualtrics.com>
>     >
>     >
>     >
>
>

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message