hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <msegel_had...@hotmail.com>
Subject Re: Coprocessor Increments
Date Fri, 11 Oct 2013 16:10:25 GMT
ok… 

A couple of things… 

First, it looks like when tuning HBase, you're not accounting for the additional overhead
caused by unknown coprocessors.  I guess even if you take a SWAG, you could end up still running
out of resources… 

Second… I'm a little confused. 
In the OP's problem… is he trying to increment a counter in the same table, or is his counter
in a different table? 
This would then become a discussion of schema design. 

Next, I would have to agree with you about the need to refactor HBase code. Reviewing and
refactoring is always a good thing if you have time. ;-) 

With respect to the OP's design… does the deadlock occur because he's trying to update a
column in a different row within the same table? 

The reason I ask is that if I were to implement an RO that updates a row in a second table,
upon instantiation, i would create an HTable object referencing the second table and then
use that to handle the data. Am I wrong to assume that in doing so, that I wouldn't be using
the internal RPC threads and that the update would look like an external client? 

Since my mind is pretty much mush by now… (long week, no sleep),  is it better to accept
the higher overhead by making the RO a client itself, or should I risk the potential of deadlock?


So what am I missing? 

Thx



On Oct 10, 2013, at 6:52 PM, Vladimir Rodionov <vrodionov@carrieriq.com> wrote:

> Michael, 
> 
> RS has 3 different pools of threads (Readers (accept RPC), Handlers (process requests),
third pool (do not remember name) of threads send request results back  to callers )
> All of them communicate to through internal Queues. Cumbersome and needs to be  refactored,
of course.
> 
> All these pools are configurable (max threads). Readers and writers are fine - they will
never get into any deadlock situation.
> Handlers are different. In my example, I described the situation when ALL handlers in
RS1 are waiting on RPC calls to RS2 and
> ALL handlers in RS2 are waiting on RPC calls to RS2. For the sake of simplicity:
> 
> RS1 and RS2 - two region servers
> handlers count = 1 - just one thread in a pool.
> 
> 1. RS2 coprocessor sends RPC to RS1. From single handler thread. Total available handlers
in RS2 = 0.  
> 2. RS1 receives request from RS2 CP2 (coprocessor). 
> 3. RS1 handler1 receives request. Total available handlers in RS1 = 0;
> 4. RS1 coprocessor makes RPC call back to RS2
> 5. RS2 Reader thread places request into Handler pool queue, but there is no handlers
available (see 1). Deadlock.
> and the request MUST timeout (fail) eventually. If it does not fail - there is the issue
in HBase RPC.
> 
> But technically, its a deadlock.
> 
> 
> 
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
> 
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Thursday, October 10, 2013 2:57 PM
> To: user@hbase.apache.org
> Subject: Re: Coprocessor Increments
> 
> Ok…
> 
> So…  I see you followed up to your earlier post.
> 
> Lets walk through this to make sure we're on the same page.
> 
> You put()  row 1 in table 1.
> The post_put() wants to insert a value in table 2.
> 
> Now suppose I have an htable pool of connections in my Region Observer.
> (Is this overkill? I'm treating the RO as if its a client connecting to HBase.)
> 
> RO performs a put() into the second table.
> 
> The RPC handlers are a Region server resource, yes?
> 
> So I can always increase them from the default (10) … but the point is that I can have
10 clients updating table A and then have a couple of different regions on the RS for the
table making RO requests.
> 
> Is that the case?
> 
> 
> On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <vrodionov@carrieriq.com> wrote:
> 
>> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>> 
>> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
>> 
>> Making RPC call from within handler's therad is:
>> 
>> A. expensive
>> B. may result in some deadlock -type of situations when no more incoming RPC calls
can be accepted because
>> everyone is connected to everyone in a spaghetti way and waiting on RPC calls to
complete
>> 
>> Let us say we have 2 region servers for simplicity:
>> RS1 and RS2. Both have pool of handler threads = 10
>> 
>> What happen when all 10 handlers in RS1  are trying to RPC RS2 and all 10 handlers
are trying to RPC RS2?
>> 
>> The same deadlock. Its all probabilistic .
>> 
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>> 
>> ________________________________________
>> From: Vladimir Rodionov
>> Sent: Thursday, October 10, 2013 12:09 PM
>> To: user@hbase.apache.org
>> Subject: RE: Coprocessor Increments
>> 
>> Classical deadlock :
>> 
>> CP-Region1 updates counter in CP-Region2 (waits on RPC)
>> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>> 
>> I think its an anti-pattern. Do not do cross region calls in region CP code.
>> 
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>> 
>> ________________________________________
>> From: Michael Segel [msegel_hadoop@hotmail.com]
>> Sent: Thursday, October 10, 2013 9:55 AM
>> To: user@hbase.apache.org
>> Cc: Ted Yu
>> Subject: Re: Coprocessor Increments
>> 
>> I think Andrew has a handle on it… my take is that you end up running out of resources
to handle an RPC connection while within your coprocessor and you're waiting for a resource
to be free and it can't because another coprocessor has an RPC resource and is also waiting
for a free resource.
>> 
>> Maybe its an over simplification, but if that's the case… you could always try
thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem)
or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
>> 
>> But the key is to really understanding and confirming what's causing the Deadlock
in detail.
>> 
>> On Oct 10, 2013, at 11:15 AM, John Weatherford <john.weatherford@telescope.tv>
wrote:
>> 
>>> Michael,
>>> I would also really like to know how this issue is caused also. I can't even
give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing
can be proved yet. While google-ing and looking for an answer I came across that old message
post  (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E).
This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but
we aren't entirely sure.
>>> 
>>> 
>>> 
>>> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>>>> Can we just take a quick pause…
>>>> 
>>>> John you wrote the following:
>>>> "We have been running into an RPC deadlock issue on HBase and from
>>>> investigation, we believe the root of the issue is in us doing cross
>>>> region increments from a coprocessor. After some further searching and
>>>> reading over this
>>>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>>>> we think that we can solve this by doing the increments locally on the
>>>> region. "
>>>> 
>>>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>>> 
>>>> Can either you or Andrew explain in detail what is meant by the RPC
>>>> deadlock?
>>>> 
>>>> This goes back to rethink how to implement coprocessors.
>>>> 
>>>> 
>>>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>>>> <john.weatherford@telescope.tv <mailto:john.weatherford@telescope.tv>>
>>>> wrote:
>>>> 
>>>>> Thank you ted. I might have to rethink my key design to accomodate
>>>>> this. With this design and using the totals in keys as you suggested,
>>>>> I would have to scan the entire "California" group to find the
>>>>> totals. Thank you very much for your help.
>>>>> 
>>>>> -John
>>>>> 
>>>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>>>> John:
>>>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>>>> 'California-**
>>>>>> 95424' is the start key of region 2. You can choose
>>>>>> 'California-**12346#total'
>>>>>> to be the row key where increment is done for region 1 and
>>>>>> 'California-**95424#total'
>>>>>> to be the row key where increment is done for region 2.
>>>>>> 
>>>>>> w.r.t. verification of whether given row key is indeed inside the
>>>>>> hosting
>>>>>> region, see the following:
>>>>>> 
>>>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>>> 
>>>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>>> 
>>>>>> throw new WrongRegionException("Requested row out of range for "
+
>>>>>> 
>>>>>>   op + " on HRegion " + this + ", startKey='" +
>>>>>> 
>>>>>> Cheers
>>>>>> 
>>>>>> 
>>>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>>>> john.weatherford@telescope.tv
>>>>>> <mailto:john.weatherford@telescope.tv>> wrote:
>>>>>> 
>>>>>>> First, Thank you everyone for the quick response.
>>>>>>> 
>>>>>>> Ted:
>>>>>>> The custom split policy could be an interesting solution.
>>>>>>> Regarding the
>>>>>>> other option you sent, if I pull the end row key, I could construct
>>>>>>> an end
>>>>>>> key, but if I suffix the row key with the end key of the region,
>>>>>>> would that
>>>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>>>> suffixed key
>>>>>>> now be "California-total-California-**12346"  and the other region's
>>>>>>> counter would be "California-total-California-**95424" and both
of
>>>>>>> these
>>>>>>> keys would actually end up on the second region since the sorting
>>>>>>> of the
>>>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>>> 
>>>>>>> Part of the question is that I don't understand if calling
>>>>>>> incrementColumnValue() on the region will always execute on the
region
>>>>>>> called, regardles of rowkey. If so, what happens when those regions
are
>>>>>>> merged?
>>>>>>> 
>>>>>>> Thanks again for the help!
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>>> 
>>>>>>>> bq. 'California-total' row in each region
>>>>>>>> 
>>>>>>>> Of course such row key needs to be suffixed with either start
or
>>>>>>>> end key
>>>>>>>> of
>>>>>>>> the corresponding region.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>>>> <mailto:yuzhihong@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> John:
>>>>>>>>> Looks like you need a 'California-total' row in each
region.
>>>>>>>>> 
>>>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1
in:
>>>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>>> 
>>>>>>>>> Cheers
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>>>> vrodionov@carrieriq.com <mailto:vrodionov@carrieriq.com>
>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Contrived Example
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor
to call
>>>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total"
 all on
>>>>>>>>>>>> 
>>>>>>>>>>> Region 1.
>>>>>>>>>> 
>>>>>>>>>> This would likely be on an insert on the same region.
But as
>>>>>>>>>> the table
>>>>>>>>>>>> grows, this secondary insert could end up
on another region.
>>>>>>>>>>>> If it is
>>>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>>>> which still
>>>>>>>>>>>> triggers a call to incrementColumnValue()
with a rowkey of
>>>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Are we now left with two rowkeys of "California-total"?
One on each
>>>>>>>>>>>> region server? If so, what happens if these
two regions are
>>>>>>>>>>>> compacted
>>>>>>>>>>>> into one?
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Best regards,
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Nope, Your "California-total" will migrate to Region
2 after region
>>>>>>>>>> split
>>>>>>>>>> is complete.
>>>>>>>>>> 
>>>>>>>>>> Vladimir Rodionov
>>>>>>>>>> Principal Platform Engineer
>>>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>>> 
>>>>>>>>>> ______________________________**__________
>>>>>>>>>> 
>>>>>>>>>> Confidentiality Notice:  The information contained
in this message,
>>>>>>>>>> including any attachments hereto, may be confidential
and is
>>>>>>>>>> intended
>>>>>>>>>> to be
>>>>>>>>>> read only by the individual or entity to whom this
message is
>>>>>>>>>> addressed. If
>>>>>>>>>> the reader of this message is not the intended recipient
or an
>>>>>>>>>> agent or
>>>>>>>>>> designee of the intended recipient, please note that
any review,
>>>>>>>>>> use,
>>>>>>>>>> disclosure or distribution of this message or its
attachments,
>>>>>>>>>> in any
>>>>>>>>>> form,
>>>>>>>>>> is strictly prohibited.  If you have received this
message in error,
>>>>>>>>>> please
>>>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com
and
>>>>>>>>>> delete or destroy any copy of this message and its
attachments.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> Confidentiality Notice:  The information contained in this message, including any
attachments hereto, may be confidential and is intended to be read only by the individual
or entity to whom this message is addressed. If the reader of this message is not the intended
recipient or an agent or designee of the intended recipient, please note that any review,
use, disclosure or distribution of this message or its attachments, in any form, is strictly
prohibited.  If you have received this message in error, please immediately notify the sender
and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its
attachments.
>> 
> 
> 


Mime
View raw message