hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rahul gidwani <rahul.gidw...@gmail.com>
Subject Re: Allowing clients to set priorities
Date Fri, 22 Apr 2016 16:53:27 GMT
Sure sorry didn't provide a good example.

There are two situations where I have thought this feature might be
useful.   Maybe more...

1.  For something like Phoenix, there are situations where you want
certain operations for tables / users to always have handlers
available to do work.  For example any write to an index table should
never block.  One way to almost guarantee this is to give it its own
special set of handlers and allow from a client call to denote that
this call is meant for this specific handler pool.

2.  Suppose you have a large cluster with 1 really large table.  You
can't really use regionserver groups and there are clients doing all
sorts of operations on this cluster.  Map/Reduce jobs (long scans,
heavy reduces), processing pipeline, random reads, etc.  There are
features already to de-prioritize long running scanners and there is
rpc throttling and we can split up the handlers into read, write.
Currently there is no easy way to say I want to reserve 10 handlers on
each regionserver for the processing pipeline and from the client you
can pass something along that would tell the server to use this
special handler pool.

Also Stack, I saw your TODO and I believe we could get rid of the
AnnotatinoReadingPriorityFunction.

We can talk about design if folks are interested.

Thanks
rahul


> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <olorinbant@gmail.com> wrote:
>
> This is interesting idea. Sorry if I missed some context - what's the
> primary incentive here? What's examples of those categorized thread pools?
>
> Sounds intersecting a bit with HBASE-15136
> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline scheduling
> for RPC requests) in the area of rpc prioritizing.
>
> -Mikhail
>
>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <stack@duboce.net> wrote:
>>
>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <rahul.gidwani@gmail.com>
>> wrote:
>>
>>> I was wondering if people would be interested in allowing the client to
>>> specify priorities?  I really think we are good responsible adults and
>> wont
>>> abuse this feature.   :)
>>>
>>> This would not just be for one particular operation but all operations.
>>> I'll make it feature complete.
>> Sounds sweet.
>>
>> RPC passes priority in the header already IIRC.
>>
>> We could then purge our ugly decompose of the request just to figure what
>> it is so we can prioritize based off annotation.
>>
>> St.Ack
>>
>>
>>
>>> As for batch operations prioirites would be at batch level.
>>>
>>> I know the phoenix guys would really like this feature as it would really
>>> help with their indexing work.
>>
>>
>>> Eventually I think it would be nice to get to a point where we can have
>>> some sort of configurable reservation system.  Where regionservers could
>>> have handler groups and we could send a little bit more info with the rpc
>>> call to specify the reserved set of handlers they would like to utilize.
>>>
>>> thanks
>>> rahul
>
>
>
> --
> Thanks,
> Michael Antonov

Mime
View raw message