hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Allowing clients to set priorities
Date Mon, 25 Apr 2016 17:02:49 GMT
On Fri, Apr 22, 2016 at 3:41 PM, rahul gidwani <rahul.gidwani@gmail.com>
wrote:

> ...
> For the exposing priority to the client I was thinking this work goes
> into OperationWithAttributes or Operation we can add a setPriority()
> and getPriority() and by default they are set to normal priority
> unless explicitly overridden.  For things like Meta table I am
> assuming there will be some resolution system we come up with that
> wont be exposed but documented.  Maybe the logic can go into
> Operation.getPriority() and then you would simply call
> controller.setProperty(operation.getPriority).  The batch operations
> could do something like take the highest priority individual operation
> and that become the priority at the entire batch level.
>
>
I think we need to wire up client setting priority as you suggest above.
How it is implemented is another matter. I think it important to cast
client 'priority' as an advisory that may or not be respected.


> Another option is to piggy back on Matteo's great work on Quotas and
> we could add a reservation type system.  Maybe similar functionality
> and behavior as quotas but phase 1 is to only be able to reserve
> handlers for (tables, users, namespaces)  - If we go that route, no
> need to expose priorities to the clients, they will just refer to some
> reservation cache and route them to the appropriate group of handlers.
> Because this information would probably be stored in the Quota  table
> and we need this prior to starting up handlers - this might require
> discussions or a design discussion.
>
> I think going the reservation route might be more elegant (but I need
> to hash out a few things before getting a design ready)  but I would
> love to hear what you guys think.
>
>
We have to do this too (smile).

To my mind, clients having to know about 'reservation caches' or 'handlers'
is TMI and another dimension on multitenancy that an operator would control
under the wraps rather than user.

Start small (smile).

Good stuff Rahul,
St.Ack



> Thanks for discussing this,
> rahul
>
>
>
>
> > On Apr 22, 2016, at 10:35 AM, Stack <stack@duboce.net> wrote:
> >
> > A few things:
> >
> > + Our Matteo reminded me of the HBASE-11048 "Support setting custom
> > priority per client RPC"/PHOENIX-938 "Use higher priority queue for index
> > updates to prevent deadlock" work which adds in a factory so clients can
> do
> > their rpccontroller. You'd build on this or controllers will always pass
> > priority regardless of the rpccontroller implementation? How would your
> > work and it relate? You'd surface priority on each method?
> > + What are you thinking regards priority on meta? How to not preempt the
> > internal workings of hbase itself?
> >
> > Answers to above can wait till design time, np.
> >
> > Is there an associated phoenix issue other than PHOENIX-938 that goes w/
> > this work?
> >
> > Thanks Rahul,
> > St.Ack
> >
> > On Fri, Apr 22, 2016 at 9:53 AM, rahul gidwani <rahul.gidwani@gmail.com>
> > wrote:
> >
> >> Sure sorry didn't provide a good example.
> >>
> >> There are two situations where I have thought this feature might be
> >> useful.   Maybe more...
> >>
> >> 1.  For something like Phoenix, there are situations where you want
> >> certain operations for tables / users to always have handlers
> >> available to do work.  For example any write to an index table should
> >> never block.  One way to almost guarantee this is to give it its own
> >> special set of handlers and allow from a client call to denote that
> >> this call is meant for this specific handler pool.
> >>
> >> 2.  Suppose you have a large cluster with 1 really large table.  You
> >> can't really use regionserver groups and there are clients doing all
> >> sorts of operations on this cluster.  Map/Reduce jobs (long scans,
> >> heavy reduces), processing pipeline, random reads, etc.  There are
> >> features already to de-prioritize long running scanners and there is
> >> rpc throttling and we can split up the handlers into read, write.
> >> Currently there is no easy way to say I want to reserve 10 handlers on
> >> each regionserver for the processing pipeline and from the client you
> >> can pass something along that would tell the server to use this
> >> special handler pool.
> >>
> >> Also Stack, I saw your TODO and I believe we could get rid of the
> >> AnnotatinoReadingPriorityFunction.
> >>
> >> We can talk about design if folks are interested.
> >>
> >> Thanks
> >> rahul
> >>
> >>
> >>>> On Apr 21, 2016, at 12:05 AM, Mikhail Antonov <olorinbant@gmail.com>
> >>> wrote:
> >>>
> >>> This is interesting idea. Sorry if I missed some context - what's the
> >>> primary incentive here? What's examples of those categorized thread
> >> pools?
> >>>
> >>> Sounds intersecting a bit with HBASE-15136
> >>> <https://issues.apache.org/jira/browse/HBASE-15136> (deadline
> scheduling
> >>> for RPC requests) in the area of rpc prioritizing.
> >>>
> >>> -Mikhail
> >>>
> >>>> On Wed, Apr 20, 2016 at 11:21 PM, Stack <stack@duboce.net> wrote:
> >>>>
> >>>> On Wed, Apr 20, 2016 at 1:47 PM, rahul gidwani <
> rahul.gidwani@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> I was wondering if people would be interested in allowing the client
> to
> >>>>> specify priorities?  I really think we are good responsible adults
> and
> >>>> wont
> >>>>> abuse this feature.   :)
> >>>>>
> >>>>> This would not just be for one particular operation but all
> operations.
> >>>>> I'll make it feature complete.
> >>>> Sounds sweet.
> >>>>
> >>>> RPC passes priority in the header already IIRC.
> >>>>
> >>>> We could then purge our ugly decompose of the request just to figure
> >> what
> >>>> it is so we can prioritize based off annotation.
> >>>>
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>>> As for batch operations prioirites would be at batch level.
> >>>>>
> >>>>> I know the phoenix guys would really like this feature as it would
> >> really
> >>>>> help with their indexing work.
> >>>>
> >>>>
> >>>>> Eventually I think it would be nice to get to a point where we can
> have
> >>>>> some sort of configurable reservation system.  Where regionservers
> >> could
> >>>>> have handler groups and we could send a little bit more info with
the
> >> rpc
> >>>>> call to specify the reserved set of handlers they would like to
> >> utilize.
> >>>>>
> >>>>> thanks
> >>>>> rahul
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Michael Antonov
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message