Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANZa=Gv_X_2DfoAKmVexzW5tgzT0P0k=kdmkPv4Jiw8kDc8WYg@mail.gmail.com>
References: 
 <CADcMMgFbbwOB5fxiGA0cBVCPzCxVAZuv6Q_hKHCanFyVO4ncyQ@mail.gmail.com>
 <CANZa=GvnrzirJjKMDePNvL0mn1qjqe=iDYA5me3xsZeTaH=ibQ@mail.gmail.com>
 <21843.61664.956285.397408@bfym.cloudera.com>
 <CADcMMgFXU+s4QBBFfGZTKzpGpPb+6o+4CA2HHVqfDNxdV1uAVg@mail.gmail.com>
 <CANZa=Gs13p81bVM5GtxhL28DEy18+sChXWGXqFr5Ji1JYTqmYQ@mail.gmail.com>
 <CAFb6Hi+BKE=is6kpeJb+aXM39JGZ_a1vC1bVzEqP+baVOneH+A@mail.gmail.com>
 <CANZa=Gv_X_2DfoAKmVexzW5tgzT0P0k=kdmkPv4Jiw8kDc8WYg@mail.gmail.com>
From: Matteo Bertozzi <theo.bertozzi@gmail.com>
Date: Wed, 13 May 2015 18:42:55 -0700
Message-ID: 
 <CAFb6HiKaAP-i+6phHrTa7ypmtBAO16YqrfvXOAK9jNr3XeT2Ow@mail.gmail.com>
Subject: Re: New post on hbase-1.1.0 throttling feature up on our Apache blog
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11c37daef91bb0051600d92a

--001a11c37daef91bb0051600d92a
Content-Type: text/plain; charset=UTF-8

@nick we have already something like that, which is HBASE-10993
and it is basically reordering requests based on how many scan.next you did.
(see the picture)
http://blog.cloudera.com/wp-content/uploads/2014/11/hbase-multi-f2.png
the problem is that we can't eject requests in execution and we are not
heavy enough on removing request from the queue and send a retry to the
client in case someone with more priority is in.

Matteo


On Wed, May 13, 2015 at 6:38 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:

> I guess what I'm thinking of is more about scheduling than
> quota/throttling. I don't want my online requests to sit in a queue behind
> MR requests while the MR work build up to it's quota amount. I want a
> scheduler to do time-slicing of operations, with preferential treatment
> given to online work over long scan ("analytical") work. For example, all
> scan RPC's "known" to cover "lots" of Cells get de-prioritized vs gets and
> short scans. Maybe this is synthesized with an RPC annotation marking it as
> "long" vs "short" -- MR scans are marked "long". I'm not sure, and I need
> to look more closely at recent scan improvements. IIRC, there's a heartbeat
> now, which maybe is a general mechanism allowing for long operations to not
> stomp over short operations. Heartbeat implies the long-running scan is
> coming up for air from time to time, allowing itself to be interrupted and
> defer to higher priority work. This isn't preemption, but does allow for an
> upper bound on how long the next queued task waits.
>
> On Wed, May 13, 2015 at 6:11 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
> wrote:
>
> > @nick what would you like to have? a match on a Job ID or something like
> > that?
> > currently only user/table/namespace are supported,
> > but group support can be easily added.
> > not sure about a job-id or job-name since we don't have that info on the
> > scan.
> >
> > On Wed, May 13, 2015 at 6:04 PM, Nick Dimiduk <ndimiduk@gmail.com>
> wrote:
> >
> > > Sorry. Yeah, sure, I can ask over there.
> > >
> > > The throttle was set by user in these tests.  You cannot directly
> > > > throttle a specific job, but do have the option to set the throttle
> > > > for a table or a namespace.  That might be sufficient for you to
> > > > achieve your objective (unless those jobs are run by one user and
> > > > access the same table.)
> > >
> > >
> > > Maybe running as different users is the key, but this seems like a very
> > > important use-case to support -- folks doing aggregate analysis
> > > concurrently on an online table.
> > >
> > > On Wed, May 13, 2015 at 5:53 PM, Stack <stack@duboce.net> wrote:
> > >
> > > > Should we add in your comments on the blog Govind: i.e. the answers
> to
> > > > Nicks' questions?
> > > > St.Ack
> > > >
> > > > On Wed, May 13, 2015 at 5:48 PM, Govind Kamat <gkamat@cloudera.com>
> > > wrote:
> > > >
> > > > >  > This is a great demonstration of these new features, thanks for
> > > > > pointing it
> > > > >  > out Stack.
> > > > >  >
> > > > >  > I'm curious: what percentile latencies are this reported? Does
> the
> > > > >  > non-throttled user see significant latency improvements in the
> 95,
> > > > 99pct
> > > > >  > when the competing, scanning users are throttled? MB/s and req/s
> > are
> > > > >  > managed at the region level? Region server level? Aggregate?
> > > > >
> > > > > The latencies reported in the post are average latencies.
> > > > >
> > > > > Yes, the non-throttled user sees an across-the-board improvement in
> > > > > the 95th and 99th percentiles, in addition to the improvement in
> > > > > average latency.  The extent of improvement is significant as well
> > but
> > > > > varies with the throttle pressure, just as in the case of the
> average
> > > > > latencies.
> > > > >
> > > > > The total throughput numbers (req/s) are aggregate numbers reported
> > by
> > > > > the YCSB client.
> > > > >
> > > > >  > These throttle points are by user? Is there a way for us to say
> > "all
> > > > MR
> > > > >  > jobs are lower priority than online queries"?
> > > > >  >
> > > > >
> > > > > The throttle was set by user in these tests.  You cannot directly
> > > > > throttle a specific job, but do have the option to set the throttle
> > > > > for a table or a namespace.  That might be sufficient for you to
> > > > > achieve your objective (unless those jobs are run by one user and
> > > > > access the same table.)
> > > > >
> > > > > Govind
> > > > >
> > > > >
> > > > >  > Thanks,
> > > > >  > Nick
> > > > >  >
> > > > >  > On Tue, May 12, 2015 at 1:58 PM, Stack <stack@duboce.net>
> wrote:
> > > > >  >
> > > > >  > > .. by our Govind.
> > > > >  > >
> > > > >  > > See here:
> > > > >  > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/hbase/entry/the_hbase_request_throttling_feature
> > > > >  > >
> > > > >  > > St.Ack
> > > > >  > >
> > > > >
> > > >
> > >
> >
>

--001a11c37daef91bb0051600d92a--