hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: HBase Region/Table Hotspotting
Date Mon, 11 Feb 2013 04:03:51 GMT
Matt Corgan summarized the pro and con of having large number of regions
here:

https://issues.apache.org/jira/browse/HBASE-7667?focusedCommentId=13575024&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13575024

Cheers

On Sun, Feb 10, 2013 at 7:43 PM, Joarder KAMAL <joarderm@gmail.com> wrote:

> Hi Kevin,
>
> Thanks a lot for your great answers.
> Regarding Q5. To clarify,
>
> lets say Facebook is using HBase for the integrated messaging/chat/email
> system in a very large-scale setup. And schema design of such system can
> change over the years (even over the months). Workload patterns may also
> change due to different usage characteristics (like the rate of messaging
> may be higher during a protest/specific event in a particular country). So,
> region/table hotspots have been created at random region servers within the
> cluster despite careful schema design and pre-planning.
>
> The facebook team rush to split the hotspotted regions manually and
> redistribute them over a new set of physical machines which are recently
> added to the system to increase scalability in the face of high user
> demand. Now hotspotted region data could be transferred into new physical
> machines gradually to handle the situations. Now if the shard (region) size
> is small enough then data transfer cost over the network could be minimum
> otherwise large volume of data needs to be transferred instantly.
>
> I have found in many places it is discouraged to have a large number of
> regions systems. However, would it be possible to have very large number of
> regions in a system thus minimizing data transfer cost in case hotspotting
> due to workload/design characteristics. Is there any drawbacks or known
> side-effects? I am rethinking other possibilities other pre-planned schema
> and row-key designs.
>
>
> Thanks again.
>
> Regards,
> Joarder Kamal
>
>
> On 11 February 2013 13:32, Kevin O'dell <kevin.odell@cloudera.com> wrote:
>
> > Hi Joarder,
> >
> >   Welcome to the HBase world.  Let me take some time to address your
> > questions the best I can:
> >
> >  1. How often you are facing Region or Table Hotspotting in HBase
> >    production systems? <--- Hotspotting is not something that just
> happens.
> >  This is usually caused by bad key design and writing to one region more
> > than the others.  I would recommend watching some of Lar's YouTube videos
> > on Schema Design in HBase.
> >    2. If a hotspot is created, how quickly it is automatically cleared
> out
> >    (assuming sudden workload change)? <--- It will not be automatically
> > "cleared out" I think you may be misinformed here.  Basically, it is on
> you
> > to watch you table and your write distribution and determine that you
> have
> > a hotspot and take the necessary action.  Usually the only action is to
> > split the region.  If hotspots become a habitual problem you would most
> > likely want to go back and re-evaluate your current key.
> >    3. How often this kind of situation happens - A hotspot is detected
> and
> >    vanished out before taking an action? or hotspots stays longer period
> of
> >    time? <--- Please see above
> >    4. Or if the hotspot is stays, how it is handled (in general) in
> >    production system? <--- Some people have to hotspot on purpose early
> on,
> > because they only write to a subset of regions.  You will have to
> manually
> > watch for hotspots(which is much easier in later releases).
> >    5. How large data transfer cost is minimized or avoid for re-sharding
> >    regions within a cluster in a single data center or within WAN? <---
> Not
> > quite sure what you are saying here, so I will take a best guess at it.
> >  Sharding is handled in HBase by region splitting.  The best way to
> success
> > in HBase is to understand your data and you needs BEFORE you create you
> > table and start writing into HBase.  This way you can presplit your table
> > to handle the incoming data and you won't have to do a massive amounts of
> > splits.  Later you can allow HBase to split your tables manually, or you
> > can set the maxfile size high and manually control the splits or
> sharding.
> >    6. Is hotspoting in HBase cluster is really a issue (big!) nowadays
> for
> >    OLAP workloads and real-time analytics? <--- Just design your schema
> > correctly and this should not be a problem for you.
> >
> > Please let me know if this answers your questions.
> >
> > On Sun, Feb 10, 2013 at 9:17 PM, Joarder KAMAL <joarderm@gmail.com>
> wrote:
> >
> > > This is my first email in the group. I am having a more general and
> > > open-ended question but hope to get some reasoning from the HBase user
> > > communities.
> > > I am a very basic HBase user and still learning. My intention to use
> > HBase
> > > in one of our research project. Recently I was looking through Lars
> > > George's book "HBase - The Definitive Guide" and two particular topics
> > > caught my eyes. One is 'Region and Table Hotspotting' and the other is
> > > 'Region Auto-Sharding and Merging'.
> > >
> > > *Scenario: *
> > > If a hotspot is created in a particular region or in a table (having
> > > multiple regions) due to sudden workload change, then one may split the
> > > region into further small pieces and distributed it to a number of
> > > available physical machine in the cluster. This process should require
> > > large data transfer between different machines in the cluster and
> incur a
> > > performance cost. One may also change the 'key' definition and manage
> the
> > > regions. But I am not sure how effective or logical to change key
> designs
> > > on a production system.
> > >
> > > *Questions:*
> > >
> > >    1. How often you are facing Region or Table Hotspotting in HBase
> > >    production systems?
> > >    2. If a hotspot is created, how quickly it is automatically cleared
> > out
> > >    (assuming sudden workload change)?
> > >    3. How often this kind of situation happens - A hotspot is detected
> > and
> > >    vanished out before taking an action? or hotspots stays longer
> period
> > of
> > >    time?
> > >    4. Or if the hotspot is stays, how it is handled (in general) in
> > >    production system?
> > >    5. How large data transfer cost is minimized or avoid for
> re-sharding
> > >    regions within a cluster in a single data center or within WAN?
> > >    6. Is hotspoting in HBase cluster is really a issue (big!) nowadays
> > for
> > >    OLAP workloads and real-time analytics?
> > >
> > >
> > > Further directions to more information about region/table hotspotting
> is
> > > most welcome.
> > >
> > > Many thanks in advance.
> > >
> > > Regards,
> > > Joarder Kamal
> > >
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message