Return-Path: 
 <user-return-34017-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A215E894
	for <apmail-hbase-user-archive@www.apache.org>;
 Mon, 11 Feb 2013 03:44:45 +0000 (UTC)
Received: (qmail 48995 invoked by uid 500); 11 Feb 2013 03:44:43 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 48645 invoked by uid 500); 11 Feb 2013 03:44:42 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 48598 invoked by uid 99); 11 Feb 2013 03:44:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Feb 2013 03:44:42 +0000
X-ASF-Spam-Status: No, hits=1.8 required=5.0
	tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of joarderm@gmail.com designates
 209.85.128.176 as permitted sender)
Received: from [209.85.128.176] (HELO mail-ve0-f176.google.com)
 (209.85.128.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Feb 2013 03:44:34 +0000
Received: by mail-ve0-f176.google.com with SMTP id cz10so4766496veb.21
        for <multiple recipients>; Sun, 10 Feb 2013 19:44:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=x-received:mime-version:in-reply-to:references:from:date:message-id
         :subject:to:cc:content-type;
        bh=CFUlMhCfFbpmkHtR5tc4O2hR5fUob3fcLNHT8PF+dRI=;
        b=TF70AveG8cmh+TdXo+6LNsVJh7Gz0W1eRVDVyhn7nb8Xm2oCl2yK+uDMVQQSkL+2pU
         BGgNtQHuoXWutmIqPbCdU2JY09a/9rFtz+Xx02Ot9UKzAEprisyeynYPz/joGS+XgFRd
         wSfKIurRpUEBQo2vETTNb9PmAHmg03Wx27Rf+kshnq7XN7O1VBSGcIpqIxRuYsaWRsyO
         3IMbu/oAPuhd7G3BR/8KiJ7X2UXmRmQ0fiT3V0QaFjrmVgo42Lm1wgWJRNujWsrN7c+b
         dw5CwEuZYW9YOLbR7G7B9USl2Xj0cPzlp1uw7NGPFIcsNBRVrSamqYU39q0MHxs6MZaW
         1WmA==
X-Received: by 10.220.219.80 with SMTP id ht16mr16730366vcb.30.1360554253812;
 Sun, 10 Feb 2013 19:44:13 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.32.68 with HTTP; Sun, 10 Feb 2013 19:43:33 -0800 (PST)
In-Reply-To: 
 <CAGngS9eVrhi0vDbKFbAZXwSMGN80mYiTw_y2tXQQsDZqTRxwzg@mail.gmail.com>
References: 
 <CAM4M=K7LE+nyJYrAv4TgsRr+Ef7iyNT-Dd0k5wT-UeOZMf8PeA@mail.gmail.com>
 <CAGngS9eVrhi0vDbKFbAZXwSMGN80mYiTw_y2tXQQsDZqTRxwzg@mail.gmail.com>
From: Joarder KAMAL <joarderm@gmail.com>
Date: Mon, 11 Feb 2013 14:43:33 +1100
Message-ID: 
 <CAM4M=K5J2F-w0TbDKzuoCDNPTEjUX4_Lx1NwEMd5FxRmQaJtQw@mail.gmail.com>
Subject: Re: HBase Region/Table Hotspotting
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=14dae9cfc894ff03ec04d56ab834
X-Virus-Checked: Checked by ClamAV on apache.org

--14dae9cfc894ff03ec04d56ab834
Content-Type: text/plain; charset=UTF-8

Hi Kevin,

Thanks a lot for your great answers.
Regarding Q5. To clarify,

lets say Facebook is using HBase for the integrated messaging/chat/email
system in a very large-scale setup. And schema design of such system can
change over the years (even over the months). Workload patterns may also
change due to different usage characteristics (like the rate of messaging
may be higher during a protest/specific event in a particular country). So,
region/table hotspots have been created at random region servers within the
cluster despite careful schema design and pre-planning.

The facebook team rush to split the hotspotted regions manually and
redistribute them over a new set of physical machines which are recently
added to the system to increase scalability in the face of high user
demand. Now hotspotted region data could be transferred into new physical
machines gradually to handle the situations. Now if the shard (region) size
is small enough then data transfer cost over the network could be minimum
otherwise large volume of data needs to be transferred instantly.

I have found in many places it is discouraged to have a large number of
regions systems. However, would it be possible to have very large number of
regions in a system thus minimizing data transfer cost in case hotspotting
due to workload/design characteristics. Is there any drawbacks or known
side-effects? I am rethinking other possibilities other pre-planned schema
and row-key designs.


Thanks again.

Regards,
Joarder Kamal


On 11 February 2013 13:32, Kevin O'dell <kevin.odell@cloudera.com> wrote:

> Hi Joarder,
>
>   Welcome to the HBase world.  Let me take some time to address your
> questions the best I can:
>
>  1. How often you are facing Region or Table Hotspotting in HBase
>    production systems? <--- Hotspotting is not something that just happens.
>  This is usually caused by bad key design and writing to one region more
> than the others.  I would recommend watching some of Lar's YouTube videos
> on Schema Design in HBase.
>    2. If a hotspot is created, how quickly it is automatically cleared out
>    (assuming sudden workload change)? <--- It will not be automatically
> "cleared out" I think you may be misinformed here.  Basically, it is on you
> to watch you table and your write distribution and determine that you have
> a hotspot and take the necessary action.  Usually the only action is to
> split the region.  If hotspots become a habitual problem you would most
> likely want to go back and re-evaluate your current key.
>    3. How often this kind of situation happens - A hotspot is detected and
>    vanished out before taking an action? or hotspots stays longer period of
>    time? <--- Please see above
>    4. Or if the hotspot is stays, how it is handled (in general) in
>    production system? <--- Some people have to hotspot on purpose early on,
> because they only write to a subset of regions.  You will have to manually
> watch for hotspots(which is much easier in later releases).
>    5. How large data transfer cost is minimized or avoid for re-sharding
>    regions within a cluster in a single data center or within WAN? <--- Not
> quite sure what you are saying here, so I will take a best guess at it.
>  Sharding is handled in HBase by region splitting.  The best way to success
> in HBase is to understand your data and you needs BEFORE you create you
> table and start writing into HBase.  This way you can presplit your table
> to handle the incoming data and you won't have to do a massive amounts of
> splits.  Later you can allow HBase to split your tables manually, or you
> can set the maxfile size high and manually control the splits or sharding.
>    6. Is hotspoting in HBase cluster is really a issue (big!) nowadays for
>    OLAP workloads and real-time analytics? <--- Just design your schema
> correctly and this should not be a problem for you.
>
> Please let me know if this answers your questions.
>
> On Sun, Feb 10, 2013 at 9:17 PM, Joarder KAMAL <joarderm@gmail.com> wrote:
>
> > This is my first email in the group. I am having a more general and
> > open-ended question but hope to get some reasoning from the HBase user
> > communities.
> > I am a very basic HBase user and still learning. My intention to use
> HBase
> > in one of our research project. Recently I was looking through Lars
> > George's book "HBase - The Definitive Guide" and two particular topics
> > caught my eyes. One is 'Region and Table Hotspotting' and the other is
> > 'Region Auto-Sharding and Merging'.
> >
> > *Scenario: *
> > If a hotspot is created in a particular region or in a table (having
> > multiple regions) due to sudden workload change, then one may split the
> > region into further small pieces and distributed it to a number of
> > available physical machine in the cluster. This process should require
> > large data transfer between different machines in the cluster and incur a
> > performance cost. One may also change the 'key' definition and manage the
> > regions. But I am not sure how effective or logical to change key designs
> > on a production system.
> >
> > *Questions:*
> >
> >    1. How often you are facing Region or Table Hotspotting in HBase
> >    production systems?
> >    2. If a hotspot is created, how quickly it is automatically cleared
> out
> >    (assuming sudden workload change)?
> >    3. How often this kind of situation happens - A hotspot is detected
> and
> >    vanished out before taking an action? or hotspots stays longer period
> of
> >    time?
> >    4. Or if the hotspot is stays, how it is handled (in general) in
> >    production system?
> >    5. How large data transfer cost is minimized or avoid for re-sharding
> >    regions within a cluster in a single data center or within WAN?
> >    6. Is hotspoting in HBase cluster is really a issue (big!) nowadays
> for
> >    OLAP workloads and real-time analytics?
> >
> >
> > Further directions to more information about region/table hotspotting is
> > most welcome.
> >
> > Many thanks in advance.
> >
> > Regards,
> > Joarder Kamal
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

--14dae9cfc894ff03ec04d56ab834--