Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 3 Sep 2014 17:44:54 +0000 (UTC)
From: "Andrey Stepachev (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12714102.1400037780000.8015.1409766294399@Atlassian.JIRA>
In-Reply-To: <JIRA.12714102.1400037780000@Atlassian.JIRA>
References: <JIRA.12714102.1400037780000@Atlassian.JIRA>
 <JIRA.12714102.1400037780201@arcas>
Subject: [jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M
 regions and beyond (50M regions?)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120142#comment-14120142 ] 

Andrey Stepachev commented on HBASE-11165:
------------------------------------------

bq.Yeah, we'll have to go this route if we are trying to keep state of a big cluster in heap. Could work on making the representation more compact. You arguing for single meta region Andrey Stepachev then? There is also the on-hdfs size to consider (write-amplification) and the r/w i/os.

For sure, compact representation doesn't implicate single meta. Compact meta allows to bother with split meta only for really big installations. But how HDFS would handle that, as [~mantonov] mentioned above.

As for compact META representations we can use other technics to reduce HDFS impact for big meta.

> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>
>                 Key: HBASE-11165
>                 URL: https://issues.apache.org/jira/browse/HBASE-11165
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: stack
>         Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf
>
>
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" and comments on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later.  This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end).
> More detail to follow.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)