Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 4 Sep 2014 03:40:53 +0000 (UTC)
From: "Francis Liu (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12714102.1400037780000.14911.1409802053889@Atlassian.JIRA>
In-Reply-To: <JIRA.12714102.1400037780000@Atlassian.JIRA>
References: <JIRA.12714102.1400037780000@Atlassian.JIRA>
 <JIRA.12714102.1400037780201@arcas>
Subject: [jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M
 regions and beyond (50M regions?)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120912#comment-14120912 ] 

Francis Liu commented on HBASE-11165:
-------------------------------------

{quote}
Yeah I know. What I'm saying is that we should work on getting there before working on the more complex split meta and split master. I would argue that we can get on par (or better) than the NN since it's doing more active writes then meta on a stable cluster. Then when that happens the NN will be the bottleneck and there will be no need for split meta.
{quote}
It's hard to make a comparison IMHO. NN uses a filer for WAL (at least for us). It's not an LSM so it doesn't suffer from write amplification. Major compaction could just creep up and you could get hosed till its done.  Having higher write throughput would definitely be a good thing but IMHO the clear way to scale, is to split meta as it addresses a bunch of issues and enables horizontal scalability for regions. Bottom line for us is we need to scale to 1M regions (soon) and beyond. The guys here will help us with any hdfs related blockers.

> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>
>                 Key: HBASE-11165
>                 URL: https://issues.apache.org/jira/browse/HBASE-11165
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: stack
>         Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf
>
>
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" and comments on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later.  This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end).
> More detail to follow.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)