hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
Date Tue, 03 Mar 2015 01:26:07 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344250#comment-14344250

stack commented on HBASE-11165:

In your list, IMO, 1. is reason enough to do small regions.

4. Smaller regions mean less of the keyspace is offline when balancing; also more even balance
is possible when regions are smaller.

bq. Can we only solve these three with many small regions?

If we just did small regions, without introducing anything new (other than doing what we already
do 'better'/'faster'), we could improve on your list without need to add custom compaction
policy(-ies) and the recording of interstices at 100MB intervals in metadata (which we'd have
to teach clients to read), etc.

Regards a problem statement, you want one on why we should tend down toward small rather than
continue our current trajectory of larger and larger regions, or do you want a problem statement
for the subject of this JIRA? Regards this JIRA, we have users who are headed toward 1M now
(Flurry reported being at 300k afraid to go up from there and Francis has 'larger' clusters)
so we have to deal.  You thinking we should explore going up from 10/20G toward 100G or 1TB?
(With stripe compactions++ and means of apportioning out the 1TB region, etc., to address
the 1-4 list above?).

> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>                 Key: HBASE-11165
>                 URL: https://issues.apache.org/jira/browse/HBASE-11165
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: stack
>         Attachments: HBASE-11165.zip, Region Scalability test.pdf, ScalableMeta.pdf,
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" and comments
on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe
even 50M later.  This issue is about discussing how we will do that (or if not 50M on a cluster,
how otherwise we can attain same end).
> More detail to follow.

This message was sent by Atlassian JIRA

View raw message