hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
Date Fri, 29 Aug 2014 17:23:55 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115524#comment-14115524

stack commented on HBASE-11165:

bq. ..by the way, could somebody estimate the distribution or read vs writes to meta table,
in terms of requests per second/networking traffic/disk access?

[~toffer] or [~virag] Do you have any numbers from your experiments?

bq. I believe there are 2 dimensions here, right?

There is a more immediate 3rd dimension/option and that is what we have now where we have
a single master and if it fails, backup assumes its role.

For your dimension 1., (also known as  "HBASE-10296 Replace ZK with a consensus lib(paxos,zab
or raft running within master processes to provide better master failover performance and
state consistency"), it would be a nice-to-have but if we can, lets try and avoid our having
to have a HA master. As long as the master comes back inside some reasonable window and the
cluster can keep on chugging while its figuring out its recovery, this would be a simpler
deploy than one that needs a HA master mini-cluster.  Multi-master would help scale reads
but as you note, doesn't help if one massive meta region and we want to cache it.

We'd go to partitioned masters, as I see it, if we can't make one master do.

bq. The viable combined solution ...

I suggest we look at single master with split meta served by regionservers first?

> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>                 Key: HBASE-11165
>                 URL: https://issues.apache.org/jira/browse/HBASE-11165
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: stack
>         Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" and comments
on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe
even 50M later.  This issue is about discussing how we will do that (or if not 50M on a cluster,
how otherwise we can attain same end).
> More detail to follow.

This message was sent by Atlassian JIRA

View raw message