hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francis Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
Date Tue, 19 Aug 2014 01:56:19 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101707#comment-14101707

Francis Liu commented on HBASE-11165:

meta is only modified by the master, and the master is mainly doing operations on meta. so
if meta is down you have a master up which is not useful at much. 
Hmm so in this context things are neither really better or worse with either option?

Also since all writes to meta are controlled by the master having meta anywhere else will
not improve things since the master is the one processing requests and you also have to go
to another server to perform the operation. 
For writes you already have to go to a bunch of other servers (datanodes) to perform the write
operation and for reads worst case remote dfs read. Also as we've pointed out in our experiments
that overhead is not much (if any at all) and overshadowed by the gains you get from horizontal
scalability through SoC. 

if you are talking about the read side, I can understand why you want to split it, but isn't
having multiple copies even as cache simpler and you'll get the same results?
I'm talking about both read and writes. Being able to split it means being able to have read/write
throughput equivalent to the number of machines hosting the table. Have less write amplification
(which is already an issue) as well as horizontally scale to have enough memory to have meta
in block cache. 

> Scaling so cluster can host 1M regions and beyond (50M regions?)
> ----------------------------------------------------------------
>                 Key: HBASE-11165
>                 URL: https://issues.apache.org/jira/browse/HBASE-11165
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: stack
>         Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" and comments
on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe
even 50M later.  This issue is about discussing how we will do that (or if not 50M on a cluster,
how otherwise we can attain same end).
> More detail to follow.

This message was sent by Atlassian JIRA

View raw message