hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: RS storage capacity mismatches
Date Fri, 22 Aug 2014 21:53:34 GMT
bq. to add some drives (if we can) to existing nodes

+1 to the above.

hdfs manages data nodes so that percentage of capacity is on par across
nodes. This would give you better balance.

Cheers


On Fri, Aug 22, 2014 at 2:49 PM, Ted Tuttle <ted@mentacapital.com> wrote:

> Ted, your question is a good one.  I hadn't thought about that much.
>
>
>
> I suppose if HBase were to be aware of local resources, then we'd have to
> configure the RS memory in accordance w/ the relative size of the storage.
> I would think we would allocate larger heaps (up to a point to avoid
> GC-induced timeouts) to processes but leave the various fractions (e.g.
> block cache fraction) intact.
>
>
>
> Now that you bring the memory mismatch into focus, I am thinking it is
> perhaps better to add some drives (if we can) to existing nodes to get them
> to parity w/ newer higher storage density nodes.
>
>
>
> Thanks for discussion!
>
>
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, August 22, 2014 2:40 PM
> *To:* Sean Busbey
> *Cc:* user@hbase.apache.org; Development
>
> *Subject:* Re: RS storage capacity mismatches
>
>
>
> Thanks for the link, Sean.
>
>
>
> Ted:
>
> What about available memory on the new nodes ? How much larger is it
> compared to old nodes ?
>
>
>
> Cheers
>
>
>
> On Fri, Aug 22, 2014 at 2:35 PM, Sean Busbey <busbey@cloudera.com> wrote:
>
> Hi Ted!
>
>
>
> HBase doesn't currently take local node resources into account when
> balancing across region servers. There's currently some discussion on
> HBASE-11780 about building this funtionality[1]. I'm sure they'd love more
> input or details about the scale you'd need to work at.
>
>
>
>
>
> -Sean
>
>
>
> [1]: https://issues.apache.org/jira/browse/HBASE-11780
>
>
>
> On Fri, Aug 22, 2014 at 4:07 PM, Ted Tuttle <ted@mentacapital.com> wrote:
>
> Hello Ted-
>
> We are considering added new nodes to existing cluster that have higher
> storage capacity than current nodes.  We hope, but do not know, that HBase
> would assign more regions to RSs that have more storage capacity.
>
> We are considering 2x storage on new nodes vs current node.
>
> Thanks,
> Ted
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, August 22, 2014 1:58 PM
> To: user@hbase.apache.org
> Cc: Development
> Subject: Re: RS storage capacity mismatches
>
>
> Some clarification: by unequal storage capacity, you are expecting certain
> region servers to host fewer regions, right ?
>
> How big is the difference between memory capacity among the old and new
> servers ?
>
> Cheers
>
> On Fri, Aug 22, 2014 at 1:52 PM, Ted Tuttle <ted@mentacapital.com<mailto:
> ted@mentacapital.com>> wrote:
> Hello-
>
> How does HBase handle region servers with unequal storage capacity?  We
> made the original nodes of our cluster a bit light on storage and are
> wondering what would happen is we started adding nodes w/ more storage
> capacity.
>
> Thanks,
> Ted
>
>
>
>
>
> --
>
> Sean
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message