hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Embree <cemb...@gmail.com>
Subject Re: Sane max storage size for DN
Date Thu, 13 Dec 2012 05:08:00 GMT
Hi Mohammed,

The amount of RAM on the NN is related to the number of blocks... so let's
do some math. :)  1G of RAM to 1M blocks seems to be the general rule.

I'll probably mess this up so someone check my math:

9 PT ~ 9,216 TB ~ 9,437,184 GB of data.  Let's put that in 128MB blocks:
 according to kcalc that's 75,497,472 of 128 MB Blocks.
Unless I missed this by an order of magnitude (entirely possible... I've
been drinking since 6), that sound like 76G of RAM (above OS requirements).
 128G should kick it's ass; 256G seems like a waste of $$.

Hmm... That makes the NN sound extremely efficient.  Someone validate me or
kick me to the curb.


On Wed, Dec 12, 2012 at 10:52 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Hello Michael,
>       It's an array. The actual size of the data could be somewhere around
> 9PB(exclusive of replication) and we want to keep the no of DNs as less as
> possible. Computations are not too frequent, as I have specified earlier.
> If I have 500TB in 1 DN, the no of DNs would be around 49. And, if the
> block size is 128MB, the no of blocks would be 201326592. So, I was
> thinking of having 256GB RAM for the NN. Does this make sense to you?
> Many thanks.
> Regards,
>     Mohammad Tariq
> On Thu, Dec 13, 2012 at 12:28 AM, Michael Segel <michael_segel@hotmail.com
> > wrote:
>> 500 TB?
>> How many nodes in the cluster? Is this attached storage or is it in an
>> array?
>> I mean if you have 4 nodes for a total of 2PB, what happens when you lose
>> 1 node?
>> On Dec 12, 2012, at 9:02 AM, Mohammad Tariq <dontariq@gmail.com> wrote:
>> Hello list,
>>           I don't know if this question makes any sense, but I would like
>> to ask, does it make sense to store 500TB (or more) data in a single DN?If
>> yes, then what should be the spec of other parameters *viz*. NN & DN
>> RAM, N/W etc?If no, what could be the alternative?
>> Many thanks.
>> Regards,
>>     Mohammad Tariq

View raw message