hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Maximum Storage size in a Single datanode
Date Wed, 30 Jan 2013 12:02:07 GMT
I completely agree with everyone in the thread. Perhaps you are not
concerned much about the processing part, but it is still not a good idea.
Remember the power of Hadoop lies in the principle of "divide and rule" and
you are trying to go against that.

On Wednesday, January 30, 2013, Chris Embree <cembree@gmail.com> wrote:
> You should probably think about this in a more cluster fashion.  A single
node with a PB of data is probably not a good allocation of CPU : Disk
ration.  In addition, you need enough RAM on your NameNode to keep track of
all of your blocks.  A few nodes with a PB each would quickly drive up NN
RAM requirements.
> As others have mentioned, the local file system that HDFS sits on top of
may have limits.  We're going to use EXT4 which should handle that much,
but it's probably still not a good idea.
> If you're just thinking of storing lots of data, you might consider
GlusterFS instead.
> I highly recommend RedHat over Ubuntu.
> Hope that helps.
> On Wed, Jan 30, 2013 at 5:40 AM, jeba earnest <jebaearnest@yahoo.com>
> I want to use either UBUNTU or REDHAT .
> I just want to know how much storage space we can allocate in a single
data node.
> Is there any limitations in hadoop for storage in single node?
> Regards,
> Jeba
> ________________________________
> From: "Pamecha, Abhishek" <apamecha@ebay.com>
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>; jeba earnest <
> Sent: Wednesday, 30 January 2013 2:45 PM
> Subject: Re: Maximum Storage size in a Single datanode
> What would be the reason you would do that?
> You would want to leverage distributed dataset for higher availability
and better response times.
> The maximum storage depends completely on the disks  capacity of your
nodes and what your OS supports. Typically I have heard of about 1-2
TB/node to start with, but I may be wrong.
> -abhishek
> From: jeba earnest <jebaearnest@yahoo.com>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>, jeba earnest
> Date: Wednesday, January 30, 2013 1:38 PM
> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: Maximum Storage size in a Single datanode
> Hi,
> Is it possible to keep 1 Petabyte in a single data node?
> If not, How much is the maximum storage for a particular data node?
> Regards,
> M. Jeba

Warm Regards,

View raw message