hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay Thakorlal <vijayj...@hotmail.com>
Subject RE: Maximum Storage size in a Single datanode
Date Wed, 30 Jan 2013 12:16:52 GMT


I'm not aware of any hadoop limitations in this respect (others may be able
to comment on this); since blocks are just files on the OS, the datanode
will create subdirectories to store blocks to avoid problems with large
numbers of files in a single directory. So I would think the limitations are
primarily around the type of file system you select, for ext3 it
theoretically supports up to 16TB (http://en.wikipedia.org/wiki/Ext3) and
for ext4 up to 1EB (http://en.wikipedia.org/wiki/Ext4). Although you're
probably already planning on deploying 64-bit servers, I believe for large
FS on ext4 you'd be better off with a 64-bit server.


As far as OS is concerned anecdotally (based on blogs, hadoop mailing lists
etc) I believe there are more production deployments using RHEL and/or
CentOS than Ubuntu. 


It's probably not practical to have nodes with 1PB of data for the reasons
that others have mentioned and due to the replication traffic that will be
generated if the node dies. Not to mention fsck times with large file






From: jeba earnest [mailto:jebaearnest@yahoo.com] 
Sent: 30 January 2013 10:40
To: user@hadoop.apache.org
Subject: Re: Maximum Storage size in a Single datanode



I want to use either UBUNTU or REDHAT .

I just want to know how much storage space we can allocate in a single data


Is there any limitations in hadoop for storage in single node?







From: "Pamecha, Abhishek" <apamecha@ebay.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>; jeba earnest
Sent: Wednesday, 30 January 2013 2:45 PM
Subject: Re: Maximum Storage size in a Single datanode


What would be the reason you would do that? 


You would want to leverage distributed dataset for higher availability and
better response times.


The maximum storage depends completely on the disks  capacity of your nodes
and what your OS supports. Typically I have heard of about 1-2 TB/node to
start with, but I may be wrong.




From: jeba earnest <jebaearnest@yahoo.com>
Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>, jeba earnest
Date: Wednesday, January 30, 2013 1:38 PM
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Maximum Storage size in a Single datanode




Is it possible to keep 1 Petabyte in a single data node?

If not, How much is the maximum storage for a particular data node? 


M. Jeba


View raw message