hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Maier <wcma...@hep.wisc.edu>
Subject Re: our experiences with various filesystems and tuning options
Date Tue, 10 May 2011 10:30:08 GMT
On Tue, May 10, 2011 at 12:03:09AM -0400, Rita wrote:
> what filesystem are they using and what is the size of each filesystem?

It sounds nuts, but each disk has its own ext3 filesystem. Beyond switching to
the deadline IO scheduler, we haven't done much tuning/tweaking. A script runs
every ten minutes to test all of the data mounts and reconfigure hdfs-site.xml
and restart the datanode if necessary. So far, this approach has allowed us to
avoid loss of space to RAID without correlating the risk of disk failure by
building larger RAID0s.

In the future, we expect to deprecate the script and rely on the datanode process
itself to handle missing/failing disks.


Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/

View raw message