hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Optimal Filesystem (and Settings) for HDFS
Date Wed, 20 May 2009 13:07:01 GMT
Bryan Duxbury wrote:
> We use XFS for our data drives, and we've had somewhat mixed results. 


Thanks for that. I've just created a wiki page to put some of these 
notes up -extensions and some hard data would be welcome

http://wiki.apache.org/hadoop/DiskSetup

One problem we have for hard data is that we need some different 
benchmarks for MR jobs. Terasort is good for measuring IO and MR 
framework performance, but for more CPU intensive algorithms, or things 
that need to seek round a bit more, you can't be sure that terasort 
benchmarks are a good predictor of what's right for you in terms of 
hardware, filesystem, etc.

Contributions in this area would be welcome.

I'd like to measure the power consumed on a run too, which is actually 
possible as far as my laptop is concerned, because you can ask it's 
battery what happened.

-steve

Mime
View raw message