hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Macdonald <cra...@dcs.gla.ac.uk>
Subject best practice: mapred.local vs dfs drives
Date Fri, 03 Apr 2009 12:36:54 GMT
Hello all,

Following recent hardware discussions, I thought I'd ask a related 
question. Our cluster nodes have 3 drives: 1x 160GB system/scratch and 
2x 500GB DFS drives.

The 160GB system drive is partitioned such that 100GB is for job 
mapred.local space. However, we find that for our application, 
mapred.local free space for map output space is the limiting parameter 
on the number of reducers we can have (our application prefers less 
reducers).

How do people normally work for dfs vs mapred.local space. Do you (a) 
share the DFS drives with the task tracker temporary files, Or do you 
(b) keep them on separate partitions or drives?

We originally went with (b) because it prevented a run-away job from 
eating all the DFS space on the machine, however, I'm beginning to 
realise the disadvantages.

Any comments?

Thanks

Craig


Mime
View raw message