hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shrinivas Joshi <jshrini...@gmail.com>
Subject io.sort.mb based on HDFS block size
Date Tue, 12 Apr 2011 19:25:40 GMT
Looking at workloads like TeraSort where intermediate map output is
proportional to HDFS block size, I was wondering whether it would be
beneficial to have a mechanism for setting buffer spaces like io.sort.mb to
be a certain factor of HDFS block size? I am sure there are other config
parameters that could benefit from such expression type values.

Please let me know your thoughts on this.

Thanks,
-Shrinivas

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message