hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haijun Cao" <hai...@kindsight.net>
Subject local bytes written (high io, low memory usage)
Date Fri, 06 Jun 2008 05:08:48 GMT
I noticed that "local bytes written/read" stat in my map reduce job is
really high, 2x, 3x, 4x of the hdfs bytes.  


When does hadoop mapred framework write to local fs? Is it done when the
jvm memory is not enough and data is spill to disk? how I can configure
so that it does not spill to disk? My memory usage is really low, while
io is nearly saturated.





  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message