hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject spill taking too long
Date Thu, 04 Nov 2010 18:36:02 GMT
Hi,
We use cdh3b2.
We sometimes see maptask timeout. Here is log from one of the maptasks:

2010-11-04 10:34:23,820 INFO org.apache.hadoop.mapred.MapTask:
Spilling map output: buffer full= true
2010-11-04 10:34:23,820 INFO org.apache.hadoop.mapred.MapTask:
bufstart = 119534169; bufend = 59763857; bufvoid = 298844160
2010-11-04 10:34:23,820 INFO org.apache.hadoop.mapred.MapTask: kvstart
= 438913; kvend = 585320; length = 983040
2010-11-04 10:34:41,615 INFO org.apache.hadoop.mapred.MapTask: Finished spill 3
2010-11-04 10:35:45,352 INFO org.apache.hadoop.mapred.MapTask:
Spilling map output: buffer full= true
2010-11-04 10:35:45,547 INFO org.apache.hadoop.mapred.MapTask:
bufstart = 59763857; bufend = 298837899; bufvoid = 298844160
2010-11-04 10:35:45,547 INFO org.apache.hadoop.mapred.MapTask: kvstart
= 585320; kvend = 731585; length = 983040
2010-11-04 10:45:41,289 INFO org.apache.hadoop.mapred.MapTask: Finished spill 4

Note how long the last spill took.

Can someone provide hint on what might be the reason ?

Thanks

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message