hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish Mallipeddi <harish.mallipe...@gmail.com>
Subject Difference between "Killed Task Attempts" and "Killed Tasks"
Date Mon, 03 Aug 2009 12:50:14 GMT
Hi,

Anyone can tell me what's the difference between "Killed Task Attempts" and
"Killed Tasks"? I ran a big job (14820 maps and 0 reduces). In the
job-details page, the web GUI reports 62 "killed task attempts". I'm
assuming this is due to "speculative execution". Now when I go to the
job-history page for the job, it reports 54 "killed tasks" (and 14820
successful map-tasks as expected).

A few questions:

* Why 62 killed task attempts vs 54 killed tasks?
* Under speculative execution, does hadoop launch a new MapTask with new
task-id or does it just launch a new MapTaskAttempt with a new
task-attempt-id?
* When a MapTaskAttempt fails, and when hadoop tries to re-launch the
MapTask, does it create a new task-id or just a new task-attempt-id?
* Does 'mapred.map.max.attempts' include all attempts launched due to
speculative-execution?

Btw this job is basically a trivial no-op job - it just scans around 1TB of
data and does nothing else in the map. I looked at the killed tasks' syslog
output and I didn't see any errors.

-- 
Harish Mallipeddi
http://blog.poundbang.in

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message