hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: how does hadoop handle the counter of the failed task and speculative task
Date Sun, 26 Dec 2010 09:58:18 GMT
Hi,

On Sun, Dec 26, 2010 at 11:33 AM, lei liu <liulei412@gmail.com> wrote:
> But I have two questions about the counter:
> 1. If the map task is retried 4 times, and the last map task is successful,
> I think the counter of the other three map tasks should be not included in
> ultima result, is that right?

Yes.

> 2. If there are speculative tasks,  I think the counter of speculative tasks
> should be not included in ultima result, is that right?

Yes.

A Job's counters are calculated via the sum of all TaskInProgress-es
(TIPs) under it ("It" being the JIP - JobInProgress). So the end
result would be only the successful ones as the counters are
recomputed for all changes in the TIP's status. During execution, it
would project the best available counter (as in, from the task attempt
that has the best overall progress compared to others).

-- 
Harsh J
www.harshj.com

Mime
View raw message