hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Virajith Jalaparti <virajit...@gmail.com>
Subject How is reduce completion % calculated?
Date Wed, 08 Jun 2011 14:09:41 GMT
Hi,

I am trying to figure out how the reduce progress for a job is calculated. I
was looking at the syslog generated by my job run and it looks like the
reducers start before the mappers complete. I figured this was the case
because even when the Map had <100% completion, the reduce completion % was
greater than 0. So, my question is what does a reduce phase completion of,
say,  10% actually mean?
I know that the shuffle, sort and (actual) reduce each contribute 0.33 to
the completion of a single reducers and then the % completion of the MR
phases of the job are calculated in in JobInProgress.java as (map
*or*reduce) progress = current_progress +
deltaprogress/(number_of_maps
*or *number of reducers) . So given that the number of reducers are small
and fixed (=3 in my case) and that each reducers has only 3 stages of
completion (0.33,0.67,1), I would expect this value of progress to increase
only by discrete values (of 100/(3*number of reducers)) and not in a
continuous manner. I see that my syslog (below) gives all sorts of %'s as
the completion.

Thanks,
Virajith

==============================
=============================================
SYSLOG for a job with 80 mappers and 3 reducers:
===========================================================================
2011-06-04 07:13:22,378 INFO org.apache.hadoop.mapred.JobClient (main):  map
0% reduce 0%
2011-06-04 07:13:46,499 INFO org.apache.hadoop.mapred.JobClient (main):  map
1% reduce 0%
2011-06-04 07:13:49,506 INFO org.apache.hadoop.mapred.JobClient (main):  map
3% reduce 0%
2011-06-04 07:13:52,511 INFO org.apache.hadoop.mapred.JobClient (main):  map
4% reduce 0%
2011-06-04 07:13:55,519 INFO org.apache.hadoop.mapred.JobClient (main):  map
6% reduce 0%
2011-06-04 07:13:58,526 INFO org.apache.hadoop.mapred.JobClient (main):  map
7% reduce 0%
2011-06-04 07:14:01,536 INFO org.apache.hadoop.mapred.JobClient (main):  map
8% reduce 0%
2011-06-04 07:14:04,544 INFO org.apache.hadoop.mapred.JobClient (main):  map
9% reduce 0%
2011-06-04 07:14:10,561 INFO org.apache.hadoop.mapred.JobClient (main):  map
10% reduce 0%
2011-06-04 07:14:13,567 INFO org.apache.hadoop.mapred.JobClient (main):  map
11% reduce 0%
2011-06-04 07:14:19,579 INFO org.apache.hadoop.mapred.JobClient (main):  map
12% reduce 0%
2011-06-04 07:14:22,585 INFO org.apache.hadoop.mapred.JobClient (main):  map
13% reduce 0%
2011-06-04 07:14:25,591 INFO org.apache.hadoop.mapred.JobClient (main):  map
14% reduce 0%
2011-06-04 07:14:28,604 INFO org.apache.hadoop.mapred.JobClient (main):  map
15% reduce 0%
2011-06-04 07:14:31,610 INFO org.apache.hadoop.mapred.JobClient (main):  map
16% reduce 0%
2011-06-04 07:14:34,616 INFO org.apache.hadoop.mapred.JobClient (main):  map
17% reduce 1%
2011-06-04 07:14:40,629 INFO org.apache.hadoop.mapred.JobClient (main):  map
18% reduce 1%
2011-06-04 07:14:43,634 INFO org.apache.hadoop.mapred.JobClient (main):  map
19% reduce 1%
2011-06-04 07:14:44,636 INFO org.apache.hadoop.mapred.JobClient (main):  map
19% reduce 2%
2011-06-04 07:14:49,646 INFO org.apache.hadoop.mapred.JobClient (main):  map
20% reduce 2%
2011-06-04 07:14:50,648 INFO org.apache.hadoop.mapred.JobClient (main):  map
20% reduce 3%
2011-06-04 07:14:59,671 INFO org.apache.hadoop.mapred.JobClient (main):  map
21% reduce 4%
2011-06-04 07:15:04,681 INFO org.apache.hadoop.mapred.JobClient (main):  map
22% reduce 5%
2011-06-04 07:15:07,688 INFO org.apache.hadoop.mapred.JobClient (main):  map
23% reduce 5%
2011-06-04 07:15:10,693 INFO org.apache.hadoop.mapred.JobClient (main):  map
24% reduce 5%
2011-06-04 07:15:11,696 INFO org.apache.hadoop.mapred.JobClient (main):  map
24% reduce 6%
2011-06-04 07:15:13,700 INFO org.apache.hadoop.mapred.JobClient (main):  map
25% reduce 6%
2011-06-04 07:15:14,703 INFO org.apache.hadoop.mapred.JobClient (main):  map
26% reduce 6%
2011-06-04 07:15:16,706 INFO org.apache.hadoop.mapred.JobClient (main):  map
27% reduce 6%
2011-06-04 07:15:17,709 INFO org.apache.hadoop.mapred.JobClient (main):  map
28% reduce 6%
2011-06-04 07:15:20,717 INFO org.apache.hadoop.mapred.JobClient (main):  map
29% reduce 6%
2011-06-04 07:15:29,736 INFO org.apache.hadoop.mapred.JobClient (main):  map
30% reduce 6%
2011-06-04 07:15:41,762 INFO org.apache.hadoop.mapred.JobClient (main):  map
31% reduce 6%
2011-06-04 07:15:43,770 INFO org.apache.hadoop.mapred.JobClient (main):  map
32% reduce 7%
2011-06-04 07:15:46,785 INFO org.apache.hadoop.mapred.JobClient (main):  map
33% reduce 7%
2011-06-04 07:15:47,787 INFO org.apache.hadoop.mapred.JobClient (main):  map
33% reduce 8%
2011-06-04 07:15:49,791 INFO org.apache.hadoop.mapred.JobClient (main):  map
34% reduce 8%
2011-06-04 07:15:50,793 INFO org.apache.hadoop.mapred.JobClient (main):  map
35% reduce 8%
2011-06-04 07:15:52,798 INFO org.apache.hadoop.mapred.JobClient (main):  map
36% reduce 8%
2011-06-04 07:15:55,803 INFO org.apache.hadoop.mapred.JobClient (main):  map
37% reduce 8%
2011-06-04 07:15:58,809 INFO org.apache.hadoop.mapred.JobClient (main):  map
38% reduce 8%
2011-06-04 07:16:04,821 INFO org.apache.hadoop.mapred.JobClient (main):  map
38% reduce 9%
2011-06-04 07:16:07,826 INFO org.apache.hadoop.mapred.JobClient (main):  map
39% reduce 9%
2011-06-04 07:16:10,833 INFO org.apache.hadoop.mapred.JobClient (main): Task
Id : attempt_201106040712_0001_m_000024_0, Status : FAILED
2011-06-04 07:16:12,086 INFO org.apache.hadoop.mapred.JobClient (main):  map
38% reduce 9%
2011-06-04 07:16:14,090 INFO org.apache.hadoop.mapred.JobClient (main): Task
Id : attempt_201106040712_0001_m_000027_0, Status : FAILED
2011-06-04 07:16:15,134 INFO org.apache.hadoop.mapred.JobClient (main):  map
37% reduce 9%
2011-06-04 07:16:26,162 INFO org.apache.hadoop.mapred.JobClient (main):  map
38% reduce 9%
2011-06-04 07:16:30,169 INFO org.apache.hadoop.mapred.JobClient (main):  map
39% reduce 9%
2011-06-04 07:16:33,177 INFO org.apache.hadoop.mapred.JobClient (main):  map
40% reduce 9%
2011-06-04 07:16:36,184 INFO org.apache.hadoop.mapred.JobClient (main):  map
41% reduce 9%
2011-06-04 07:16:42,199 INFO org.apache.hadoop.mapred.JobClient (main):  map
42% reduce 9%
2011-06-04 07:17:07,258 INFO org.apache.hadoop.mapred.JobClient (main):  map
42% reduce 10%
2011-06-04 07:17:15,274 INFO org.apache.hadoop.mapred.JobClient (main):  map
43% reduce 10%
2011-06-04 07:17:16,276 INFO org.apache.hadoop.mapred.JobClient (main):  map
44% reduce 10%
2011-06-04 07:17:18,283 INFO org.apache.hadoop.mapred.JobClient (main):  map
45% reduce 10%
2011-06-04 07:17:19,285 INFO org.apache.hadoop.mapred.JobClient (main):  map
46% reduce 10%
2011-06-04 07:17:22,290 INFO org.apache.hadoop.mapred.JobClient (main):  map
47% reduce 10%
2011-06-04 07:17:24,294 INFO org.apache.hadoop.mapred.JobClient (main):  map
48% reduce 10%
2011-06-04 07:17:25,296 INFO org.apache.hadoop.mapred.JobClient (main):  map
49% reduce 10%
2011-06-04 07:17:28,303 INFO org.apache.hadoop.mapred.JobClient (main):  map
50% reduce 11%
2011-06-04 07:17:33,318 INFO org.apache.hadoop.mapred.JobClient (main):  map
51% reduce 11%
2011-06-04 07:17:42,337 INFO org.apache.hadoop.mapred.JobClient (main):  map
52% reduce 11%
2011-06-04 07:17:46,345 INFO org.apache.hadoop.mapred.JobClient (main):  map
53% reduce 11%
2011-06-04 07:17:49,352 INFO org.apache.hadoop.mapred.JobClient (main):  map
54% reduce 11%
2011-06-04 07:17:52,463 INFO org.apache.hadoop.mapred.JobClient (main):  map
55% reduce 11%
2011-06-04 07:17:55,469 INFO org.apache.hadoop.mapred.JobClient (main):  map
56% reduce 11%
2011-06-04 07:17:58,476 INFO org.apache.hadoop.mapred.JobClient (main):  map
57% reduce 11%
2011-06-04 07:18:17,525 INFO org.apache.hadoop.mapred.JobClient (main):  map
57% reduce 12%
2011-06-04 07:18:19,528 INFO org.apache.hadoop.mapred.JobClient (main):  map
58% reduce 12%
2011-06-04 07:18:22,535 INFO org.apache.hadoop.mapred.JobClient (main):  map
59% reduce 12%
2011-06-04 07:18:23,537 INFO org.apache.hadoop.mapred.JobClient (main):  map
60% reduce 12%
2011-06-04 07:18:25,541 INFO org.apache.hadoop.mapred.JobClient (main):  map
60% reduce 13%
2011-06-04 07:18:26,542 INFO org.apache.hadoop.mapred.JobClient (main):  map
61% reduce 13%
2011-06-04 07:18:29,548 INFO org.apache.hadoop.mapred.JobClient (main):  map
63% reduce 13%
2011-06-04 07:18:32,554 INFO org.apache.hadoop.mapred.JobClient (main):  map
64% reduce 13%
2011-06-04 07:18:35,560 INFO org.apache.hadoop.mapred.JobClient (main):  map
65% reduce 13%
2011-06-04 07:18:38,566 INFO org.apache.hadoop.mapred.JobClient (main):  map
66% reduce 14%
2011-06-04 07:18:44,580 INFO org.apache.hadoop.mapred.JobClient (main):  map
67% reduce 14%
2011-06-04 07:18:47,586 INFO org.apache.hadoop.mapred.JobClient (main):  map
67% reduce 15%
2011-06-04 07:18:50,593 INFO org.apache.hadoop.mapred.JobClient (main):  map
68% reduce 15%
2011-06-04 07:18:53,601 INFO org.apache.hadoop.mapred.JobClient (main):  map
69% reduce 15%
2011-06-04 07:18:59,614 INFO org.apache.hadoop.mapred.JobClient (main):  map
70% reduce 15%
2011-06-04 07:19:02,621 INFO org.apache.hadoop.mapred.JobClient (main):  map
71% reduce 15%
2011-06-04 07:19:08,633 INFO org.apache.hadoop.mapred.JobClient (main):  map
72% reduce 15%
2011-06-04 07:19:16,653 INFO org.apache.hadoop.mapred.JobClient (main):  map
73% reduce 15%
2011-06-04 07:19:19,660 INFO org.apache.hadoop.mapred.JobClient (main):  map
74% reduce 15%
2011-06-04 07:19:25,673 INFO org.apache.hadoop.mapred.JobClient (main):  map
75% reduce 15%
2011-06-04 07:19:28,680 INFO org.apache.hadoop.mapred.JobClient (main):  map
76% reduce 15%
2011-06-04 07:19:30,684 INFO org.apache.hadoop.mapred.JobClient (main):  map
76% reduce 16%
2011-06-04 07:19:36,699 INFO org.apache.hadoop.mapred.JobClient (main):  map
77% reduce 16%
2011-06-04 07:19:39,706 INFO org.apache.hadoop.mapred.JobClient (main):  map
78% reduce 16%
2011-06-04 07:19:42,714 INFO org.apache.hadoop.mapred.JobClient (main):  map
79% reduce 16%
2011-06-04 07:19:48,730 INFO org.apache.hadoop.mapred.JobClient (main):  map
80% reduce 17%
2011-06-04 07:19:51,736 INFO org.apache.hadoop.mapred.JobClient (main):  map
81% reduce 17%
2011-06-04 07:19:58,750 INFO org.apache.hadoop.mapred.JobClient (main):  map
82% reduce 17%
2011-06-04 07:20:01,757 INFO org.apache.hadoop.mapred.JobClient (main):  map
83% reduce 18%
2011-06-04 07:20:04,764 INFO org.apache.hadoop.mapred.JobClient (main):  map
84% reduce 18%
2011-06-04 07:20:09,775 INFO org.apache.hadoop.mapred.JobClient (main):  map
85% reduce 18%
2011-06-04 07:20:12,785 INFO org.apache.hadoop.mapred.JobClient (main):  map
86% reduce 18%
2011-06-04 07:20:15,791 INFO org.apache.hadoop.mapred.JobClient (main):  map
87% reduce 19%
2011-06-04 07:20:18,796 INFO org.apache.hadoop.mapred.JobClient (main):  map
88% reduce 19%
2011-06-04 07:20:21,803 INFO org.apache.hadoop.mapred.JobClient (main):  map
89% reduce 19%
2011-06-04 07:20:24,810 INFO org.apache.hadoop.mapred.JobClient (main):  map
90% reduce 19%
2011-06-04 07:20:29,820 INFO org.apache.hadoop.mapred.JobClient (main): Task
Id : attempt_201106040712_0001_m_000069_0, Status : FAILED
2011-06-04 07:20:31,270 INFO org.apache.hadoop.mapred.JobClient (main):  map
89% reduce 19%
2011-06-04 07:20:37,284 INFO org.apache.hadoop.mapred.JobClient (main):  map
90% reduce 19%
2011-06-04 07:20:46,303 INFO org.apache.hadoop.mapred.JobClient (main):  map
91% reduce 20%
2011-06-04 07:20:49,310 INFO org.apache.hadoop.mapred.JobClient (main):  map
92% reduce 20%
2011-06-04 07:20:52,315 INFO org.apache.hadoop.mapred.JobClient (main):  map
93% reduce 20%
2011-06-04 07:20:55,323 INFO org.apache.hadoop.mapred.JobClient (main):  map
94% reduce 20%
2011-06-04 07:20:58,330 INFO org.apache.hadoop.mapred.JobClient (main):  map
95% reduce 20%
2011-06-04 07:20:59,333 INFO org.apache.hadoop.mapred.JobClient (main): Task
Id : attempt_201106040712_0001_m_000073_0, Status : FAILED
2011-06-04 07:21:00,347 INFO org.apache.hadoop.mapred.JobClient (main):  map
94% reduce 20%
2011-06-04 07:21:04,354 INFO org.apache.hadoop.mapred.JobClient (main):  map
95% reduce 20%
2011-06-04 07:21:13,377 INFO org.apache.hadoop.mapred.JobClient (main):  map
96% reduce 20%
2011-06-04 07:21:19,390 INFO org.apache.hadoop.mapred.JobClient (main):  map
98% reduce 20%
2011-06-04 07:21:25,401 INFO org.apache.hadoop.mapred.JobClient (main):  map
99% reduce 21%
2011-06-04 07:21:40,441 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 21%
2011-06-04 07:22:22,539 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 33%
2011-06-04 07:22:31,570 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 35%
2011-06-04 07:22:35,579 INFO org.apache.hadoop.mapred.JobClient (main): Task
Id : attempt_201106040712_0001_r_000000_0, Status : FAILED
2011-06-04 07:22:36,659 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 29%
2011-06-04 07:22:40,668 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 30%
2011-06-04 07:22:45,679 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 31%
2011-06-04 07:22:51,690 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 32%
2011-06-04 07:22:55,700 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 33%
2011-06-04 07:23:01,714 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 34%
2011-06-04 07:23:10,733 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 35%
2011-06-04 07:23:26,766 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 36%
2011-06-04 07:23:32,778 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 37%
2011-06-04 07:23:52,820 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 38%
2011-06-04 07:23:56,827 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 39%
2011-06-04 07:24:05,848 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 40%
2011-06-04 07:24:16,871 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 41%
2011-06-04 07:24:34,908 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 42%
2011-06-04 07:24:48,938 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 43%
2011-06-04 07:24:54,953 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 44%
2011-06-04 07:25:06,976 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 45%
2011-06-04 07:25:07,979 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 56%
2011-06-04 07:25:15,996 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 57%
2011-06-04 07:25:26,034 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 58%
2011-06-04 07:25:38,058 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 59%
2011-06-04 07:25:47,076 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 60%
2011-06-04 07:25:50,082 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 61%
2011-06-04 07:25:59,099 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 62%
2011-06-04 07:26:02,106 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 63%
2011-06-04 07:26:11,135 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 64%
2011-06-04 07:26:17,146 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 65%
2011-06-04 07:26:29,170 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 66%
2011-06-04 07:26:41,195 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 67%
2011-06-04 07:26:47,209 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 68%
2011-06-04 07:26:41,195 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 67%
2011-06-04 07:26:47,209 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 68%
2011-06-04 07:26:50,215 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 69%
2011-06-04 07:26:59,242 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 70%
2011-06-04 07:27:08,276 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 71%
2011-06-04 07:27:20,301 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 72%
2011-06-04 07:27:26,318 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 73%
2011-06-04 07:27:29,324 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 84%
2011-06-04 07:27:32,329 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 85%
2011-06-04 07:27:38,342 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 86%
2011-06-04 07:27:47,363 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 87%
2011-06-04 07:27:50,369 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 88%
2011-06-04 07:27:59,390 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 89%
2011-06-04 07:28:08,419 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 90%
2011-06-04 07:28:21,447 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 91%
2011-06-04 07:28:33,471 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 92%
2011-06-04 07:28:48,501 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 93%
2011-06-04 07:29:00,532 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 94%
2011-06-04 07:29:15,560 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 95%
2011-06-04 07:29:30,592 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 96%
2011-06-04 07:29:45,625 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 97%
2011-06-04 07:30:00,655 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 98%
2011-06-04 07:30:18,693 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 99%
2011-06-04 07:30:30,724 INFO org.apache.hadoop.mapred.JobClient (main):  map
100% reduce 100%

Mime
View raw message