hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stone <stones....@gmail.com>
Subject Re: Hadoop in Pseudo-Distributed mode on Mac OS X 10.8
Date Thu, 30 Aug 2012 17:24:04 GMT
I got the same issue today. map tasks finished quickly but reduce is always
0%. I am also running Mac OS X 10.8. (cdh3u4)

12/08/31 01:13:03 INFO mapred.JobClient:  map 0% reduce 0%
12/08/31 01:13:07 INFO mapred.JobClient:  map 100% reduce 0%
12/08/31 01:23:14 INFO mapred.JobClient: Task Id :
attempt_201208310112_0001_r_000000_0, Status : FAILED
Task attempt_201208310112_0001_r_000000_0 failed to report status for 600
seconds. Killing!



logs for the reducer :

Task Logs: 'attempt_201208310112_0001_r_000000_0'



stdout logs


stderr logs
2012-08-31 01:13:06.316 java[46834:1203] Unable to load realm info from
SCDynamicStore


syslog logs
2012-08-31 01:13:06,421 INFO
org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
set up for Hadoop, not re-installing.
2012-08-31 01:13:06,674 WARN org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2012-08-31 01:13:06,848 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2012-08-31 01:13:06,945 INFO org.apache.hadoop.mapred.Task:  Using
ResourceCalculatorPlugin : null
2012-08-31 01:13:06,957 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=144965632, MaxSingleShuffleLimit=36241408
2012-08-31 01:13:06,962 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread waiting: Thread for merging
on-disk files
2012-08-31 01:13:06,963 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for merging in
memory files
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Thread started: Thread for polling Map
Completion Events
2012-08-31 01:13:06,964 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 0
is already in progress
2012-08-31 01:13:06,965 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:13:11,966 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:14:07,996 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:15:08,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:16:08,069 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:16:08,070 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:17:08,106 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:17:08,107 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Need another 1 map output(s) where 1
is already in progress
2012-08-31 01:18:08,147 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_201208310112_0001_r_000000_0 Scheduled 0 outputs (0 slow hosts and0
dup hosts)

I can successfully run the wordcount example on my old macbook (os x 10.6)
previously.

Any suggestions ?

Best Regards,
Stone



On Mon, Aug 13, 2012 at 12:21 PM, Subho Banerjee <subs.zero@gmail.com>wrote:

> Hello,
>
> I am running hadoop v1.0.3 in Mac OS X 10.8 with Java_1.6.0_33-b03-424
>
>
> When running hadoop on pseudo-distributed mode, the map seems to work, but
> it cannot compute the reduce.
>
> 12/08/13 08:58:12 INFO mapred.JobClient: Running job: job_201208130857_0001
> 12/08/13 08:58:13 INFO mapred.JobClient: map 0% reduce 0%
> 12/08/13 08:58:27 INFO mapred.JobClient: map 20% reduce 0%
> 12/08/13 08:58:33 INFO mapred.JobClient: map 30% reduce 0%
> 12/08/13 08:58:36 INFO mapred.JobClient: map 40% reduce 0%
> 12/08/13 08:58:39 INFO mapred.JobClient: map 50% reduce 0%
> 12/08/13 08:58:42 INFO mapred.JobClient: map 60% reduce 0%
> 12/08/13 08:58:45 INFO mapred.JobClient: map 70% reduce 0%
> 12/08/13 08:58:48 INFO mapred.JobClient: map 80% reduce 0%
> 12/08/13 08:58:51 INFO mapred.JobClient: map 90% reduce 0%
> 12/08/13 08:58:54 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 08:59:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000000_0, Status : FAILED
> Too many fetch-failures
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
> 12/08/13 08:59:14 WARN mapred.JobClient: Error reading task outputServer
> returned HTTP response code: 403 for URL:
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
> 12/08/13 08:59:18 INFO mapred.JobClient: map 89% reduce 0%
> 12/08/13 08:59:21 INFO mapred.JobClient: map 100% reduce 0%
> 12/08/13 09:00:14 INFO mapred.JobClient: Task Id :
> attempt_201208130857_0001_m_000001_0, Status : FAILED
> Too many fetch-failures
>
> Here is what I get when I try to see the tasklog using the links given in
> the output
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stderr
>  --->
> 2012-08-13 08:58:39.189 java[74092:1203] Unable to load realm info from
> SCDynamicStore
>
>
> http://10.1.66.17:50060/tasklog?plaintext=true&attemptid=attempt_201208130857_0001_m_000000_0&filter=stdout
>  --->
>
> I have changed my hadoop-env.sh acoording to Mathew Buckett in
> https://issues.apache.org/jira/browse/HADOOP-7489
>
> Also this error of Unable to load realm info from SCDynamicStore does not
> show up when I do 'hadoop namenode -format' or 'start-all.sh'
>
> I am also attaching a zipped copy of my logs
>
>
> Cheers,
>
> Subho.
>

Mime
View raw message