hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen, Richard" <richard.c...@intel.com>
Subject RE: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example
Date Fri, 21 Feb 2014 06:42:38 GMT
Thanks for the reply!

I intended to run the job on a cluster of 5 physical nodes.
Please find attached my /etc/hadoop configuration directory (*.sh was renamed as *.sh_ to
avoid mail fitering)

As you can see the configurations are pointing to samdev06 host where ResourceManager is deployed.
I don’t know why it seems to run in a local mode.

I referenced to multiple sources for the configuration so I can’t give you one such source.


From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] On Behalf Of Vinod Kumar Vavilapalli
Sent: Friday, February 21, 2014 1:37 PM
To: user@hadoop.apache.org
Subject: Re: hadoop-3.0.0-SNAPSHOT runtime exception when trying PI example

Are you trying to run it on a cluster or locally? There is something called 'local mode' which
is getting activated here (see that job-id has a _local_ string).

You need to point your configuration to the real ResourceManager. Share your configs. Or better
share the resource that you are following as instructions.


On Feb 19, 2014, at 10:09 PM, Chen, Richard <richard.chen@intel.com<mailto:richard.chen@intel.com>>

Dear Hadoop User group,

This is my virgin post. So please bear with me if I made any mistakes at my early time here.

Recently I successfully downloaded and compiled Hadoop trunk code (svn co http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk)
I also deployed the hadoop-3.0.0-SNAPSHOT distribution to a physical cluster of 5 nodes each
running 64-bit Cent OS 6.

After careful configuration, when I tried running the 1st PI job example:
 $hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar pi 4 10

The result was successfully calculated but some exception message was also dumped during the
execution and I couldn’t figure out why.
The exception message goes like this:

Starting Job
14/02/20 13:34:12 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/02/20 13:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker,
14/02/20 13:34:12 INFO input.FileInputFormat: Total input paths to process : 4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: number of splits:4
14/02/20 13:34:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/02/20 13:34:12 INFO mapreduce.Job: Running job: job_local1598491292_0001
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/02/20 13:34:12 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/02/20 13:34:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1598491292_0001_m_000000_0
14/02/20 13:34:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/02/20 13:34:13 INFO mapred.MapTask: Processing split:hdfs://samdev06:9000/user/hadoop/QuasiMonteCarlo_1392874450186_1650772770/in/part0:0+118
14/02/20 13:34:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/02/20 13:34:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/02/20 13:34:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/02/20 13:34:13 INFO mapred.MapTask: soft limit at 83886080
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/02/20 13:34:13 INFO mapred.LocalJobRunner:
14/02/20 13:34:13 INFO mapred.MapTask: Starting flush of map output
14/02/20 13:34:13 INFO mapred.MapTask: Spilling map output
14/02/20 13:34:13 INFO mapred.MapTask: bufstart = 0; bufend = 18; bufvoid = 104857600
14/02/20 13:34:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568);
length = 5/6553600
14/02/20 13:34:13 INFO mapred.MapTask: Finished spill 0
14/02/20 13:34:13 INFO mapred.Task: Task:attempt_local1598491292_0001_m_000000_0 is done.
And is in the process of committing
14/02/20 13:34:13 WARN mapred.Task: Could not find output size
java.io.FileNotFoundException: File does not exist: /tmp/hadoop/mapred/local/localRunner/hadoop/jobcache/job_local1598491292_0001/attempt_local1598491292_0001_m_000000_0/output/file.out
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
                at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
                at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1124)
                at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1102)
                at org.apache.hadoop.mapred.Task.done(Task.java:1048)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
                at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:744)

Please kindly help me on this.

Thanks in advance!


NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You.
View raw message