Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8604DD9E7 for ; Fri, 9 Nov 2012 17:08:24 +0000 (UTC) Received: (qmail 43625 invoked by uid 500); 9 Nov 2012 17:08:19 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 43359 invoked by uid 500); 9 Nov 2012 17:08:19 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 43346 invoked by uid 99); 9 Nov 2012 17:08:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Nov 2012 17:08:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yinghua.hu@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Nov 2012 17:08:11 +0000 Received: by mail-ie0-f176.google.com with SMTP id k11so7063138iea.35 for ; Fri, 09 Nov 2012 09:07:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=LxovqxzcjL4OQaxAzEEEC1yaxX/94OemPeFHF2FIpKQ=; b=p2Cop9/aORkB79N6IKfkpKQ4IHebWeQzUED5du2nhe1go0yUvDSM9NdoBYCZD0zNsC 6ghSUAx8Gt3r9R0aIZ1wg8JE34PFMw0wDfNlefYwK2+VIaT0sWZ6/C6LZhWnnG77DFVw B9p8v+bkFcwJ5NDGvS3M6HmG+kgtRLaJ5l8zAX2zB0TOHYt16x/EgkNOiJwYCnFDSEAD eskTE/c0xaEwRbcG9uR/GRU3msedXFfhT2NFGdlgALWCVTR6CnuLbIhxamD4j5ISow+9 Cn1dfrf5E65lXurSZ0rDcJsGphu0vQZxuxaqJs4o7AeBUpfUrQLTpiy4dT7IlEfreAho At4w== MIME-Version: 1.0 Received: by 10.50.95.161 with SMTP id dl1mr2059298igb.0.1352480870817; Fri, 09 Nov 2012 09:07:50 -0800 (PST) Received: by 10.64.12.66 with HTTP; Fri, 9 Nov 2012 09:07:50 -0800 (PST) Date: Fri, 9 Nov 2012 09:07:50 -0800 Message-ID: Subject: Error running the pi program on a 4 node cluster From: yinghua hu To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e89a8f3b9c3fdee1a204ce12fdda X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f3b9c3fdee1a204ce12fdda Content-Type: text/plain; charset=ISO-8859-1 Hi, all Could some help looking at this problem? I am setting up a four node cluster on EC2 and seems that the cluster is set up fine until I start testing. I have tried password-less ssh from each node to all the nodes and there is no problem connecting. Any advice will be greatly appreciated! [hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi - Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -libjars share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000 Number of Maps = 16 Samples per Map = 10000 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Wrote input for Map #4 Wrote input for Map #5 Wrote input for Map #6 Wrote input for Map #7 Wrote input for Map #8 Wrote input for Map #9 Wrote input for Map #10 Wrote input for Map #11 Wrote input for Map #12 Wrote input for Map #13 Wrote input for Map #14 Wrote input for Map #15 Starting Job 12/11/09 12:02:59 INFO input.FileInputFormat: Total input paths to process : 16 12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16 12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files 12/11/09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar 12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files 12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 12/11/09 12:02:59 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 12/11/09 12:02:59 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 12/11/09 12:02:59 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 12/11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps 12/11/09 12:02:59 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 12/11/09 12:02:59 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 12/11/09 12:03:00 INFO mapred.ResourceMgrDelegate: Submitted application application_1352478937343_0002 to ResourceManager at master/ 10.12.181.233:60400 12/11/09 12:03:00 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1352478937343_0002/ 12/11/09 12:03:00 INFO mapreduce.Job: Running job: job_1352478937343_0002 12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 running in uber mode : false 12/11/09 12:03:01 INFO mapreduce.Job: map 0% reduce 0% 12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 failed with state FAILED due to: Application application_1352478937343_0002 failed 1 times due to Error launching appattempt_1352478937343_0002_000001. Got exception: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:115) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:258) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""domU-12-31-39-16-45-3B":46641; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:144) at $Proxy24.startContainer(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:104) ... 5 more Caused by: java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: ""domU-12-31-39-16-45-3B":46641; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:713) at org.apache.hadoop.ipc.Client$Connection.(Client.java:236) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1188) at org.apache.hadoop.ipc.Client.call(Client.java:1068) at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141) ... 7 more Caused by: java.net.UnknownHostException ... 11 more . Failing the application. 12/11/09 12:03:01 INFO mapreduce.Job: Counters: 0 Job Finished in 2.672 seconds java.io.FileNotFoundException: File does not exist: hdfs://master:9000/user/hduser/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:738) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1685) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1709) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) -- Regards, Yinghua --e89a8f3b9c3fdee1a204ce12fdda Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi, all

Could some help looking at this probl= em? I am setting up a four node cluster on EC2 and seems that the cluster i= s set up fine until I start testing.

I have tried = password-less ssh from each node to all the nodes and there is no problem c= onnecting. Any advice will be greatly=A0appreciated!

[hduser@ip-XX-XX-XXX-XXX hadoop]$ bin/hadoop jar share/= hadoop/mapreduce/hadoop-mapreduce-examples-0.23.4.jar pi -Dmapreduce.clientfactory.class.name=3Dorg.apache.hadoop.mapred.YarnClientFactory -libjars share/hadoop/mapre= duce/hadoop-mapreduce-client-jobclient-0.23.4.jar 16 10000
Number of Maps =A0=3D 16
Samples per Map =3D 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote in= put for Map #2
Wrote input for Map #3
Wrote input for M= ap #4
Wrote input for Map #5
Wrote input for Map #6
Wrot= e input for Map #7
Wrote input for Map #8
Wrote input f= or Map #9
Wrote input for Map #10
Wrote input for Map #= 11
Wrote input for Map #12
Wrote input for Map #13
Wr= ote input for Map #14
Wrote input for Map #15
Starting = Job
12/11/09 12:02:59 INFO input.FileInputFormat: Total input pat= hs to process : 16
12/11/09 12:02:59 INFO mapreduce.JobSubmitter: number of splits:16
12/11/09 12:02:59 WARN conf.Configuration: mapred.job.classpath.file= s is deprecated. Instead, use mapreduce.job.classpath.files
12/11= /09 12:02:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, us= e mapreduce.job.jar
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files is depre= cated. Instead, use mapreduce.job.cache.files
12/11/09 12:02:59 W= ARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecate= d. Instead, use mapreduce.map.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks is depr= ecated. Instead, use mapreduce.job.reduces
12/11/09 12:02:59 WARN= conf.Configuration: mapred.output.value.class is deprecated. Instead, use = mapreduce.job.output.value.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.reduce.tasks.specula= tive.execution is deprecated. Instead, use mapreduce.reduce.speculative
12/11/09 12:02:59 WARN conf.Configuration: mapred.used.genericoption= sparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.u= sed
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.map.class is depr= ecated. Instead, use mapreduce.job.map.class
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.reduce.class is d= eprecated. Instead, use mapreduce.job.reduce.class
12/11/09 12:02= :59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Ins= tead, use mapreduce.job.inputformat.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.input.dir is depreca= ted. Instead, use mapreduce.input.fileinputformat.inputdir
12/11/= 09 12:02:59 WARN conf.Configuration: mapred.output.dir is deprecated. Inste= ad, use mapreduce.output.fileoutputformat.outputdir
12/11/09 12:02:59 WARN conf.Configuration: mapreduce.outputformat.clas= s is deprecated. Instead, use mapreduce.job.outputformat.class
12= /11/09 12:02:59 WARN conf.Configuration: mapred.map.tasks is deprecated. In= stead, use mapreduce.job.maps
12/11/09 12:02:59 WARN conf.Configuration: mapred.cache.files.timestam= ps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
12/11/09 12:02:59 WARN conf.Configuration: mapred.output.key.class is de= precated. Instead, use mapreduce.job.output.key.class
12/11/09 12:02:59 WARN conf.Configuration: mapred.working.dir is depre= cated. Instead, use mapreduce.job.working.dir
12/11/09 12:03:00 I= NFO mapred.ResourceMgrDelegate: Submitted application application_135247893= 7343_0002 to ResourceManager at master/10.12.181.233:60400
12/11/09 12:03:00 INFO mapreduce.Job: The url to track the job: http://maste= r:8088/proxy/application_1352478937343_0002/
12/11/09 12:03:0= 0 INFO mapreduce.Job: Running job: job_1352478937343_0002
12/11/09 12:03:01 INFO mapreduce.Job: Job job_1352478937343_0002 runni= ng in uber mode : false
12/11/09 12:03:01 INFO mapreduce.Job: =A0= map 0% reduce 0%
12/11/09 12:03:01 INFO mapreduce.Job: Job job_13= 52478937343_0002 failed with state FAILED due to: Application application_1= 352478937343_0002 failed 1 times due to Error launching appattempt_13524789= 37343_0002_000001. Got exception: java.lang.reflect.UndeclaredThrowableExce= ption
=A0 =A0 =A0 =A0 at org.apache.hadoop.yarn.api.impl.pb.client.Container= ManagerPBClientImpl.startContainer(ContainerManagerPBClientImpl.java:111)
=A0 =A0 =A0 =A0 at org.apache.hadoop.yarn.server.resourcemanager.a= mlauncher.AMLauncher.launch(AMLauncher.java:115)
=A0 =A0 =A0 =A0 at org.apache.hadoop.yarn.server.resourcemanager.amlau= ncher.AMLauncher.run(AMLauncher.java:258)
=A0 =A0 =A0 =A0 at java= .util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)=
=A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worke= r.run(ThreadPoolExecutor.java:603)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(Thread.java:722)
Cau= sed by: com.google.protobuf.ServiceException: java.net.UnknownHostException= : Invalid host name: local host is: (unknown); destination host is: "&= quot;domU-12-31-39-16-45-3B":46641; java.net.UnknownHostException; For= more details see: =A0http://wiki.apache.org/hadoop/UnknownHost
=A0 =A0 =A0 =A0 at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine= $Invoker.invoke(ProtoOverHadoopRpcEngine.java:144)
=A0 =A0 =A0 = =A0 at $Proxy24.startContainer(Unknown Source)
=A0 =A0 =A0 =A0 at= org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagerPBClientImpl.sta= rtContainer(ContainerManagerPBClientImpl.java:104)
=A0 =A0 =A0 =A0 ... 5 more
Caused by: java.net.UnknownHostEx= ception: Invalid host name: local host is: (unknown); destination host is: = ""domU-12-31-39-16-45-3B":46641; java.net.UnknownHostExcepti= on; For more details see: =A0http://wiki.apache.org/hadoop/UnknownHost
=A0 =A0 =A0 =A0 at org.apache.hadoop.net.NetUtils.wrapException(NetUti= ls.java:713)
=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Client$Conn= ection.<init>(Client.java:236)
=A0 =A0 =A0 =A0 at org.apach= e.hadoop.ipc.Client.getConnection(Client.java:1188)
=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Client.call(Client.java:1068)=
=A0 =A0 =A0 =A0 at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpc= Engine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:141)
=A0 =A0 = =A0 =A0 ... 7 more
Caused by: java.net.UnknownHostException
=A0 =A0 =A0 =A0 ...= 11 more
. Failing the application.
12/11/09 12:03:01 I= NFO mapreduce.Job: Counters: 0
Job Finished in 2.672 seconds
java.io.FileNotFoundException: File does not exist: hdfs://master:9000/user= /hduser/QuasiMonteCarlo_TMP_3_141592654/out/reduce-out
=A0 =A0 = =A0 =A0 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(Distr= ibutedFileSystem.java:738)
=A0 =A0 =A0 =A0 at org.apache.hadoop.io.SequenceFile$Reader.<init&g= t;(SequenceFile.java:1685)
=A0 =A0 =A0 =A0 at org.apache.hadoop.i= o.SequenceFile$Reader.<init>(SequenceFile.java:1709)
=A0 = =A0 =A0 =A0 at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiM= onteCarlo.java:314)
=A0 =A0 =A0 =A0 at org.apache.hadoop.examples.QuasiMonteCarlo.run(Quas= iMonteCarlo.java:351)
=A0 =A0 =A0 =A0 at org.apache.hadoop.util.T= oolRunner.run(ToolRunner.java:69)
=A0 =A0 =A0 =A0 at org.apache.h= adoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
=A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native= Method)
=A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAccessorImpl.= invoke(NativeMethodAccessorImpl.java:57)
=A0 =A0 =A0 =A0 at sun.r= eflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav= a:43)
=A0 =A0 =A0 =A0 at java.lang.reflect.Method.invoke(Method.java:601)
=A0 =A0 =A0 =A0 at org.apache.hadoop.util.ProgramDriver$ProgramDesc= ription.invoke(ProgramDriver.java:72)
=A0 =A0 =A0 =A0 at org.apac= he.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
=A0 =A0 =A0 =A0 at org.apache.hadoop.examples.ExampleDriver.main(Examp= leDriver.java:68)
=A0 =A0 =A0 =A0 at sun.reflect.NativeMethodAcce= ssorImpl.invoke0(Native Method)
=A0 =A0 =A0 =A0 at sun.reflect.Na= tiveMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
=A0 =A0 =A0 =A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Del= egatingMethodAccessorImpl.java:43)
=A0 =A0 =A0 =A0 at java.lang.r= eflect.Method.invoke(Method.java:601)
=A0 =A0 =A0 =A0 at org.apac= he.hadoop.util.RunJar.main(RunJar.java:208)


--
Regards,

Yinghua
--e89a8f3b9c3fdee1a204ce12fdda--