Return-Path: X-Original-To: apmail-bigtop-user-archive@www.apache.org Delivered-To: apmail-bigtop-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E57DB11D5E for ; Wed, 14 May 2014 04:30:08 +0000 (UTC) Received: (qmail 69610 invoked by uid 500); 14 May 2014 00:30:08 -0000 Delivered-To: apmail-bigtop-user-archive@bigtop.apache.org Received: (qmail 69542 invoked by uid 500); 14 May 2014 00:30:08 -0000 Mailing-List: contact user-help@bigtop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@bigtop.apache.org Delivered-To: mailing list user@bigtop.apache.org Received: (qmail 69534 invoked by uid 99); 14 May 2014 00:30:08 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 May 2014 00:30:08 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jayunit100@gmail.com designates 209.85.192.48 as permitted sender) Received: from [209.85.192.48] (HELO mail-qg0-f48.google.com) (209.85.192.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 May 2014 00:30:03 +0000 Received: by mail-qg0-f48.google.com with SMTP id i50so1640119qgf.35 for ; Tue, 13 May 2014 17:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:mime-version:subject :message-id:date:references:in-reply-to:to; bh=p74UfxUtXP5wQPh/2Cr3mglhJxY9+0bkuZyDTiXYePA=; b=K0M+RaiB3vNeG2+QelP4Yj9UAUyZWh/I0dQj8j/LXnr9LaHSkROOdbvYy2pFCeDO7I a8FWvai1n4Z0y3VEUHksG3tvjsX1vchHazU/T6nyBFte8UBwRpe8OA5lBiUX0HSS/5qQ p2Ne3hGUXuxSaz5MwyELLP0NwHoI1g4aPzbMaB6qgDA7CcqD7KDYjKX/WHDg/V1tIa1z gTvWEk9Y+3qMPBQOi0Y+VNGog8RZrMku80u53/FcTDy6b+xOLOD/omfigw7LByq9cbxg 19dVMZ3ohMjk9o/a/5UMdX2reSpTWkC/axoQ+KyEjiSRULPrE1SPJ86io0MYEXetFaAk P1Sw== X-Received: by 10.140.92.37 with SMTP id a34mr315164qge.91.1400027380022; Tue, 13 May 2014 17:29:40 -0700 (PDT) Received: from [10.71.104.218] (mobile-198-228-193-029.mycingular.net. [198.228.193.29]) by mx.google.com with ESMTPSA id q16sm105940qga.42.2014.05.13.17.29.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 13 May 2014 17:29:37 -0700 (PDT) From: Jay Vyas Content-Type: multipart/alternative; boundary=Apple-Mail-29D438D5-C1F2-435E-8ADF-D8B8586DAC6D Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Subject: Re: MR Jobs failing in a vagrant deployment Message-Id: Date: Tue, 13 May 2014 20:29:35 -0400 References: In-Reply-To: To: "user@bigtop.apache.org" X-Mailer: iPhone Mail (11D201) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-29D438D5-C1F2-435E-8ADF-D8B8586DAC6D Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable No probably :) In the end I fixed it by exporting the right env variables ii= rc > On May 13, 2014, at 7:47 PM, Mark Grover wrote: >=20 > Sorry about the delay, Jay. Not entirely sure what's going on there but th= e shuffle properties were renamed in Hadoop 2.2.0. >=20 > If the VM is using that or later version of hadoop, it might be worthwhile= to ensure property names are correct. >=20 > Please see https://issues.apache.org/jira/browse/BIGTOP-1118 for details. >=20 > Mark >=20 >=20 >> On Wed, May 7, 2014 at 8:48 PM, Jay Vyas wrote: >> Hi folks. poking around with the vagrant deployer today, I saw that i co= uldnt run mapreduce jobs. All the services are running.... >>=20 >>=20 >> 8551 DataNode >> 12182 ResourceManager >> 13245 JobHistoryServer >> 8119 NameNode >> 7293 SecondaryNameNode >> 7737 WebAppProxyServer >> 17489 Jps >> 12788 NodeManager >>=20 >> My site.csv file is:=20 >>=20 >> cat > /bigtop-puppet/config/site.csv << EOF >> hadoop_head_node,$1 >> hadoop_storage_dirs,/data/1,/data/2 >> bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6= /x86_64 >> jdk_package_name,java-1.7.0-openjdk-devel.x86_64 >> components,mapred-app,hadoop,yarn, >> EOF >>=20 >> But the "shuffle port" value is returning -1.... Any thoughts on why thi= s might be occuring? Ive tried restarting nodemanager and resource manger b= ut no luck .=20 >>=20 >> 14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop= .yarn.client.YarnClientImpl is inited. >> 14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop= .yarn.client.YarnClientImpl is started. >> 14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to proces= s : 1 >> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1 >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated. Inst= ead, use mapreduce.job.jar >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks.speculative.e= xecution is deprecated. Instead, use mapreduce.map.speculative >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is depreca= ted. Instead, use mapreduce.job.reduces >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is d= eprecated. Instead, use mapreduce.job.output.value.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks.speculativ= e.execution is deprecated. Instead, use mapreduce.reduce.speculative >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is depreca= ted. Instead, use mapreduce.job.map.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is deprecated.= Instead, use mapreduce.job.name >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is depr= ecated. Instead, use mapreduce.job.reduce.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is= deprecated. Instead, use mapreduce.job.inputformat.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is deprecated= . Instead, use mapreduce.input.fileinputformat.inputdir >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is deprecate= d. Instead, use mapreduce.output.fileoutputformat.outputdir >> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class i= s deprecated. Instead, use mapreduce.job.outputformat.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is deprecated= . Instead, use mapreduce.job.maps >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is dep= recated. Instead, use mapreduce.job.output.key.class >> 14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is deprecat= ed. Instead, use mapreduce.job.working.dir >> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job:= job_1399518957219_0005 >> 14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit i= s no longer used. >> 14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application appli= cation_1399518957219_0005 to ResourceManager at bigtop1.vagrant/10.10.10.12:= 8032 >> 14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job: http://bi= gtop1.vagrant:20888/proxy/application_1399518957219_0005/ >> 14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005= >> 14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running i= n uber mode : false >> 14/05/08 03:42:07 INFO mapreduce.Job: map 0% reduce 0% >> 14/05/08 03:42:07 INFO mapreduce.Job: Task Id : attempt_1399518957219_000= 5_m_000000_0, Status : FAILED >> Container launch failed for container_1399518957219_0005_01_000002 : java= .lang.IllegalStateException: Invalid shuffle port number -1 returned for att= empt_1399518957219_0005_m_000000_0 >> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$= Container.launch(ContainerLauncherImpl.java:170) >> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$= EventProcessor.run(ContainerLauncherImpl.java:396) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:615) >> at java.lang.Thread.run(Thread.java:744) >>=20 >>=20 >> --=20 >> Jay Vyas >> http://jayunit100.blogspot.com >=20 --Apple-Mail-29D438D5-C1F2-435E-8ADF-D8B8586DAC6D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
No probably :) In the end I fixed it by exporting the right env variables iirc

On May 13, 2014, at 7:47 PM, Mark Grover <mark@apache.org> wrote:

Sorry about the delay, Jay. Not entirely sure what's going on there but the shuffle properties were renamed in Hadoop 2.2.0.

If the VM is using that or later version of hadoop, it might be worthwhile to ensure property names are correct.


Mark


On Wed, May 7, 2014 at 8:48 PM, Jay Vyas <jayunit100@gmail.com> wrote:
Hi folks.  poking around with the vagrant deployer today, I saw that i couldnt run mapreduce jobs.  All the services are running....


8551 DataNode
12182 ResourceManager
13245 JobHistoryServer
8119 NameNode
7293 SecondaryNameNode
7737 WebAppProxyServer
17489 Jps
12788 NodeManager

My site.csv file is:

cat > /bigtop-puppet/config/site.csv << EOF
hadoop_head_node,$1
hadoop_storage_dirs,/data/1,/data/2
bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
jdk_package_name,java-1.7.0-openjdk-devel.x86_64
components,mapred-app,hadoop,yarn,
EOF

But the "shuffle port" value is returning -1....  Any thoughts on why this might be occuring?   Ive tried restarting nodemanager and resource manger but no luck .

14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to process : 1
14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1
14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1399518957219_0005
14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit is no longer used.
14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application application_1399518957219_0005 to ResourceManager at bigtop1.vagrant/10.10.10.12:8032
14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job: http://bigtop1.vagrant:20888/proxy/application_1399518957219_0005/
14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005
14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running in uber mode : false
14/05/08 03:42:07 INFO mapreduce.Job:  map 0% reduce 0%
14/05/08 03:42:07 INFO mapreduce.Job: Task Id : attempt_1399518957219_0005_m_000000_0, Status : FAILED
Container launch failed for container_1399518957219_0005_01_000002 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1399518957219_0005_m_000000_0
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)


--
Jay Vyas
http://jayunit100.blogspot.com

--Apple-Mail-29D438D5-C1F2-435E-8ADF-D8B8586DAC6D--