bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Grover <m...@apache.org>
Subject Re: MR Jobs failing in a vagrant deployment
Date Tue, 13 May 2014 23:47:42 GMT
Sorry about the delay, Jay. Not entirely sure what's going on there but the
shuffle properties were renamed in Hadoop 2.2.0.

If the VM is using that or later version of hadoop, it might be worthwhile
to ensure property names are correct.

Please see https://issues.apache.org/jira/browse/BIGTOP-1118 for details.

Mark


On Wed, May 7, 2014 at 8:48 PM, Jay Vyas <jayunit100@gmail.com> wrote:

> Hi folks.  poking around with the vagrant deployer today, I saw that i
> couldnt run mapreduce jobs.  All the services are running....
>
>
> 8551 DataNode
> 12182 ResourceManager
> 13245 JobHistoryServer
> 8119 NameNode
> 7293 SecondaryNameNode
> 7737 WebAppProxyServer
> 17489 Jps
> 12788 NodeManager
>
> My site.csv file is:
>
> cat > /bigtop-puppet/config/site.csv << EOF
> hadoop_head_node,$1
> hadoop_storage_dirs,/data/1,/data/2
> bigtop_yumrepo_uri,
> http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
> jdk_package_name,java-1.7.0-openjdk-devel.x86_64
> components,mapred-app,hadoop,yarn,
> EOF
>
> But the "shuffle port" value is returning -1....  Any thoughts on why this
> might be occuring?   Ive tried restarting nodemanager and resource manger
> but no luck .
>
> 14/05/08 03:41:59 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
> 14/05/08 03:41:59 INFO service.AbstractService:
> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
> 14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 14/05/08 03:41:59 WARN conf.Configuration:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
> 14/05/08 03:41:59 WARN conf.Configuration:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is
> deprecated. Instead, use mapreduce.job.reduce.class
> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is deprecated.
> Instead, use mapreduce.input.fileinputformat.inputdir
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1399518957219_0005
> 14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit is
> no longer used.
> 14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application
> application_1399518957219_0005 to ResourceManager at bigtop1.vagrant/
> 10.10.10.12:8032
> 14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job:
> http://bigtop1.vagrant:20888/proxy/application_1399518957219_0005/
> 14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005
> 14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running
> in uber mode : false
> 14/05/08 03:42:07 INFO mapreduce.Job:  map 0% reduce 0%
> 14/05/08 03:42:07 INFO mapreduce.Job: Task Id :
> attempt_1399518957219_0005_m_000000_0, Status : FAILED
> Container launch failed for container_1399518957219_0005_01_000002 :
> java.lang.IllegalStateException: Invalid shuffle port number -1 returned
> for attempt_1399518957219_0005_m_000000_0
>     at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
>     at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:744)
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

Mime
View raw message