bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: MR Jobs failing in a vagrant deployment
Date Wed, 14 May 2014 00:29:35 GMT
No probably :) In the end I fixed it by exporting the right env variables iirc

> On May 13, 2014, at 7:47 PM, Mark Grover <mark@apache.org> wrote:
> 
> Sorry about the delay, Jay. Not entirely sure what's going on there but the shuffle properties
were renamed in Hadoop 2.2.0.
> 
> If the VM is using that or later version of hadoop, it might be worthwhile to ensure
property names are correct.
> 
> Please see https://issues.apache.org/jira/browse/BIGTOP-1118 for details.
> 
> Mark
> 
> 
>> On Wed, May 7, 2014 at 8:48 PM, Jay Vyas <jayunit100@gmail.com> wrote:
>> Hi folks.  poking around with the vagrant deployer today, I saw that i couldnt run
mapreduce jobs.  All the services are running....
>> 
>> 
>> 8551 DataNode
>> 12182 ResourceManager
>> 13245 JobHistoryServer
>> 8119 NameNode
>> 7293 SecondaryNameNode
>> 7737 WebAppProxyServer
>> 17489 Jps
>> 12788 NodeManager
>> 
>> My site.csv file is: 
>> 
>> cat > /bigtop-puppet/config/site.csv << EOF
>> hadoop_head_node,$1
>> hadoop_storage_dirs,/data/1,/data/2
>> bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
>> jdk_package_name,java-1.7.0-openjdk-devel.x86_64
>> components,mapred-app,hadoop,yarn,
>> EOF
>> 
>> But the "shuffle port" value is returning -1....  Any thoughts on why this might
be occuring?   Ive tried restarting nodemanager and resource manger but no luck . 
>> 
>> 14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl
is inited.
>> 14/05/08 03:41:59 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl
is started.
>> 14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to process : 1
>> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated. Instead, use
mapreduce.job.jar
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks.speculative.execution
is deprecated. Instead, use mapreduce.map.speculative
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead,
use mapreduce.job.reduces
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is deprecated.
Instead, use mapreduce.job.output.value.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution
is deprecated. Instead, use mapreduce.reduce.speculative
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead,
use mapreduce.job.map.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is deprecated. Instead,
use mapreduce.job.name
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is deprecated.
Instead, use mapreduce.job.reduce.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is deprecated.
Instead, use mapreduce.job.inputformat.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is deprecated. Instead,
use mapreduce.input.fileinputformat.inputdir
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is deprecated. Instead,
use mapreduce.output.fileoutputformat.outputdir
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class is deprecated.
Instead, use mapreduce.job.outputformat.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is deprecated.
Instead, use mapreduce.job.output.key.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is deprecated. Instead,
use mapreduce.job.working.dir
>> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1399518957219_0005
>> 14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit is no longer
used.
>> 14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application application_1399518957219_0005
to ResourceManager at bigtop1.vagrant/10.10.10.12:8032
>> 14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job: http://bigtop1.vagrant:20888/proxy/application_1399518957219_0005/
>> 14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005
>> 14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running in uber
mode : false
>> 14/05/08 03:42:07 INFO mapreduce.Job:  map 0% reduce 0%
>> 14/05/08 03:42:07 INFO mapreduce.Job: Task Id : attempt_1399518957219_0005_m_000000_0,
Status : FAILED
>> Container launch failed for container_1399518957219_0005_01_000002 : java.lang.IllegalStateException:
Invalid shuffle port number -1 returned for attempt_1399518957219_0005_m_000000_0
>>     at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
>>     at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
>>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>     at java.lang.Thread.run(Thread.java:744)
>> 
>> 
>> -- 
>> Jay Vyas
>> http://jayunit100.blogspot.com
> 

Mime
View raw message