bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: MR Jobs failing in a vagrant deployment
Date Sun, 18 May 2014 00:26:03 GMT
Hey ! I just reprovisioned my vagrant node and realized, its MORE than just
the right env variables.

Ive filed a jira for this:

BIGTOP-1118 (mapreduce.shuffle -> mapreduce_shuffle)  makes it so that
puppet recipes are not backward compatible, and that breaks our vagrant
provisioners.

Maybe we can update puppet provisioner to be "Version aware" so that it
warns or fails if you try to use it on a wrong version of hadoop ?






On Tue, May 13, 2014 at 8:29 PM, Jay Vyas <jayunit100@gmail.com> wrote:

> No probably :) In the end I fixed it by exporting the right env variables
> iirc
>
> On May 13, 2014, at 7:47 PM, Mark Grover <mark@apache.org> wrote:
>
> Sorry about the delay, Jay. Not entirely sure what's going on there but
> the shuffle properties were renamed in Hadoop 2.2.0.
>
> If the VM is using that or later version of hadoop, it might be worthwhile
> to ensure property names are correct.
>
> Please see https://issues.apache.org/jira/browse/BIGTOP-1118 for details.
>
> Mark
>
>
> On Wed, May 7, 2014 at 8:48 PM, Jay Vyas <jayunit100@gmail.com> wrote:
>
>> Hi folks.  poking around with the vagrant deployer today, I saw that i
>> couldnt run mapreduce jobs.  All the services are running....
>>
>>
>> 8551 DataNode
>> 12182 ResourceManager
>> 13245 JobHistoryServer
>> 8119 NameNode
>> 7293 SecondaryNameNode
>> 7737 WebAppProxyServer
>> 17489 Jps
>> 12788 NodeManager
>>
>> My site.csv file is:
>>
>> cat > /bigtop-puppet/config/site.csv << EOF
>> hadoop_head_node,$1
>> hadoop_storage_dirs,/data/1,/data/2
>> bigtop_yumrepo_uri,
>> http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
>> jdk_package_name,java-1.7.0-openjdk-devel.x86_64
>> components,mapred-app,hadoop,yarn,
>> EOF
>>
>> But the "shuffle port" value is returning -1....  Any thoughts on why
>> this might be occuring?   Ive tried restarting nodemanager and resource
>> manger but no luck .
>>
>> 14/05/08 03:41:59 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>> 14/05/08 03:41:59 INFO service.AbstractService:
>> Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>> 14/05/08 03:41:59 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: number of splits:1
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.jar is deprecated.
>> Instead, use mapreduce.job.jar
>> 14/05/08 03:41:59 WARN conf.Configuration:
>> mapred.map.tasks.speculative.execution is deprecated. Instead, use
>> mapreduce.map.speculative
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.reduce.tasks is
>> deprecated. Instead, use mapreduce.job.reduces
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.value.class is
>> deprecated. Instead, use mapreduce.job.output.value.class
>> 14/05/08 03:41:59 WARN conf.Configuration:
>> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
>> mapreduce.reduce.speculative
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.map.class is
>> deprecated. Instead, use mapreduce.job.map.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.job.name is
>> deprecated. Instead, use mapreduce.job.name
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.reduce.class is
>> deprecated. Instead, use mapreduce.job.reduce.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.inputformat.class is
>> deprecated. Instead, use mapreduce.job.inputformat.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.input.dir is
>> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.dir is
>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>> 14/05/08 03:41:59 WARN conf.Configuration: mapreduce.outputformat.class
>> is deprecated. Instead, use mapreduce.job.outputformat.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.map.tasks is
>> deprecated. Instead, use mapreduce.job.maps
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.output.key.class is
>> deprecated. Instead, use mapreduce.job.output.key.class
>> 14/05/08 03:41:59 WARN conf.Configuration: mapred.working.dir is
>> deprecated. Instead, use mapreduce.job.working.dir
>> 14/05/08 03:41:59 INFO mapreduce.JobSubmitter: Submitting tokens for job:
>> job_1399518957219_0005
>> 14/05/08 03:42:00 WARN mapred.JobConf: The variable mapred.child.ulimit
>> is no longer used.
>> 14/05/08 03:42:00 INFO client.YarnClientImpl: Submitted application
>> application_1399518957219_0005 to ResourceManager at bigtop1.vagrant/
>> 10.10.10.12:8032
>> 14/05/08 03:42:00 INFO mapreduce.Job: The url to track the job:
>> http://bigtop1.vagrant:20888/proxy/application_1399518957219_0005/
>> 14/05/08 03:42:00 INFO mapreduce.Job: Running job: job_1399518957219_0005
>> 14/05/08 03:42:07 INFO mapreduce.Job: Job job_1399518957219_0005 running
>> in uber mode : false
>> 14/05/08 03:42:07 INFO mapreduce.Job:  map 0% reduce 0%
>> 14/05/08 03:42:07 INFO mapreduce.Job: Task Id :
>> attempt_1399518957219_0005_m_000000_0, Status : FAILED
>> Container launch failed for container_1399518957219_0005_01_000002 :
>> java.lang.IllegalStateException: Invalid shuffle port number -1 returned
>> for attempt_1399518957219_0005_m_000000_0
>>     at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:170)
>>     at
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:396)
>>     at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>     at java.lang.Thread.run(Thread.java:744)
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>


-- 
Jay Vyas
http://jayunit100.blogspot.com

Mime
View raw message