hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: Invalide URI in job start
Date Mon, 27 Jan 2014 19:00:10 GMT
Need your help to debug this. Seems like the scheme is getting lost somewhere along the way.
Clearly as you say if job.jar is on the file-system, then JobClient is properly uploading
it. There are multilple things that you'll need to check
 - Check the NodeManager logs for the URL. It does print what URL it is trying to download
from. Check if the scheme is getting there or not.
 - If that doesn't tell you something, change JobClient to print the URL before it constructs
the ContainerLaunchContext for the ApplicationMaster. You'll need to do this in YarnRunner.java.
Specifically the method createApplicationResource.

+Vinod Kumar Vavilapalli
Hortonworks Inc.

On Jan 27, 2014, at 2:05 AM, Lukas Kairies <lukas.xtreemfs@googlemail.com> wrote:

> Hello,
> I try to use XtreemFS as an alternative file system for Hadoop 2.x. There is an existing
FileSystem implementation for Hadoop 1.x that works fine. First think I did was to implement
a DelegateToFileSystem subclass to provide an AbstractFileSystem implementation for XtreemFS
(just constructors that use the FileSystem implementation). When I start the wordcount example
application I get the following Exception on the NodeManager:
> 2014-01-20 14:18:19,349 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Failed to parse resource-request
> java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/lkairies/.staging/job_1390223418764_0004/job.jar
>        at java.net.URI$Parser.fail(URI.java:2829)
>        at java.net.URI$Parser.failExpecting(URI.java:2835)
>        at java.net.URI$Parser.parse(URI.java:3027)
>        at java.net.URI.<init>(URI.java:753)
>        at org.apache.hadoop.yarn.util.ConverterUtils.getPathFromYarnURL(ConverterUtils.java:80)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.<init>(LocalResourceRequest.java:46)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:529)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:497)
>        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
>        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:864)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:73)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:815)
>        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:808)
>        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:134)
>        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:81)
>        at java.lang.Thread.run(Thread.java:724)
> Additionally the following is printed on the console:
> 14/01/27 11:02:14 INFO input.FileInputFormat: Total input paths to process : 1
> 14/01/27 11:02:14 INFO mapreduce.JobSubmitter: number of splits:1
> 14/01/27 11:02:15 INFO Configuration.deprecation: user.name is deprecated. Instead, use
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.jar is deprecated. Instead,
use mapreduce.job.jar
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.output.value.class is deprecated.
Instead, use mapreduce.job.output.value.class
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapreduce.combine.class is deprecated.
Instead, use mapreduce.job.combine.class
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapreduce.map.class is deprecated.
Instead, use mapreduce.job.map.class
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead,
use mapreduce.job.name
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated.
Instead, use mapreduce.job.reduce.class
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead,
use mapreduce.input.fileinputformat.inputdir
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead,
use mapreduce.output.fileoutputformat.outputdir
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.output.key.class is deprecated.
Instead, use mapreduce.job.output.key.class
> 14/01/27 11:02:15 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead,
use mapreduce.job.working.dir
> 14/01/27 11:02:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1390816735288_0001
> 14/01/27 11:02:15 INFO impl.YarnClientImpl: Submitted application application_1390816735288_0001
to ResourceManager at /
> 14/01/27 11:02:15 INFO mapreduce.Job: The url to track the job: http://ludiwg:8088/proxy/application_1390816735288_0001/
> 14/01/27 11:02:15 INFO mapreduce.Job: Running job: job_1390816735288_0001
> 14/01/27 11:02:19 INFO mapreduce.Job: Job job_1390816735288_0001 running in uber mode
: false
> 14/01/27 11:02:19 INFO mapreduce.Job:  map 0% reduce 0%
> 14/01/27 11:02:19 INFO mapreduce.Job: Job job_1390816735288_0001 failed with state FAILED
due to: Application application_1390816735288_0001 failed 2 times due to AM Container for
appattempt_1390816735288_0001_000002 exited with  exitCode: -1000 due to: .Failing this attempt..
Failing the application.
> 14/01/27 11:02:19 INFO mapreduce.Job: Counters: 0
> The job files are created in XtreemFS. After a lot of debugging I still did not find
the problem.
> Any idea how to fix this?
> Regards,
> Lukas

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message