hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Varun Vasudev <vvasu...@hortonworks.com>
Subject Re: URI missing scheme and authority in job start with new FileSystem implementation
Date Fri, 08 May 2015 16:10:11 GMT
Hi Silvan,

Are you trying to run MapReduce? You could look at createApplicationSubmissionContext and
createApplicationResource in YarnRunner.java. Have you overridden the resolvePath function
in your filesystem implementation?

-Varun

From:  Silvan Kaiser
Reply-To:  "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
Date:  Friday, May 8, 2015 at 3:52 PM
To:  "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
Subject:  URI missing scheme and authority in job start with new FileSystem implementation


Hello!
I am running a new FileSystem implementation, testing it with a simple standalone operation
from the Hadoop "Single Node Setup" page. Staging example jobs works alright but upon scheduling
the container management fails because of an URI without scheme
(and authority).

NodeManager log snippet:
...

2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Got exception parsing jobSubmitDir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo

...


The path element of the URI is correct, it simply is missing it's scheme and authority but
i cannot find out why so far.
Please note that the URIs used during staging are correct and contain scheme & authority
correctly. This means the incomplete URI is created during job creation.

My guess so far: this is either because I am missing some configuration for the Nodemanager
(therefore he does not know about my FileSystem implementation and throws the unknown scheme
& authority out the window) or that the FileSystem implementation has
an error / did not override a method that requires adoption.

Can someone please give me a hint where the URIs used by the NodeManager are constructed prior
to scheduling? This would allow me to trace back where the incomplete URI is coming from...

Best regards
Silvan


--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/><http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender









--
Quobyte GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com <http://www.quobyte.com/>Amtsgericht Berlin-Charlottenburg,
HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
Mime
View raw message