hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Silvan Kaiser <sil...@quobyte.com>
Subject URI missing scheme and authority in job start with new FileSystem implementation
Date Fri, 08 May 2015 13:52:49 GMT
Hello!
I am running a new FileSystem implementation, testing it with a simple
standalone operation from the Hadoop "Single Node Setup" page. Staging
example jobs works alright but upon scheduling the container management
fails because of an URI without scheme (and authority).

NodeManager log snippet:
...
2015-05-08 15:18:15,331 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Got exception parsing jobSubmitDir/job.splitmetainfo and value resource {
port: -1 file:
"/tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION
2015-05-08 15:18:15,331 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Failed to parse resource-request
java.net.URISyntaxException: Expected scheme name at index 0:
:///tmp/hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo
...

The path element of the URI is correct, it simply is missing it's scheme
and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain
scheme & authority correctly. This means the incomplete URI is created
during job creation.

My guess so far: this is either because I am missing some configuration for
the Nodemanager (therefore he does not know about my FileSystem
implementation and throws the unknown scheme & authority out the window) or
that the FileSystem implementation has an error / did not override a method
that requires adoption.

Can someone please give me a hint where the URIs used by the NodeManager
are constructed prior to scheduling? This would allow me to trace back
where the incomplete URI is coming from...

Best regards
Silvan


-- 
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com<http://www.quobyte.com/>
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender

-- 

--
*Quobyte* GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender

Mime
View raw message