hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Angus Davis (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-6725) NodeManager cannot localize files when URL contains opaque authority
Date Wed, 21 Jun 2017 02:09:00 GMT
Angus Davis created YARN-6725:

             Summary: NodeManager cannot localize files when URL contains opaque authority
                 Key: YARN-6725
                 URL: https://issues.apache.org/jira/browse/YARN-6725
             Project: Hadoop YARN
          Issue Type: Bug
          Components: yarn
    Affects Versions: 2.8.1
            Reporter: Angus Davis
            Priority: Minor

When presented with a URI or URL that has a valid structure, but where the URI / URL authority
cannot be interpreted as a valid hostname, the java URI#getHost method will return null. In
the YARN api records the URL message is decomposed into hostname, username, port, etc, but
it does not provide a means to transport an opaque authority. This becomes problematic when
using object stores or other file systems that use something other than a hostname as an authority.

In our particular case, Google Cloud Storage allows buckets to be named with underscores which
is causing the NodeManagers to attempt to localize resources of the form 'gs:///path/to/object'
when they should be attempting to localize 'gs://bucket/path/to/object'. Components that are
written in terms of o.a.h.fs.Path instances seem to handle this case properly, as do most
Path utility methods (toUri, etc).

It seems that by transporting the authority along with the host, user info, etc, a YARN api
URL can properly represent o.a.h.fs.Path instances of this particular form. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

View raw message