hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Angus Davis (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-6725) NodeManager cannot localize files when URL contains opaque authority
Date Wed, 21 Jun 2017 02:14:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Angus Davis updated YARN-6725:
    Attachment: YARN-6721.001.patch

First pass at adding authority to YARN URL pb.

> NodeManager cannot localize files when URL contains opaque authority
> --------------------------------------------------------------------
>                 Key: YARN-6725
>                 URL: https://issues.apache.org/jira/browse/YARN-6725
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 2.8.1
>            Reporter: Angus Davis
>            Priority: Minor
>         Attachments: YARN-6721.001.patch
> When presented with a URI or URL that has a valid structure, but where the URI / URL
authority cannot be interpreted as a valid hostname, the java URI#getHost method will return
null. In the YARN api records the URL message is decomposed into hostname, username, port,
etc, but it does not provide a means to transport an opaque authority. This becomes problematic
when using object stores or other file systems that use something other than a hostname as
an authority. 
> In our particular case, Google Cloud Storage allows buckets to be named with underscores
which is causing the NodeManagers to attempt to localize resources of the form 'gs:///path/to/object'
when they should be attempting to localize 'gs://bucket/path/to/object'. Components that are
written in terms of o.a.h.fs.Path instances seem to handle this case properly, as do most
Path utility methods (toUri, etc).
> It seems that by transporting the authority along with the host, user info, etc, a YARN
api URL can properly represent o.a.h.fs.Path instances of this particular form. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message