hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billie Rinaldi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8079) YARN native service should respect source file of ConfigFile inside Service/Component spec
Date Tue, 27 Mar 2018 18:50:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416086#comment-16416086

Billie Rinaldi commented on YARN-8079:

The source file is currently used only for the HADOOP_XML and TEMPLATE config file types.
In those cases the source file is read and config properties are merged into the source file
to create the destination / remote file. The TEMPLATE type could work for your use case, but
the file contents will be read into AM memory and written to a new file. If that isn't a good
idea for the files you want to localize, I think what we would need to do is introduce a new
file type such as STATIC where the AM would localize the file without reading it.

> YARN native service should respect source file of ConfigFile inside Service/Component
> ------------------------------------------------------------------------------------------
>                 Key: YARN-8079
>                 URL: https://issues.apache.org/jira/browse/YARN-8079
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Blocker
>         Attachments: YARN-8079.001.patch
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly read srcFile,
instead it always construct {{remoteFile}} by using componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code} 
> To me it is a common use case which services have some files existed in HDFS and need
to be localized when components get launched. (For example, if we want to serve a Tensorflow
model, we need to localize Tensorflow model (typically not huge, less than GB) to local disk.
Otherwise launched docker container has to access HDFS.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message