hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gour Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8079) YARN native service should respect source file of ConfigFile inside Service/Component spec
Date Wed, 28 Mar 2018 02:38:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416623#comment-16416623
] 

Gour Saha commented on YARN-8079:
---------------------------------

[~leftnoteasy] with reference to 003 patch, why do we need both STATIC and ARCHIVE types?
Seems like they both cover the same usecase, so we can choose one and discard the other. I
would vote for STATIC since not all files that needs to be localized will be archive files.
Otherwise, the patch looks good to me. We can cover advanced usecases like NFS gateway, object
store URL, different security-enabled HDFS later in the future.

> YARN native service should respect source file of ConfigFile inside Service/Component
spec
> ------------------------------------------------------------------------------------------
>
>                 Key: YARN-8079
>                 URL: https://issues.apache.org/jira/browse/YARN-8079
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Blocker
>         Attachments: YARN-8079.001.patch, YARN-8079.002.patch, YARN-8079.003.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly read srcFile,
instead it always construct {{remoteFile}} by using componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code} 
> To me it is a common use case which services have some files existed in HDFS and need
to be localized when components get launched. (For example, if we want to serve a Tensorflow
model, we need to localize Tensorflow model (typically not huge, less than GB) to local disk.
Otherwise launched docker container has to access HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message