hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8079) Support static and archive unmodified local resources in service AM
Date Sun, 20 May 2018 23:31:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482076#comment-16482076
] 

Eric Yang commented on YARN-8079:
---------------------------------

[~leftnoteasy] When the files are placed in resources directory, patch 10 implementation prevents
mistake to overwrite system level generated files, such as .token file, and launch_container.sh.
 However, this design can created inconvenience for some users because existing Hadoop workload
may already be using the top level localized directory instead of resource directory.  We
may not need to worry about launch_container.sh getting overwritten because container-executor
generates the file after static files are localized.  Apps will try to avoid .token files
because they can not contact HDFS from containers, if they overwrites the token files.  In
summary, it is likely safe to remove the requirement of "resources" directory from my point
of view.

> Support static and archive unmodified local resources in service AM
> -------------------------------------------------------------------
>
>                 Key: YARN-8079
>                 URL: https://issues.apache.org/jira/browse/YARN-8079
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Suma Shivaprasad
>            Priority: Critical
>             Fix For: 3.2.0, 3.1.1
>
>         Attachments: YARN-8079.001.patch, YARN-8079.002.patch, YARN-8079.003.patch, YARN-8079.004.patch,
YARN-8079.005.patch, YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, YARN-8079.009.patch,
YARN-8079.010.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly read srcFile,
instead it always construct {{remoteFile}} by using componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code} 
> To me it is a common use case which services have some files existed in HDFS and need
to be localized when components get launched. (For example, if we want to serve a Tensorflow
model, we need to localize Tensorflow model (typically not huge, less than GB) to local disk.
Otherwise launched docker container has to access HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message