hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-8079) Support static and archive unmodified local resources in service AM
Date Mon, 21 May 2018 00:21:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482076#comment-16482076
] 

Eric Yang edited comment on YARN-8079 at 5/21/18 12:20 AM:
-----------------------------------------------------------

[~leftnoteasy] When the files are placed in resources directory, patch 10 implementation prevents
mistake to overwrite system level generated files, such as .token file, and launch_container.sh.
 However, this design can created inconvenience for some users because existing Hadoop workload
may already be using the top level localized directory instead of resource directory.  We
may not need to worry about launch_container.sh getting overwritten because container-executor
generates the file after static files are localized.  Apps will try to avoid .token files
because they can not contact HDFS from containers, if they overwrites the token files.  

With resources directory, it maybe easier for end user to specify a single relative directory
to bind-mount instead of specifying individual files to bind-mount in yarnfile.  By removing
the resources directory, user will need to think a bit more on how to manage the bind-mount
directories to reduce wordy syntax.

With both approaches considered, it all comes down to usability of which approach is easiest
to use, while not creating too much clutters.  In summary, it might be safe to remove the
requirement of "resources" directory from my point of view.


was (Author: eyang):
[~leftnoteasy] When the files are placed in resources directory, patch 10 implementation prevents
mistake to overwrite system level generated files, such as .token file, and launch_container.sh.
 However, this design can created inconvenience for some users because existing Hadoop workload
may already be using the top level localized directory instead of resource directory.  We
may not need to worry about launch_container.sh getting overwritten because container-executor
generates the file after static files are localized.  Apps will try to avoid .token files
because they can not contact HDFS from containers, if they overwrites the token files.  In
summary, it is likely safe to remove the requirement of "resources" directory from my point
of view.

> Support static and archive unmodified local resources in service AM
> -------------------------------------------------------------------
>
>                 Key: YARN-8079
>                 URL: https://issues.apache.org/jira/browse/YARN-8079
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Suma Shivaprasad
>            Priority: Critical
>             Fix For: 3.2.0, 3.1.1
>
>         Attachments: YARN-8079.001.patch, YARN-8079.002.patch, YARN-8079.003.patch, YARN-8079.004.patch,
YARN-8079.005.patch, YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, YARN-8079.009.patch,
YARN-8079.010.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly read srcFile,
instead it always construct {{remoteFile}} by using componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code} 
> To me it is a common use case which services have some files existed in HDFS and need
to be localized when components get launched. (For example, if we want to serve a Tensorflow
model, we need to localize Tensorflow model (typically not huge, less than GB) to local disk.
Otherwise launched docker container has to access HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message