hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xun Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-8714) [Submarine] Support files/tarballs to be localized for a training job.
Date Wed, 14 Nov 2018 07:53:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684794#comment-16684794
] 

Xun Liu edited comment on YARN-8714 at 11/14/18 7:52 AM:
---------------------------------------------------------

1) Support for multiple files or directories.

2) Must maintain the original state, such as folder subdirectory structure, zip file or normal
file, do not make any changes.

3) Mount the specified file or folder to the absolute path in the container. Because the
user can customize the WORKDIR of the container through the Dockerfile, And the containers
are dedicated and custom made, So the user knows exactly where to mount the file.

4) Keep consistent with the use of docker, use`{color:#FF0000}:{color}` split source directory
and destination directory.

5) Parameter format: -localizations hdfs:///user/yarn{color:#FF0000}:{color}/absolute/path


was (Author: liuxun323):
# Support files and folders
 # HDFS type and local file type supporting HDFS:// prefix at the same time
 # Keep it intact: the uploaded package is compressed, and the submarine is automatically
decompressed into the container, which is not suitable, because if I rush to save the file
that needs to upload the compressed package format, it will be destroyed. And it also introduces
ambiguity.
 # Parameter format: {color:#FF0000}-localizations hdfs:///user/yarn>{color:#ff0000}.{color}{color}
# indicates the current execution path of the container
 # Parameter format: {color:#FF0000}-localizations hdfs:///user/yarn>./abc{color} # Indicates
the abc folder under the current execution path of the container (submarine marks the file
under hdfs:///user/yarn as an abc .tar.gz compression package, extract the abc folder when
pulling up the container, then mount it in it)

> [Submarine] Support files/tarballs to be localized for a training job.
> ----------------------------------------------------------------------
>
>                 Key: YARN-8714
>                 URL: https://issues.apache.org/jira/browse/YARN-8714
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Assignee: Zhankun Tang
>            Priority: Major
>         Attachments: YARN-8714-WIP1-trunk-001.patch
>
>
> See https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
{{job run --localizations ...}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message