hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xun Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8563) [Submarine] Support users to specify Python/TF package/version/dependencies for training job.
Date Fri, 21 Sep 2018 15:19:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623762#comment-16623762
] 

Xun Liu commented on YARN-8563:
-------------------------------

I don't think submarine needs to care about image management and production issues for the
following reasons:
 # YARN-3.x has supported runtime environment isolation through docker. If too much consideration
is given to supporting dynamic installation dependencies on a base image, then the meaning
of docker is lost.
 # Submarine need to provide a docker-hub repository for the submarine project, which provides
several stable versions of the tensorflow version, plus a full python library as the base
image, which is sufficient for the user to use.
Because ordinary users are generally able to use the off-the-shelf docker image to complete
their work, it is satisfied.
If they are advanced users, they will customize it using our base image.

So I think that the production and management of the image is handled by the user and does
not need to be included in the submarine project. Submarine need more focus on the high performance,
stability and ease of use of the submarine.

> [Submarine] Support users to specify Python/TF package/version/dependencies for training
job.
> ---------------------------------------------------------------------------------------------
>
>                 Key: YARN-8563
>                 URL: https://issues.apache.org/jira/browse/YARN-8563
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wangda Tan
>            Priority: Major
>
> YARN-8561 assumes all Python / Tensorflow dependencies will be packed to docker image.
In practice, user doesn't want to build docker image. Instead, user can provide python package
/ dependencies (like .whl), Python and TF version. And Submarine can localize specified dependencies
to prebuilt base Docker images.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message