hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
Date Wed, 23 Aug 2017 18:44:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138862#comment-16138862
] 

Eric Yang edited comment on YARN-7066 at 8/23/17 6:43 PM:
----------------------------------------------------------

The current proposed syntax looks like this:

{code}
{
  "name": "hbase-app-1",
  "components": [
    {
      "name": "hbasemaster",
      ...
      "configuration": {
        "env": {
          "HBASE_LOG_DIR": "<LOG_DIR>",
          "MOUNTS": "[{ \"source\":\"/home/${USER}\", \"target\":\"/mnt/hdfs/user/${USER}\",
\"option\":\"ro\" },{ \"source\":\"/tmp/${USER}/data\", \"target\":\"/mnt/hdfs/tmp/${USER}/data\"
}]"
        },
    },
    {
      ...
    }
  ],
  ...
}
{code}

Where "MOUNTS" is a string of JSON that specifies list of mount point source, target, and
option.

{code}
{
  "source": "/home/${USER}",
  "target": "/mnt/hdfs/${USER}",
  "option": "ro"
}
{code}

The nicer design looks like this in Yarnfile:

{code}
    {
      "name": "serving",
      ...
      "configuration": {
        "volumes": [
          {
            "source": "/mnt/hdfs/user/${USER}",
            "target": "/home/${USER}",
            "option": "ro"
          }
        ]
      }
    }
{code}

The nice design will break a couple Yarn container interface because the original design doesn't
contain volumes.  Hence, I will go with environment variable implementation.  It might be
possible to expose the volumes keyword for Yarnfile, then pass the information through interface
using the environment variables to avoid changes to container interface.


was (Author: eyang):
The current proposed syntax looks like this:

{code}
{
  "name": "hbase-app-1",
  "components": [
    {
      "name": "hbasemaster",
      ...
      "configuration": {
        "env": {
          "HBASE_LOG_DIR": "<LOG_DIR>",
          "MOUNTS": "[{ \"source\":\"/home/${USER}\", \"target\":\"/mnt/hdfs/user/${USER}\",
\"option\":\"ro\" },{ \"source\":\"/tmp/${USER}/data\", \"target\":\"/mnt/hdfs/tmp/${USER}/data\"
}]"
        },
    },
    {
      ...
    }
  ],
  ...
}
{code}

Where "MOUNTS" is a string of JSON that specifies list of mount point source, target, and
option.

{code}
{
  "source": "/home/${USER}",
  "target": "/mnt/hdfs/${USER}",
  "option": "ro"
}
{code}

The nicer design looks like this in Yarnfile:

{code}
    {
      "name": "serving",
      ...
      "configuration": {
        "volumes": [
          {
            "source": "/mnt/hdfs/user/${user}",
            "target": "/home/${user}",
            "option": "ro"
          }
        ]
      }
    }
{code}

The nice design will break a couple Yarn container interface because the original design doesn't
contain volumes.  Hence, I will go with environment variable implementation.  It might be
possible to expose the volumes keyword for Yarnfile, then pass the information through interface
using the environment variables to avoid changes to container interface.

> Add ability to specify volumes to mount for DockerContainerRuntime
> ------------------------------------------------------------------
>
>                 Key: YARN-7066
>                 URL: https://issues.apache.org/jira/browse/YARN-7066
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn-native-services
>    Affects Versions: 3.0.0-beta1
>            Reporter: Eric Yang
>
> Yarnfile describes environment, docker image, and configuration template for launching
docker containers in YARN.  It would be nice to have ability to specify the volumes to mount.
 This can be used in combination to AMBARI-21748 to mount HDFS as data directories to docker
containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message