hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
Date Sat, 22 Sep 2018 16:30:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624736#comment-16624736
] 

Weiwei Yang commented on YARN-8811:
-----------------------------------

Hi [~eyang], [~leftnoteasy]

Thanks for the feedback.
{quote}Mount propagation flags, file system type, source and destination mount points.
{quote}
[~eyang], thanks for pointing these out. Mount propagation is not in the scope of this feature,
in another word we will only support \{{private}} mount here. File system type can be specified
in the CSI spec, see more in VolumeCapability - MountVolume - fs_type, so this will depend
on the storage system's capability. CSI has API to validate VolumeCapability to avoid invalid
calls. Source/destination mount points are roughly introduced in section 5.2. I can add more
info for these. For the comment about object store user API key information, I am not sure
about this point, could you please elaborate.
{quote}The name "IGNORABLE" is ambiguous, 
{quote}
[~leftnoteasy], I am totally fine with other names, maybe *{color:#333333}UNMANAGED{color}*
{color:#333333}is a better one{color}?
{quote}user will be configured about is it ignorable by main scheduler / nm or app, etc. And
from the name it looks like "no one cares about the resource type
{quote}
For built-in resources, we'll disallow user to modify the resource type. User will only be
allowed to configure type for new user defined resource, and they take the responsibility
to make it right. 
{quote}I prefer to have a String[] tags added to each ResourceInformation
{quote}
I thought about this before, but gave up after a PoC. Pros of tags is we provide the flexibility
to support resource tagging/filtering, AKA filter resources by user given tags. But if we
want to support this, then we run into scenario like following,

For example, for a resource {{yarn.io/gpu}}, a cluster has 2 GPUs from 2 vendors, then user
want to mark one device with tag {{vendor_1}}, and the other tag with {{vendor_2}}. 

So when NMs report resource,

{code}

// NM1

yarn.io/gpu {

  tags : ["default", "vendor_1"]

  ...

}

// NM2

yarn.io/gpu {

   tags : ["default", "vendor_2"]

   ...

}

{code}

Note, they have to be two different {{ResoueceInformation}} instances, so when user ask for
{{vendor_1}}, we could allocate {{vendor_1}} gpu correctly on NM1 to the request. If user
defines more tags, we will have to deal with more instances, this is going to be complex and
hurt the perf. (This is because our resource model is flat, multi-dimensional resource is
not supported yet).

If you are suggesting just to use static tags, which means a resource can only associate with
a certain set of tags. Then I don't see how it is different with resource URI. Am I misunderstanding
your point? Please let me know.

Thanks

> Support Container Storage Interface (CSI) in YARN
> -------------------------------------------------
>
>                 Key: YARN-8811
>                 URL: https://issues.apache.org/jira/browse/YARN-8811
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Major
>         Attachments: Support Container Storage Interface(CSI) in YARN_design doc_20180921.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge Container
Orchestrators and Storage Providers. With the adoption of CSI in YARN, it will be easier to
integrate 3rd party storage systems, and provide the ability to attach persistent volumes
for stateful applications.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message