hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhankun Tang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-8851) [Umbrella] A new pluggable device plugin framework to ease vendor plugin development
Date Thu, 18 Oct 2018 03:49:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654557#comment-16654557
] 

Zhankun Tang edited comment on YARN-8851 at 10/18/18 3:48 AM:
--------------------------------------------------------------

[~leftnoteasy] 

5) DeviceRuntimeSpec is empty, what you plan to add?

{color:#d04437}Zhankun–>{color} It's a set of classeds and is described more clearly
in the UML figure of the design doc. In general, it is returned by vendor plugin's implementation
hook "OnDeviceAllocated" to describe the requirements of environment or volume creation or
docker command updates.etc. This DeviceRuntimeSpec will be translated to YARN internal operations
by the "DevicePluginAdapter". For instance, GPUv1 might require a volume creation before container
launch. And in this DeviceRuntimeSpec, it is a volumeClaim to describe it and let NM to create
it. Another example is GPUv2 needs additional environment when running Docker container, this
is described by "envs". And for cgroups device isolation, it is described by "MountDeviceSpec".

The class is like this:
{code:java}
class DeviceRuntimeSpec {
String runtime; // describe needed runtime. maybe not needed?
Map<String, String> envs; // describe needed environment variables before using devices
Set<MountVolumeSpec> volumeMounts; // describe volumes need to be mounted before using
devices
Set<MountDeviceSpec> devices; // describe devices needed to be mount
Set<VolumeSpec> volumeClaim; // describe volume to be created/delete before using devices
}
{code}
 

 


was (Author: tangzhankun):
[~leftnoteasy] 

5) DeviceRuntimeSpec is empty, what you plan to add?

{color:#d04437}Zhankun–>{color} It's a set of classeds and is described more clearly
in the UML figure of the design doc. In general, it is returned by vendor plugin's implementation
hook "OnDeviceAllocated" to describe the requirements of environment or volume creation or
docker command updates.etc. This DeviceRuntimeSpec will be translated to YARN internal operations
by the "DevicePluginAdapter". For instance, GPUv1 might require a volume creation before container
launch. And in this DeviceRuntimeSpec, it is a volumeClaim to describe it and let NM to create
it. Another example is GPUv2 needs additional environment when running Docker container, this
is described by "envs". And for cgroups device isolation, it is described by "MountDeviceSpec".

The class is like this:
{code:java}
class DeviceRuntimeSpec {
Map<String, String> envs; // describe needed environment variables before using devices
Set<MountVolumeSpec> volumeMounts; // describe volumes need to be mounted before using
devices
Set<MountDeviceSpec> devices; // describe devices needed to be mount
Set<VolumeSpec> volumeClaim; // describe volume to be created/delete before using devices
}
{code}
 

 

> [Umbrella] A new pluggable device plugin framework to ease vendor plugin development
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-8851
>                 URL: https://issues.apache.org/jira/browse/YARN-8851
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: yarn
>            Reporter: Zhankun Tang
>            Assignee: Zhankun Tang
>            Priority: Major
>         Attachments: YARN-8851-WIP2-trunk.001.patch, YARN-8851-WIP3-trunk.001.patch,
YARN-8851-WIP4-trunk.001.patch, [YARN-8851] YARN_New_Device_Plugin_Framework_Design_Proposal-3.pdf,
[YARN-8851] YARN_New_Device_Plugin_Framework_Design_Proposal.pdf
>
>
> At present, we support GPU/FPGA device in YARN through a native, coupling way. But it's
difficult for a vendor to implement such a device plugin because the developer needs much
knowledge of YARN internals. And this brings burden to the community to maintain both YARN
core and vendor-specific code.
> Here we propose a new device plugin framework to ease vendor device plugin development
and provide a more flexible way to integrate with YARN NM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message