hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhankun Tang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5983) [Umbrella] Support for FPGA as a Resource in YARN
Date Fri, 28 Apr 2017 08:58:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988445#comment-15988445

Zhankun Tang commented on YARN-5983:

[~devaraj.k], Thanks a lot for the review.
The scheduler only considers non-exclusive resource. The exclusive resources may
have extra attributes needs to be matched when scheduling. Not just simply add or
reduce a number. For instance, in our PoC, a FPGA slot in one node may already
have one IP flashed so that the scheduler should try to match this IP attribute to
reuse it.
If you are passing all the attributes of the FPGA resources to RM scheduler, why do you want
to have the NM side resource management? Can you give some details about the attributes passing
to the RM and details maintain by the NM side resource management in abstract terms?
Zhankun --> The RM maintains all attributes of FPGA resources is an ideal solution which
needs to extend YARN-3926 scheduler implementation. But in this design proposal, we'd prefer
to keep YARN-3926 RM scheduler unchanged(which only consider the count of resource) and add
NM a local FPGA resource handler which will mainly:
a). Schedule the FPGA resource request(2 MCP for instance) into real FPGA device
b). Handle FPGA IP download and re-configure of specific slot
The key part of this proposal is similar to GPU design(YARN-6223) but add the vendor framework
Does this need to be done for each container, Can it be done one time during the cluster installation?
Zhankun: --> Good question. Actually the admin can do the IP download and configuration
of FPGA one time to the cluster manually. But if an application wants to use a new FPGA IP
that needs re-configuration, a good way of doing these may be in YARN itself. This automation
in YARN can potentially increase the FPGA utilization(not good as the ideal RM scheduler)
if we have large amount of FPGA devices.
And these work should be done for each container, although we can skip it if an FPGA slot
already matches IP requirements.
Can FPGA slots share my multiple containers? How do we prevent if any container(Non FPGA allocated
container)/application try to use the FPGA resources which are not allocated to them?
Zhankun: --> We can utilize cgroups/Docker to do FPGA slot isolation to avoid invalid access.
But I guess there may be requirements that don't want isolation in order to maximize device
utilization( the user may have a daemon managing requests from multiple containers to FPGA
IP), we can provide option for user to disable isolation?
Any changes to ContainerExecutor, how does the application code running in the container come
to know about the allocated FPGA resource to access/use the FPFA?
Zhankun:--> It should be no changes to both DefaultContainerExecutor and LinuxContainerExecutor.
Knowing about which FPGA resource to use depends on the vendor specific SDK I think. For instance,
container application code may utilize Intel AAL SDK and the latter will know which FPGA slot
to use (maybe based on some environment setting).
What are the configurations user to need to configure for the application to use FPGA resources?
Zhankun: --> For YARN configurations before cluster start, the admin need to define different
type of FPGA resources and its count. When a user wants to submit the application, it needs
to provide two inputs:
a). the required FPGA resource for container request (AM will use this to request FPGA container
from RM)
b). the ID/name of the required FPGA IP (can be set in container's environment variable for
container run-time to use)

And great thanks again for the questions and remind me some points uncertain. I'll provide
a patch to make it more clear later.

> [Umbrella] Support for FPGA as a Resource in YARN
> -------------------------------------------------
>                 Key: YARN-5983
>                 URL: https://issues.apache.org/jira/browse/YARN-5983
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: yarn
>            Reporter: Zhankun Tang
>            Assignee: Zhankun Tang
>         Attachments: YARN-5983-Support-FPGA-resource-on-NM-side_v1.pdf
> As various big data workload running on YARN, CPU will no longer scale eventually and
heterogeneous systems will become more important. ML/DL is a rising star in recent years,
applications focused on these areas have to utilize GPU or FPGA to boost performance. Also,
hardware vendors such as Intel also invest in such hardware. It is most likely that FPGA will
become popular in data centers like CPU in the near future.
> So YARN as a resource managing and scheduling system, would be great to evolve to support
this. This JIRA proposes FPGA to be a first-class citizen. The changes roughly includes:
> 1. FPGA resource detection and heartbeat
> 2. Scheduler changes
> 3. FPGA related preparation and isolation before launch container
> We know that YARN-3926 is trying to extend current resource model. But still we can leave
some FPGA related discussion here

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message