hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei Guo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4793) [Umbrella] Simplified API layer for services and beyond
Date Tue, 30 Aug 2016 14:07:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449102#comment-15449102

Lei Guo commented on YARN-4793:

As this Jira targets to build a unified interface, I'd like to share some thoughts related
to the resource modeling part. The core of Yarn is still to map the resource and workload.
For the resource modeling proposed in YARN-3926, it extends the current Yarn static resource
modeling to be a flat resource modeling. The end user has the potential to define/schedule
their own resource. I am considering whether we should do further extension to make the resource
modeling to be a hierarchy based modeling. The use case I see for future is the heterogenous
environment with different hardware accelerators (GPU, Intel Xeon Phi, FPGA, etc). For example,
if you treat one GPU as a unit of special resource, the flat resource modeling is good enough.
But we are seeing cases that GPU to be shared between applications, even the application prefer
to allocate certain range of memory inside GPU to avoid cache rotation issue. In this case,
it's hard for scheduler to handle. There is relationship between resource (just like the relationship
between applications in Slider). Scheduler must allocate GPU memory and GPU core on the same

If we do have vision to cover more complicate environments with Yarn, maybe it's time to consider
further extension on the resource modeling together with Slider integration and unified service

> [Umbrella] Simplified API layer for services and beyond
> -------------------------------------------------------
>                 Key: YARN-4793
>                 URL: https://issues.apache.org/jira/browse/YARN-4793
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Gour Saha
>         Attachments: 20160603-YARN-Simplified-V1-API-Examples.adoc, 20160603-YARN-Simplified-V1-API-Layer-For-Services.pdf,
20160603-YARN-Simplified-V1-API-Layer-For-Services.yaml, YARN-4793-yarn-native-services.001.patch
> [See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces
and sub-section 3.3.2 to track the specific sub-item.]
> Bringing a new service on YARN today is not a simple experience. The APIs of existing
frameworks are either too low­ level (native YARN), require writing new code (for frameworks
with programmatic APIs ) or writing a complex spec (for declarative frameworks).
> In addition to building critical building blocks inside YARN (as part of other efforts
at YARN-4692), we should also look to simplifying the user facing story for building services.
Experience of projects like Slider building real-­life services like HBase, Storm, Accumulo,
Solr etc gives us some very good learnings on how simplified APIs for building services will
look like.
> To this end, we should look at a new simple-services API layer backed by REST interfaces.
The REST layer can act as a single point of entry for creation and lifecycle management of
YARN services. Services here can range from simple single-­component apps to the most complex,
multi­-component applications needing special orchestration needs.
> We should also look at making this a unified REST based entry point for other important
features like resource­-profile management (YARN-3926), package-definitions' lifecycle­-management
and service­-discovery (YARN-913 / YARN-4757). We also need to flesh out its relation to
our present much ­lower level REST APIs (YARN-1695) in YARN for application-­submission
and management.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message