hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2140) Add support for network IO isolation/scheduling for containers
Date Fri, 13 Jun 2014 13:58:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030643#comment-14030643

Robert Joseph Evans commented on YARN-2140:

We are working on similar things for storm.  I am very interested in your design, because
for any streaming system to truly have a chance on YARN soft guarantees on network I/O are
critical.  There are several big problems with network I/O even if the user can effectively
estimate what they will need.  The first is that the resource is not limited to a single node
in the cluster.  The network has a topology and a bottlekneck can show up at any point in
that topology.  So you may think you are fine because each node in a rack is not scheduled
to be using the full bandwidth that the network card(s) can support.  But you can easily have
saturated the top of rack switch without knowing it.  To solve this problem you effectively
have to know the topology of the application itself.  So that you can schedule the node to
node network connections within that application. if users don't know how much network they
are going to use at a high level, they will never have any idea at a low level.  But then
you also have the big problem of batch being very bursty in its network usage.  The only way
to solve this is going to require network hardware support for prioritizing packets.

But I'll wait for your design before writing too much more.

> Add support for network IO isolation/scheduling for containers
> --------------------------------------------------------------
>                 Key: YARN-2140
>                 URL: https://issues.apache.org/jira/browse/YARN-2140
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>            Assignee: Wei Yan

This message was sent by Atlassian JIRA

View raw message