hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7468) Provide means for container network policy control
Date Tue, 14 Nov 2017 00:13:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250526#comment-16250526

Vinod Kumar Vavilapalli commented on YARN-7468:

bq. This is not a request for full-scale software-defined-networking integration into YARN.
Glad you pointed this out! Though it will be interesting to see how such an integration will
look like and what fundamental building blocks will be needed in YARN.

bq. 1A. We would setup iptables rules statically beforehand to ensure traffic for the various
YARN agreed upon cgroup contexts, bridge devices or network namespaces could only flow where
we want; we'd do this via out-of-band configuration management – no need for YARN to do
this setup.
If these rules have to be static, they cannot be tied to specific apps, but only to more static
concepts like user-name / group-name or queue name. The NM doesn't know the queue information,
so may be we should stick to user information.

Of course, this means user information is the same on all the machines in the YARN cluster.
This is already be a requirement in secure clusters.

bq. 2. Then, when a user submit's a job, YARN would setup the OS control (cgroup, network
namespace or the bridge interface) for those processes to match the user's name, a queue or
some other deterministic handle. (We would use that handle for our configuration-managed matching
iptables rules which would be pre-configured.)
I think we could use the same underlying linux functionality as that of traffic shaping to
tag the traffic from containers depending on the admin specific rules. To reuse YARN-2140,
we could split the underlying related container-executor functionality into some sort of a
networking module similar to what YARN-6852 did with GPU module (but not cgroups - that part
still remains to be cleaned up).

> Provide means for container network policy control
> --------------------------------------------------
>                 Key: YARN-7468
>                 URL: https://issues.apache.org/jira/browse/YARN-7468
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Clay B.
>            Priority: Minor
> To prevent data exfiltration from a YARN cluster, it would be very helpful to have "firewall"
rules able to map to a user/queue's containers.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message