hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "cntic (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2681) Support bandwidth enforcement for containers while reading from HDFS
Date Wed, 22 Oct 2014 15:27:34 GMT

     [ https://issues.apache.org/jira/browse/YARN-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

cntic updated YARN-2681:
    Attachment: yarn-site.xml

1) Configuration for testing HDFS Bandwidth Enforcement (yarn-site,xml)
- Enable enforcement: yarn.nodemanager.hdfs-bandwidth-enforcement.enable = true
- Port which Datanodes are listening:  yarn.nodemanager.hdfs-bandwidth-enforcement.port =
- Devices' list of machine of data node: yarn.nodemanager.hdfs-bandwidth-enforcement.devices
= lo, eth0
- Interval for checking new tc config from persistence (ms): yarn.nodemanager.hdfs-bandwidth-enforcement.check-tc-config-interval
= 1000
- Since  only the API for Resource has been upgrated to get/set HDFS Bandwidth Enforcement,
but the ResourceRequest has not been implemented yet, for test purpose, the rate and burst
use to define tc class can be given in YARN configuration file:
   + test rate will be writen to an HDFS file: yarn.nodemanager.hdfs-bandwidth-enforcement.test-rate-file
= test-rate-file
   + test rate file example content : 30mbps ( rate unit:  kbps, mbps, kbit, mbit. See also:
   + test burst : yarn.nodemanager.hdfs-bandwidth-enforcement.test-burst = 50Kb

2) The patch is tested by running TestDFSIO-read

> Support bandwidth enforcement for containers while reading from HDFS
> --------------------------------------------------------------------
>                 Key: YARN-2681
>                 URL: https://issues.apache.org/jira/browse/YARN-2681
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, nodemanager, resourcemanager
>    Affects Versions: 2.5.1
>         Environment: Linux
>            Reporter: cntic
>         Attachments: HADOOP-2681.patch, HADOOP-2681.patch, Traffic Control Design.png,
> To read/write data from HDFS on data node, applications establise TCP/IP connections
with the datanode. The HDFS read can be controled by setting Linux Traffic Control  (TC) subsystem
on the data node to make filters on appropriate connections.
> The current cgroups net_cls concept can not be applied on the node where the container
is launched, netheir on data node since:
> -   TC hanldes outgoing bandwidth only, so it can be set on container node (HDFS read
= incoming data for the container)
> -   Since HDFS data node is handled by only one process,  it is not possible to use net_cls
to separate connections from different containers to the datanode.
> Tasks:
> 1) Extend Resource model to define bandwidth enforcement rate
> 2) Monitor TCP/IP connection estabilised by container handling process and its child
> 3) Set Linux Traffic Control rules on data node base on address:port pairs in order to
enforce bandwidth of outgoing data
> Concept:
> http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf

This message was sent by Atlassian JIRA

View raw message