apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Priyanka Gugale (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2045) Add bandwidth control feature to Apex
Date Tue, 05 Apr 2016 12:35:25 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226158#comment-15226158

Priyanka Gugale commented on APEXMALHAR-2045:

high level design we are considering, We would have following components:
1. BandwidthManager
This keeps track of current bandwidth usage of system and takes decision if requested data
bandwidth can be used right away or not. To do this it used Leaky bucket algorithm where it
emits data as long as it has not overused bandwidth (i.e. bandwidth consumption is >=0)
and then wait to accumulate bandwidth for a while (till bandwidth goes from -ve value to +ve).

2. BandwidthLimitingInputOperator
Any Input operator which want to implement bandwidth restriction should implement BandwidthLimitingInputOperator.
The operator have abstract method  to initialize instance of BandwidthManager and a method
to emit tuple with bandwidth restriction to emit tuples as per available bandwidth.

3. BandwidthPartitioner
Bandwidth partitioner is introduced for static partitioning. If static partitioning is used
by default StatelessPartitioner class is initialized. With bandwidth restriction we want to
equally divide bandwidth amongst available partitions. BandwidthPartitioner should take care
of it. It extends StatelessPartitioner, it just sets right bandwidth on all partitions after
StatelessPartitioner creates/deletes partitiolns. In case of dynamic partitioning the operator
implementing definePartitions, should take care of bandwidth distribution.

This design takes care of basic bandwidth restriction, also takes care of partitions by equally
distributing available bandwidth among all partitions. Also this is open enough to do further
modifications to take care of complex situations. 

> Add bandwidth control feature to Apex
> -------------------------------------
>                 Key: APEXMALHAR-2045
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2045
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: Priyanka Gugale
>            Assignee: Priyanka Gugale
>  bandwidth restrictions on input operator for number of bytes to be consumed per second.

This message was sent by Atlassian JIRA

View raw message