storm-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (STORM-132) Default Storm scheduler should evenly use resources when managing multiple topologies
Date Tue, 18 Feb 2014 15:30:19 GMT

    [ https://issues.apache.org/jira/browse/STORM-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904148#comment-13904148
] 

ASF GitHub Bot commented on STORM-132:
--------------------------------------

GitHub user xiaokang opened a pull request:

    https://github.com/apache/incubator-storm/pull/36

    STORM-132 sort supervisor by free slot in desending order

    Sort supervisor by free slot in desending order in EvenScheduler to schedule more evenly
between supervisor

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/xiaokang/incubator-storm STORM-132

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-storm/pull/36.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #36
    
----
commit 6b275d95fbdbc8374a215ecb2551f0fca3438d81
Author: Kang Xiao <kxiao.tiger@gmail.com>
Date:   2014-02-18T15:23:50Z

    STORM-132 sort supervisor by free slot in desending order
    in EvenScheduler to schedule more evenly between supervisor

----


> Default Storm scheduler should evenly use resources when managing multiple topologies
> -------------------------------------------------------------------------------------
>
>                 Key: STORM-132
>                 URL: https://issues.apache.org/jira/browse/STORM-132
>             Project: Apache Storm (Incubating)
>          Issue Type: Improvement
>            Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/359
> Currently, a single topology is evenly spread across the cluster, but this is not the
case for multiple topologies (it targets one node first, then the rest). The default scheduler
should order the hosts for scheduling in terms of which has the least slots used.
> ----------
> lyogavin: To confirm, we want to firstly look at the number of used slots on each host.
Firstly try to balance the used slot number across all the hosts. For the assignment inside
each host, we also try to evenly balance the usage of each slot. Am I understanding correctly?
> When i coding, i realized it's actually a little complicated. Looks like there are many
policies here we want to consider:
> 1. Evenness of resource usage. (Do we want to evaluate evenness according to number of
slots used in each host or the number of executors? Maybe number of executors is better, but
also make it very complicated)
> 2. Least rescheduling. We probabaly also want to make the assignment change as less as
possible. Looks like this is why DefaultScheduler.bad-slots is coded the current way.
> 3. Number of workers.
> Then the question is how do we prioritize those policies. Sometimes they conflict to
each other. For example, sometimes the most even distribution may need the most reassignment.
> Any thoughts?
> ----------
> xumingming: @lyogavin I think you might have over-thought this, as @nathanmarz already
confirmed, you just need to update EvenScheduler.sort-slots to make sure the slots in the
least used node appear first in the available-slots list.
> ----------
> lyogavin: Thanks James. I got what you mean. So i'll not worry about the usage of the
executor usage. Only consider the balance of slots usage.
> But I think just simply change the sort-slots to sort the slots based on slots usage
may still not work too well. For example, let's say there are 2 hosts, with 10 slots on each.
Say 1st one used 1 slots, 2nd one used 2 slots, and we want to assign another topology with
8 workers. If we simply sort the slots based on usage, we'll end up with the list [0 0 0 0
0 0 0 0 1 1], then the new assignment we'll get would be all from the 1st host. Not balanced.
> So in the above pull request, I implemented a solution in the way similar to watershed
algorithm. It would firstly pick the slots from least used host, until that host uses the
same number of slots as the second least used slots. Then it evenly picks slots from the 2
least used hosts until reaches the 3rd one. Iterating this way, we can get the best balanced
assignment.
> What do you think?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message