helix-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiajun Wang (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (HELIX-672) Improve partition distribution strategy for small clusters
Date Wed, 31 Jan 2018 23:39:00 GMT

     [ https://issues.apache.org/jira/browse/HELIX-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jiajun Wang closed HELIX-672.
       Resolution: Fixed
    Fix Version/s: 0.8.x

We have new algorithm CRUSH-ed released in 0.8.0

> Improve partition distribution strategy for small clusters
> ----------------------------------------------------------
>                 Key: HELIX-672
>                 URL: https://issues.apache.org/jira/browse/HELIX-672
>             Project: Apache Helix
>          Issue Type: Task
>            Reporter: Jiajun Wang
>            Assignee: Jiajun Wang
>            Priority: Major
>             Fix For: 0.8.x
> Current, CRUSH (also multi-round CRUSH) strategy provides even partition distribution
based on statistically randomization. All CRUSH based strategy works well when the cluster
is reasonably large.
> However, in some of our small production clusters, we find the distribution is not good
as expected. The difference between heavy load node and idle node could be 30% in our case.
Given we assign resource according to the max load, we are wasting 30% of resource on the
idle nodes. And it could be worse in other clusters.
> We need to find a new algorithm or improve CRUSH for better evenness for small clusters.
> Note that, meanwhile, we should keep the good features such as minimum movements, deterministic
calculating result.

This message was sent by Atlassian JIRA

View raw message