incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kishore gopalakrishna (JIRA)" <>
Subject [jira] [Commented] (S4-141) Enhance S4 with the elasticity feature
Date Sat, 07 Sep 2013 06:19:52 GMT


kishore gopalakrishna commented on S4-141:

Great write up Kurtt. Definitely the right direction

Agree with Matthieu that we should first get one version relying on distributed storage like
NFS, HDFS, S3, Hbase etc.

However each node having local storage is definitely more appealing. The only problem would
be handling failure which will then require replicating the state. We can definitely do this
by storing the checkpoint data on more than one node. For example we can have a node in standby
state in helix and the leader node when it checkpoints, sends the  checkpoint data over to
in standby node as well. This will allow us to handle failures as well. 

For elasticity, we can do something like you have outlined but with ability to save the checkpoint
data on multiple nodes we can handle failures even during elasticity.

You are right that Helix does not have any rebalancer that moves partitions around based on
health/performance metrics. For small clusters writing health metrics to zookeeper per partition
works but for larger cluster/partitions zk wont scale.But the rebalancer can be agnostic of
where it gets the metrics.

We have a contribution(patch) that uses google-or tools to periodically recompute the mapping
but we should probably do this in next phase.

S4-110 had to clean up some of the existing code in S4. We can start again from trunk. The
main piece of code in S4 that needed change was emitter and listener. Emitter and listener
kind of own the partition routing but ideally it should be the Sender and Receiver that should
keep track of topology changes.

As Flavio suggested, we can break it up into multiple subtasks. I can take a stab at it.

Excited to see this happening!!!


> Enhance S4 with the elasticity feature
> --------------------------------------
>                 Key: S4-141
>                 URL:
>             Project: Apache S4
>          Issue Type: New Feature
>            Reporter: Kurtt.Lin
>         Attachments: S4-141.pdf
>   Original Estimate: 2,016h
>  Remaining Estimate: 2,016h
> Attached is a design documentation draft, which aims at enhancing S4 with the elasticity
> Issues discussed are:
> * introduction to elasticity
> * current status with S4
> * possible roadmap
> * the design
> * implementation with Helix

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message