incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: can we dynamically change nbTasks?
Date Mon, 10 Jun 2013 12:36:01 GMT
S4 clusters are currently statically partitioned.

We have been looking at how to enable some elasticity by providing means to (efficiently)
repartition the cluster while it's running. 

This is the objective of S4-110 (https://issues.apache.org/jira/browse/S4-110) where we leverage
Apache Helix for that purpose. It's not that simple though and we don't have a working implementation
yet. Any help would be welcome!

Regards,

Matthieu



On Jun 9, 2013, at 08:32 , baojian Zhou wrote:

> when we start a newCluster, we must define the number of Tasks, and i notice that in
TaskSetup.class:
> 
> public void setup(String cluster, int tasks, int initialPort) {
> 		try {
> 			zkclient.createPersistent("/s4/streams", true);
> 		} catch (ZkException ignored) {
> 			// ignore if exists
> 		}
> 
> 		zkclient.createPersistent("/s4/clusters/" + cluster + "/tasks", true);
> 		zkclient.createPersistent("/s4/clusters/" + cluster + "/process", true);
> 		zkclient.createPersistent("/s4/clusters/" + cluster + "/app", true);
> 		for (int i = 0; i < tasks; i++) {
> 			String taskId = "Task-" + i;
> 			ZNRecord record = new ZNRecord(taskId);
> 			record.putSimpleField("taskId", taskId);
> 			record.putSimpleField("port", String.valueOf(initialPort + i));
> 			record.putSimpleField("partition", String.valueOf(i));
> 			record.putSimpleField("cluster", cluster);
> 			zkclient.createPersistent("/s4/clusters/" + cluster + "/tasks/"
> 					+ taskId, record);
> 		}
> 	}
> 
> 
> so my question is:
> if i want to change the nbTasks dynamically and then add the corresponding S4Nodes ,
do i just need to write some codes in the  TaskSetUp class´╝č or i have to change some other
codes


Mime
View raw message