helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shekhar Bansal <shekhar0...@yahoo.com>
Subject HDFS read load distribution using helix
Date Mon, 19 Jun 2017 14:24:15 GMT
I have a standalone java app(containerised), it reads data from HDFS, does some transformations
and write data to remote storage. I want to make it scalable by launching multiple instances
of this java app. My problem is how to assign tasks among these instances. can helix solve
this problem?
If yes, can you please help me with following    
   - I referred helix quickstart example and created 1 resource per file but node1 got assigned
master for all resources, is it because of simple StateModelDefinition used in quickstart
example or I am using it wrong way or is it some limitation of helix   

   - I want to avoid running a separate controller process, so If I run start controller as
part of setup will helix be able to elect master controller (in standalone mode), is it advisable
to run tens of controllers in distributed mode.   

   - I schedule my app every five minutes using kubernetes cron, is it advisable to use helix
for such short lived processes

View raw message