helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Half Warp <halfw...@gmail.com>
Subject Figuring out how to use Helix
Date Fri, 24 Apr 2015 10:14:14 GMT

I'm trying to use Helix as the core of a project I'm starting. I'll start
by describing the project
in simple terms.

The main goal of this project is to have a way to launch jobs (which will
actually be new JVM processes)
in a cluster of nodes (physical machines). The idea is to have dummy a
dummy Java process (which I call Minion) in each of
the nodes. From an external client, I wanted to them launch applications to
be run on nodes. The way I was thinking
about it was to first decide in which nodes the Application and its
replicas would run, then tell the Minion processes running
on those nodes to start the job as a separate JVM process (using a message
queue approach or something to be decided).

How could this map to Helix? At first sight I thought that Minions could be
Helix Participants and Applications could be Helix Resources.
Upon manually starting Minions on each physical machine, each would
register itself as a Helix Participant. In order to start an
Application in the cluster, a client would create a new Helix Resource,
configure its replication and partitioning schema, and add it to the
cluster. Helix would then decide in which Minions (Helix Participants) the
Application would run. I'd then lookup the list of Minion (Helix
Participants) the resource was assigned to (using Helix's External View
maybe), and send a message to those Minions to tell them
to start a new JVM process.

So far I think this more or less matches Helix's model, right?

However, I'm not sure how to model Application processes. I still want them
to be part of the cluster, but they don't seem to fit neither the
Participant nor the Resource concept... I would like to be aware when they

Sorry for the long description,



View raw message