helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Figuring out how to use Helix
Date Fri, 24 Apr 2015 15:39:25 GMT
Hi Jose,

Your minions are very similar to the Helix Agent. But there is slight
difference, Helix agents were used to launch non java processes but in your
case you plan to launch jvm applications on demand.

Your description makes sense. Applications can also modeled as resources.

Here is how you can model it.

There are two types of Participants : Minions and Application  Processes.

At first, you start one Minion on each machine. There will be one resource
created to manage the minion. Lets call it MINION_RESOURCE.

Each  minion that is started and joins the cluster and tags itself as
MINION. We will see why we tag it as MINION in the next section.

There are two ways to model each Application Process, create one uber
resource for all Processes or one per Application Group. For now, I will go
with uber resource.
You create a new uber resource called JVM_APPS and tag it as MINION. The
idea is all sub parts of this resource will be assigned to minions. When
the minions get a state transition per application, they simply launch that

Lets say you want to start 5 application processes/jvms name them as
MyApp_0, .. MyApp_1.  You can add them to the JVM_APPS resource. The
assignment mapping of MyApp to a specific MINION can be automatic or custom
by using the appropriate helix rebalancer (See AUTO, SEMI AUTO, CUSTOM
idealstate modes).

When each minion launches a JVM, the JVM process registers itself in the
cluster and tags it self as MyApp. After this you can create a MyApp
resource (tag it has MyApp) and configure number of  partitions, replicas
etc. This partitions/replicas of this resource will only be assigned to
these nodes.

Hope that make sense.

This will be a great recipe :-)

Kishore G

On Fri, Apr 24, 2015 at 3:14 AM, Half Warp <halfwarp@gmail.com> wrote:

> Hi!
> I'm trying to use Helix as the core of a project I'm starting. I'll start
> by describing the project
> in simple terms.
> The main goal of this project is to have a way to launch jobs (which will
> actually be new JVM processes)
> in a cluster of nodes (physical machines). The idea is to have dummy a
> dummy Java process (which I call Minion) in each of
> the nodes. From an external client, I wanted to them launch applications
> to be run on nodes. The way I was thinking
> about it was to first decide in which nodes the Application and its
> replicas would run, then tell the Minion processes running
> on those nodes to start the job as a separate JVM process (using a message
> queue approach or something to be decided).
> How could this map to Helix? At first sight I thought that Minions could
> be Helix Participants and Applications could be Helix Resources.
> Upon manually starting Minions on each physical machine, each would
> register itself as a Helix Participant. In order to start an
> Application in the cluster, a client would create a new Helix Resource,
> configure its replication and partitioning schema, and add it to the
> cluster. Helix would then decide in which Minions (Helix Participants) the
> Application would run. I'd then lookup the list of Minion (Helix
> Participants) the resource was assigned to (using Helix's External View
> maybe), and send a message to those Minions to tell them
> to start a new JVM process.
> So far I think this more or less matches Helix's model, right?
> However, I'm not sure how to model Application processes. I still want
> them to be part of the cluster, but they don't seem to fit neither the
> Participant nor the Resource concept... I would like to be aware when they
> fail.
> Sorry for the long description,
> Regards,
> José

View raw message