helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Helix 0.6.2- Messages on startup
Date Thu, 20 Feb 2014 22:20:16 GMT
Hi Sandeep,

The first part is correct, if you add a resource X and say it should be
assigned to two nodes, Helix will chose 2 of A,B, C to host the resources.

Regarding second part, Helix does not interfere in the data path i.e
request/response. Helix does not have the peer-peer communication as of now
(we have plans to work on helix-ipc module), that will allow peer to peer
message exchange.

Do all install requests get routed to the Master and then pushed to the
relevant slaves?
In order to route request to master, you will need to use a spectator that
will observe the cluster state and provide you information about the
current master. Helix will not route the request to master nor will the
master push the requests to all slaves.

>From the initial description, I thought your requirement was to manage the
deployment topology (mapping of component to Node) dynamically. Apart from
the peer-peer message exchanging other requirements match the Helix
features.

We are available on IRC #apachehelix. It is probably easier to discuss the
details on IRC.


thanks,
Kishore G








On Thu, Feb 20, 2014 at 1:48 PM, Sandeep Nayak <osgigeek@gmail.com> wrote:

> So let me take a few steps back and put forth a use-case which might
> throw some light on what I am pursuing.
>
> Lets say I have 3 nodes A, B and C and I want to create a peer-peer
> cluster so that all of them join a cluster (like the jgroups model). I
> then want to add resources to A or B or C and expect the specific node
> to push the resource information to the cluster based on the
> resource's replication strategy. So lets say X is installed on A and X
> states I want to be available on exactly 2 nodes then I imagine the
> cluster will pick two of A, B or C to hold the resource.
>
> So a few questions, please bear with my ignorance
>
> * In this use-case a good fit for Helix, I would love to get the
> replication strategy through Helix but if I am dealing with a inherent
> master-slave model which is not peer-peer am I using Helix for the
> wrong thing?
> * Do all install requests get routed to the Master and then pushed to
> the relevant slaves?
>
> Thanks,
>
> Sandeep
>
> On Thu, Feb 20, 2014 at 1:07 PM, Kanak Biscuitwala <kanak.b@hotmail.com>
> wrote:
> > Are you talking about the controller or the participants (instances)? If
> you would like to see if the controller is running, check under
> /clusterName/CONTROLLER to see if there is a ZNode called LEADER.
> >
> > For participants, nothing will be assigned until resources are defined.
> Ideal states and external views are created and used on a per-resource
> level. Helix can manage multiple distributed resources at once, so you need
> to add a resource to define things like the state model it follows, the
> assignment policy (rebalance mode), and number of partitions and replicas.
> This step generates the ideal state for the resource. The quickstart and
> tutorial detail these steps.
> >
> > Kanak
> > ----------------------------------------
> >> Date: Thu, 20 Feb 2014 12:53:08 -0800
> >> Subject: Re: Helix 0.6.2- Messages on startup
> >> From: osgigeek@gmail.com
> >> To: user@helix.apache.org
> >>
> >> Thanks Jason for the pointers. I do see the nodes and InstanceConfigs
> >> but there is no indication of which is master, do I need to add
> >> resources to the cluster for that information to surface? If nodes are
> >> added to the cluster should I be able to see which is the master or
> >> leader and which are the followers?
> >>
> >> Sandeep
> >>
> >> On Thu, Feb 20, 2014 at 11:51 AM, Zhen Zhang <zzhang@linkedin.com>
> wrote:
> >>> For checking external-view, you can use zoo inspector:
> >>> http://helix.apache.org/0.7.0-incubating-docs/Quickstart.html
> >>>
> >>> The zookeeper path is for the external view is
> >>> /{clusterName}/{externalView}/{resourceName}
> >>>
> >>> If you want do it programmatically, you can use spectator:
> >>> http://helix.apache.org/0.7.0-incubating-docs/tutorial_spectator.html
> >>>
> >>>
> >>> The time for a node to transit to its target state depends on how long
> the
> >>> state transition takes.
> >>>
> >>> In normal case, if the external-view matches the ideal-state, it
> indicates
> >>> the cluster converges.
> >>>
> >>> Thanks,
> >>> Jason
> >>>
> >>> On 2/20/14 11:29 AM, "Sandeep Nayak" <osgigeek@gmail.com> wrote:
> >>>
> >>>>I probably missed it in the documentation, but how can I check the
> >>>>external view? Also, is there a time period after which the first node
> >>>>logs that it has identified itself as the leader? Just wanted to know
> >>>>what is the indicator that things work and I do have a leader and
> >>>>follower in the cluster?
> >>>>
> >>>>Thanks,
> >>>>
> >>>>Sandeep
> >>>>
> >>>>On Thu, Feb 20, 2014 at 9:34 AM, Kanak Biscuitwala <
> kanak.b@hotmail.com>
> >>>>wrote:
> >>>>> We didn't, but now we do.
> >>>>> ________________________________
> >>>>>> Date: Thu, 20 Feb 2014 09:26:10 -0800
> >>>>>> Subject: Re: Helix 0.6.2- Messages on startup
> >>>>>> From: g.kishore@gmail.com
> >>>>>> To: user@helix.apache.org
> >>>>>>
> >>>>>> Do we have a jira for this ?
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 19, 2014 at 11:20 PM, Kanak Biscuitwala
> >>>>>> <kanak.b@hotmail.com<mailto:kanak.b@hotmail.com>>
wrote:
> >>>>>> Hi Sandeep,
> >>>>>>
> >>>>>> Does Helix still generate an external view correctly despite
these
> log
> >>>>>> messages?
> >>>>>>
> >>>>>> I believe the first message is a known issue: the callbacks
are
> >>>>>> initialized once by the HelixManager implementation, and then
> >>>>>> initialized again when the controller becomes leader (for the
first
> >>>>>> controller, these will happen very close to one another). It's
> >>>>>> generally a harmless message.
> >>>>>>
> >>>>>> The second one indicates that the controller is getting events
> (which
> >>>>>> path was changed to trigger the event?), but hasn't yet become
> leader.
> >>>>>> When it does, it will invoke the pipeline and do the right thing.
> >>>>>>
> >>>>>> I think in either case, a race is causing the log messages to
> appear,
> >>>>>> but it shouldn't affect the correctness of the Helix controller
> >>>>>> operation.
> >>>>>>
> >>>>>> Kanak
> >>>>>>
> >>>>>> Date: Wed, 19 Feb 2014 23:13:19 -0800
> >>>>>> Subject: Helix 0.6.2- Messages on startup
> >>>>>> From: osgigeek@gmail.com<mailto:osgigeek@gmail.com>
> >>>>>> To: user@helix.apache.org<mailto:user@helix.apache.org>
> >>>>>>
> >>>>>> Hi guys,
> >>>>>>
> >>>>>> So I finally got my setup done with states etc. But when I start
the
> >>>>>> first node I see the following messages
> >>>>>>
> >>>>>> expected types: [CALLBACK, FINALIZE] but was INIT
> >>>>>> I see this for /cluster/Controller,
> >>>>>> /cluster/INSTANCES/Instance/MESSAGES and several other paths.
> >>>>>> The other message I see is
> >>>>>> Cluster manager: first-node is not leader. Pipeline will not
be
> invoked
> >>>>>> Any ideas on what I am missing here?
> >>>>>> Thanks,
> >>>>>> Sandeep
> >>>>>>
> >>>
>

Mime
View raw message