helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Is zookeeper for Helix has standalone JAVA process?
Date Sat, 15 Jun 2013 00:46:12 GMT
Hi Erwin,

Yes thats possible. Good idea.

thanks,
Kishore G


On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <erwin@optinity.com> wrote:

> Kishore,
>
> Thank you a lot for detailed explanation.
>
> I thought to define in configuration of each participant whether it will
> act as controller or Zookeeper as well. All participants will know those
> controllers and zookeepers that defined.
> For example I will define in 5 participants that they also should act as
> controller and zookeepers as well on the same process of the participants.
> Is it possible?
>
> Cheers,
> Erwin
> On Jun 14, 2013 6:21 PM, "kishore g" <g.kishore@gmail.com> wrote:
>
>> Hi Erwin,
>>
>> Here are the options:
>>
>> 1. participant controller and zookeeper on one process on all 50 boxes:
>> This is not possible, you need only 3 or 5 zookeepers running and all
>> participant/controllers need to know the machinename: port of the zookeeper
>> boxes upfront.
>>
>> 2. separate zookeeper but participant and controller in one process: This
>> is possible, controller can be started as part of all participants but only
>> one controller will be active. This is fine until you have like 100 nodes,
>> after which its better to have a separate controller.
>>
>> 3. zookeeper and controller on same box but separate process,
>> participants in different: This is possible, note that once you this
>> configuration all additional clusters you deploy can share the same
>> zookeeper/controller. So if you plan to have many clusters this is a better
>> option and much easier to manage the clusters. You can use standard
>> commands to start zookeeper and helix controller.
>>
>> 4. zookeeper and controller on same box and in one process: This also
>> works and the advantages of 3 holds good here as well. You will have to
>> start
>> both of them programmatically. Any time you restart a process, both
>> zookeeper and controller will be upgraded.
>>
>>
>> My preference is 3) 4) 2), you cannot do 1).
>>
>> I agree with you that it requires additional monitoring but actual
>> experience suggests that co-locating zookeeper with participants causes
>> un-wanted problems. For example, the participants are heavily loaded and
>> your application might have large GC pauses/consume al cpu/memory on that
>> box. This will adversely impact the functionality of zookeeper.
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <erwin@optinity.com>wrote:
>>
>>> My question is, whether is it possible to  collocate participant,
>>> controller and zookeeper on one process, is there any limitations in term
>>> of Helix or Zookeeper?
>>> Any direction regarding above approach would be appreciated.
>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <mingfang@mac.com> wrote:
>>>
>>>> That's not possible at this time.
>>>> ZAC currently must run as a standalone process and only allow one
>>>> instance.
>>>>
>>>> Sent from my iPad
>>>>
>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <erwin@optinity.com> wrote:
>>>>
>>>> If I have 40 app server instances (JVMs) could I run participants on
>>>> all of them, controller and zookeeper just on 5 of then, but on the same
>>>> process of the 5 participants?
>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mingfang@mac.com> wrote:
>>>>
>>>>> ZAC currently runs Zookeeper in standalone mode.
>>>>> Therefore you can only run one instance.
>>>>> If you also run the participant in it then that limits you to one
>>>>> instance of the participant also.
>>>>>
>>>>>
>>>>> Sent from my iPad
>>>>>
>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <erwin@optinity.com>
wrote:
>>>>>
>>>>> Hi Ming,
>>>>>
>>>>> Thank you for fast turnaround response.
>>>>>
>>>>> Could I run my app that uses Helix (participant), controller and
>>>>> zookeeper on the same process?
>>>>>
>>>>> Cheers,
>>>>> Erwin
>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mingfang@mac.com> wrote:
>>>>>
>>>>>> Erwin
>>>>>>
>>>>>> Here is an example of running Zookeeper, and Controller in the same
>>>>>> process,
>>>>>>
>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>>>> We have this in production.
>>>>>>
>>>>>> Sent from my iPad
>>>>>>
>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <erwin@optinity.com>
>>>>>> wrote:
>>>>>>
>>>>>> I know that it's better that participants, controller and zookeeper
>>>>>> will have separate java processes but we have more then 40 boxes
with app
>>>>>> server that we'd like to manage them with Helix Leader election and
cannot
>>>>>> add more processes because each additional java process requires
management
>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid the additional
>>>>>> java processes. We'll not manage to approve additional processes
>>>>>>
>>>>>> Your insight would appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Erwin
>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g.kishore@gmail.com>
wrote:
>>>>>>
>>>>>>> Hi Erwin,
>>>>>>>
>>>>>>> This is how I would setup your cluster. Allocate 3 boxes that
only
>>>>>>> run zookeeper and Helix controller. Its better that they are
two separate
>>>>>>> process. These boxes need not be powerful but isolating zookeeper
and helix
>>>>>>> controller from actual server is a good deployment practice.
See
>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon
how to use zookeeper clustering ensemble.
>>>>>>>
>>>>>>>
>>>>>>> Run the real servers on the rest of the boxes. Do not start
>>>>>>> zookeeper as part of these processes, instead simply provide
the zookeeper
>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke Helix
Api.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kishore G
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <erwin@optinity.com>wrote:
>>>>>>>
>>>>>>>> Hi Kishore,
>>>>>>>>
>>>>>>>> We have  more then 50 nodes.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Erwin
>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g.kishore@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>> Hi Erwin,
>>>>>>>>>
>>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Kishore G
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi <erwin@optinity.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hello Maters,
>>>>>>>>>>
>>>>>>>>>> We don't want to have separate/specific JAVA process
for
>>>>>>>>>> Zookeeper.
>>>>>>>>>> As I figured out from the Master-Slave sample we
can start the
>>>>>>>>>> Zookeeper programmatically using the Zookeeper API.
>>>>>>>>>>
>>>>>>>>>> I have several questions:
>>>>>>>>>>
>>>>>>>>>> 1. If we are using the Zookeeper API for initializing
the
>>>>>>>>>> Zookeeper from the Helix app (out app that uses Helix)
would we have
>>>>>>>>>> specific process for Zookeeper? Would we have one
process for our app and
>>>>>>>>>> another for the Zookeeper?
>>>>>>>>>> What happens if we'll not initialize the Zookeeper
from the Helix
>>>>>>>>>> app programmatically but using the commands, whether
then would we have
>>>>>>>>>> specific process for Zookeeper?
>>>>>>>>>>
>>>>>>>>>> 2. If we'll decide to initialize Zookeeper from Helix
app,
>>>>>>>>>> programmatically, How can we use Zookeeper clustering
(ensemble) , is there
>>>>>>>>>> any API for Zookeeper clustering, any direction would
highly appreciated.
>>>>>>>>>>
>>>>>>>>>> 3. Is it possible to start the Zookeeper by using
commands and
>>>>>>>>>> not initializing specific process for it but using
the Helix app (our app
>>>>>>>>>> that uses Helix API)?
>>>>>>>>>>
>>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance,
>>>>>>>>>> Erwin Karbasi
>>>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>

Mime
View raw message