helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erwin Karbasi <er...@optinity.com>
Subject Re: Is zookeeper for Helix has standalone JAVA process?
Date Sat, 22 Jun 2013 22:31:12 GMT
Hi Kishore,

Do you have any documentation/sample regarding zookeeper implementation
programmatically?

Thanks,
Erwin
On Jun 15, 2013 11:28 AM, "Erwin Karbasi" <erwin@optinity.com> wrote:

> Hi kishore,
>
> Thank you for preservance and helpful support.
>
> Do you have any direction how can we implement zookeeper clustering
> programmatically in my aforementioned suggestion?
> As far as I know we need to use quorum.  Do you know any example that can
> help us.
>
> Thanks,
> Erwin
> On Jun 15, 2013 3:46 AM, "kishore g" <g.kishore@gmail.com> wrote:
>
>> Hi Erwin,
>>
>> Yes thats possible. Good idea.
>>
>> thanks,
>> Kishore G
>>
>>
>> On Fri, Jun 14, 2013 at 3:42 PM, Erwin Karbasi <erwin@optinity.com>wrote:
>>
>>> Kishore,
>>>
>>> Thank you a lot for detailed explanation.
>>>
>>> I thought to define in configuration of each participant whether it will
>>> act as controller or Zookeeper as well. All participants will know those
>>> controllers and zookeepers that defined.
>>> For example I will define in 5 participants that they also should act as
>>> controller and zookeepers as well on the same process of the participants.
>>> Is it possible?
>>>
>>> Cheers,
>>> Erwin
>>> On Jun 14, 2013 6:21 PM, "kishore g" <g.kishore@gmail.com> wrote:
>>>
>>>> Hi Erwin,
>>>>
>>>> Here are the options:
>>>>
>>>> 1. participant controller and zookeeper on one process on all 50 boxes:
>>>> This is not possible, you need only 3 or 5 zookeepers running and all
>>>> participant/controllers need to know the machinename: port of the zookeeper
>>>> boxes upfront.
>>>>
>>>> 2. separate zookeeper but participant and controller in one process:
>>>> This is possible, controller can be started as part of all participants but
>>>> only one controller will be active. This is fine until you have like 100
>>>> nodes, after which its better to have a separate controller.
>>>>
>>>> 3. zookeeper and controller on same box but separate process,
>>>> participants in different: This is possible, note that once you this
>>>> configuration all additional clusters you deploy can share the same
>>>> zookeeper/controller. So if you plan to have many clusters this is a better
>>>> option and much easier to manage the clusters. You can use standard
>>>> commands to start zookeeper and helix controller.
>>>>
>>>> 4. zookeeper and controller on same box and in one process: This also
>>>> works and the advantages of 3 holds good here as well. You will have to
>>>> start
>>>> both of them programmatically. Any time you restart a process, both
>>>> zookeeper and controller will be upgraded.
>>>>
>>>>
>>>> My preference is 3) 4) 2), you cannot do 1).
>>>>
>>>> I agree with you that it requires additional monitoring but actual
>>>> experience suggests that co-locating zookeeper with participants causes
>>>> un-wanted problems. For example, the participants are heavily loaded and
>>>> your application might have large GC pauses/consume al cpu/memory on that
>>>> box. This will adversely impact the functionality of zookeeper.
>>>>
>>>> Thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jun 14, 2013 at 5:44 AM, Erwin Karbasi <erwin@optinity.com>wrote:
>>>>
>>>>> My question is, whether is it possible to  collocate participant,
>>>>> controller and zookeeper on one process, is there any limitations in
term
>>>>> of Helix or Zookeeper?
>>>>> Any direction regarding above approach would be appreciated.
>>>>> On Jun 14, 2013 3:23 PM, "Ming Fang" <mingfang@mac.com> wrote:
>>>>>
>>>>>> That's not possible at this time.
>>>>>> ZAC currently must run as a standalone process and only allow one
>>>>>> instance.
>>>>>>
>>>>>> Sent from my iPad
>>>>>>
>>>>>> On Jun 14, 2013, at 7:35 AM, Erwin Karbasi <erwin@optinity.com>
>>>>>> wrote:
>>>>>>
>>>>>> If I have 40 app server instances (JVMs) could I run participants
on
>>>>>> all of them, controller and zookeeper just on 5 of then, but on the
same
>>>>>> process of the 5 participants?
>>>>>> On Jun 14, 2013 2:29 PM, "Ming Fang" <mingfang@mac.com> wrote:
>>>>>>
>>>>>>> ZAC currently runs Zookeeper in standalone mode.
>>>>>>> Therefore you can only run one instance.
>>>>>>> If you also run the participant in it then that limits you to
one
>>>>>>> instance of the participant also.
>>>>>>>
>>>>>>>
>>>>>>> Sent from my iPad
>>>>>>>
>>>>>>> On Jun 14, 2013, at 7:23 AM, Erwin Karbasi <erwin@optinity.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Ming,
>>>>>>>
>>>>>>> Thank you for fast turnaround response.
>>>>>>>
>>>>>>> Could I run my app that uses Helix (participant), controller
and
>>>>>>> zookeeper on the same process?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Erwin
>>>>>>> On Jun 14, 2013 2:16 PM, "Ming Fang" <mingfang@mac.com>
wrote:
>>>>>>>
>>>>>>>> Erwin
>>>>>>>>
>>>>>>>> Here is an example of running Zookeeper, and Controller in
the same
>>>>>>>> process,
>>>>>>>>
>>>>>>>> https://github.com/mingfang/apache-helix/blob/master/helix-example/src/main/java/org/apache/helix/examples/ZAC.java
>>>>>>>> We have this in production.
>>>>>>>>
>>>>>>>> Sent from my iPad
>>>>>>>>
>>>>>>>> On Jun 14, 2013, at 3:34 AM, Erwin Karbasi <erwin@optinity.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I know that it's better that participants, controller and
zookeeper
>>>>>>>> will have separate java processes but we have more then 40
boxes with app
>>>>>>>> server that we'd like to manage them with Helix Leader election
and cannot
>>>>>>>> add more processes because each additional java process requires
management
>>>>>>>> (monitoring, alarms, KPI, etc...), so we prefer to avoid
the additional
>>>>>>>> java processes. We'll not manage to approve additional processes
>>>>>>>>
>>>>>>>> Your insight would appreciated.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Erwin
>>>>>>>> On Jun 14, 2013 9:58 AM, "kishore g" <g.kishore@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>> Hi Erwin,
>>>>>>>>>
>>>>>>>>> This is how I would setup your cluster. Allocate 3 boxes
that only
>>>>>>>>> run zookeeper and Helix controller. Its better that they
are two separate
>>>>>>>>> process. These boxes need not be powerful but isolating
zookeeper and helix
>>>>>>>>> controller from actual server is a good deployment practice.
See
>>>>>>>>> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetupon
how to use zookeeper clustering ensemble.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Run the real servers on the rest of the boxes. Do not
start
>>>>>>>>> zookeeper as part of these processes, instead simply
provide the zookeeper
>>>>>>>>> address (zk1:port1,zk2:port2,zk3:port3) when you invoke
Helix Api.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kishore G
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 13, 2013 at 10:05 PM, Erwin Karbasi <
>>>>>>>>> erwin@optinity.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kishore,
>>>>>>>>>>
>>>>>>>>>> We have  more then 50 nodes.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Erwin
>>>>>>>>>> On Jun 14, 2013 1:51 AM, "kishore g" <g.kishore@gmail.com>
wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Erwin,
>>>>>>>>>>>
>>>>>>>>>>> How many nodes do you have in your cluster ?
>>>>>>>>>>>
>>>>>>>>>>> thanks,
>>>>>>>>>>> Kishore G
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 13, 2013 at 1:43 PM, Erwin Karbasi
<
>>>>>>>>>>> erwin@optinity.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello Maters,
>>>>>>>>>>>>
>>>>>>>>>>>> We don't want to have separate/specific JAVA
process for
>>>>>>>>>>>> Zookeeper.
>>>>>>>>>>>> As I figured out from the Master-Slave sample
we can start the
>>>>>>>>>>>> Zookeeper programmatically using the Zookeeper
API.
>>>>>>>>>>>>
>>>>>>>>>>>> I have several questions:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. If we are using the Zookeeper API for
initializing the
>>>>>>>>>>>> Zookeeper from the Helix app (out app that
uses Helix) would we have
>>>>>>>>>>>> specific process for Zookeeper? Would we
have one process for our app and
>>>>>>>>>>>> another for the Zookeeper?
>>>>>>>>>>>> What happens if we'll not initialize the
Zookeeper from the
>>>>>>>>>>>> Helix app programmatically but using the
commands, whether then would we
>>>>>>>>>>>> have specific process for Zookeeper?
>>>>>>>>>>>>
>>>>>>>>>>>> 2. If we'll decide to initialize Zookeeper
from Helix app,
>>>>>>>>>>>> programmatically, How can we use Zookeeper
clustering (ensemble) , is there
>>>>>>>>>>>> any API for Zookeeper clustering, any direction
would highly appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Is it possible to start the Zookeeper
by using commands and
>>>>>>>>>>>> not initializing specific process for it
but using the Helix app (our app
>>>>>>>>>>>> that uses Helix API)?
>>>>>>>>>>>>
>>>>>>>>>>>> I hope my questions are enough clear.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>> Erwin Karbasi
>>>>>>>>>>>> AT&T, Senior Software Architect
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>
>>

Mime
View raw message