manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Graeme Seaton <li...@graemes.com>
Subject Re: Zookeeper in Apache ManifoldCF
Date Fri, 27 Jun 2014 10:15:00 GMT
Hi Lalit,

For production use, you will want to spin up your own ZK cluster using 
the instructions on the zookeeper site (as pointed out earlier at least 
3 is recommended)....

You then need to modify the properties.xml file in 
multiprocess-zk-example to point to the list of Zookeeper servers.  You 
also need to modify properties-global.xml with the appropriate global 
settings i.e. logging levels, Postgresql database etc. and then run 
setglobalproperties.sh to register the settings in ZK.

To test that is working, set up a crawl and then tail the manifoldcf.log 
file on each of your nodes to check that they are all crawling in parallel.

HTH,

Graeme

On 25/06/14 12:19, Karl Wright wrote:
> Hi Lalit,
>
> Zookeeper does not use a database; it keeps its stuff in the local 
> file system.  Each Zookeeper node has its own local data, and 
> everything else is socket communication between them.
>
> As for information: http://zookeeper.apache.org/
>
> Karl
>
>
>
> On Wed, Jun 25, 2014 at 6:56 AM, lalit jangra 
> <lalit.j.jangra@gmail.com <mailto:lalit.j.jangra@gmail.com>> wrote:
>
>     Thanks Karl,
>
>     Apologies as i am not very familiar with Zookeeper and trying to
>     figure out on same.
>
>     Is there any more documentation/pointers available for same as
>     that would be more helpful.
>
>     Also i have 2 tomcat servers in cluster, each having MCF 1.5.1
>     setup and configured to point to same PostGreSQL DB & DB is backed
>     up for failover. From your inputs, it seems that we need to
>     configure a separate standalone Zookeeper server which will act as
>     Master and both nodes in cluster will need to work as slaves and
>     talk to standalone Zookeeper master.
>
>     Also the Zookeeper server will have its own DB so either we can
>     host it separately or we can use same Postgres DB?
>
>     Regards.
>
>
>
>     On Wed, Jun 25, 2014 at 11:33 AM, Karl Wright <daddywri@gmail.com
>     <mailto:daddywri@gmail.com>> wrote:
>
>         Hi Lalit,
>
>         1. zookeeper is already spun into MCF.  in fact you start a
>         zookeeper instance when you run the mcf zookeeper example. 
>         They recommend, though, that for failover you have 3
>         instances, etc.
>         2. Looks like the documentation is out of date and something
>         old is left in there.
>         3. Zookeeper is a client/server kind of arrangement.  You need
>         at least ONE zookeeper server, and each cluster member
>         includes a zookeeper client, which is configured to talk with
>         ALL the zookeeper server instances you have.
>         4.  There is ONE database instance; the instance may be
>         supported by failover and redundant Postgresql, but it appears
>         as one instance.  TO get failover from Postgres you need the
>         Enterprise Edition, which costs money.
>
>         Karl
>
>
>
>
>         On Wed, Jun 25, 2014 at 4:47 AM, lalit jangra
>         <lalit.j.jangra@gmail.com <mailto:lalit.j.jangra@gmail.com>>
>         wrote:
>
>             Thanks Karl,
>
>             That was helpful.
>
>             I am setting clustered setup on Tomcats as i was following
>             instructions @
>             http://manifoldcf.apache.org/release/trunk/en_US/how-to-build-and-deploy.html#Simplified+multi-process+model+using+ZooKeeper-based+synchronization
>             and i need some suggestions here.
>
>             1. Do we need to download zookeeper and put it in
>             multiprocess-zk-example folder or it is already spun into
>             MCF and we are good to go?
>             2. It says all jars under /processes /should be put into
>             classpath but i can not see any /processes /folder under MCF?
>             3. Do we need to setup Zookeeper on both nodes or only at
>             one node, i assume we need to do on both nodes ?
>             4. Do we also need to setup databases separately on both
>             nodes again. Also can we setup Zookeeper DB using same
>             PostGreSQL or it will use its own HSQL DB?
>
>             Finally how can i test that my Zookeeper is setp and ready
>             to roll?
>
>             Thanks for your help.
>
>             Regards.
>
>
>             On Tue, Jun 24, 2014 at 1:56 PM, Karl Wright
>             <daddywri@gmail.com <mailto:daddywri@gmail.com>> wrote:
>
>                 Hi Lalit,
>                 ZooKeeper is standard for cluster deployments these
>                 days.  See the multiprocess-zookeeper example for
>                 ideas about how to deploy it.  It's also important to
>                 read the how-to-build-and-deploy page to understand
>                 the example.
>
>                 Thanks,
>                 Karl
>
>
>
>                 On Tue, Jun 24, 2014 at 8:04 AM, lalit jangra
>                 <lalit.j.jangra@gmail.com
>                 <mailto:lalit.j.jangra@gmail.com>> wrote:
>
>                     Hi,
>
>                     I am planning to use MCF in cluster mode. For
>                     same, i want to know if Zookeeper is of any help here?
>
>                     If yes, how can it be leveraged in distributed MCF
>                     servers?
>
>                     Regards,
>                     Lalit Jangra.
>
>
>
>
>
>             -- 
>             Regards,
>             Lalit Jangra.
>
>
>
>
>
>     -- 
>     Regards,
>     Lalit Jangra.
>
>


Mime
View raw message