manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Tavard <olivier.tav...@francelabs.com>
Subject Re: ZK based synchronization questions
Date Tue, 31 Oct 2017 16:23:26 GMT
Hi all,
Just to clarify my concern on ZK:
To my knowledge, best practices concerning ZK connections are to not go beyond 60. Is there
any rationale for setting it at 1000 for MCF ? 
Could this can have side effects on our ZK cluster shared by MCF and SolrCloud ?

Thanks,

Olivier


> Le 23 oct. 2017 à 17:19, Olivier Tavard <olivier.tavard@francelabs.com> a écrit
:
> 
> Hello,
> 
> We configured MCF to use ZK sync instead of file sync. We noticed a huge improvement
regarding the stability of the MCF jobs in every case especially for large data to index (15M
of files using the Windows Share repository connector). Before that, we had some errors when
the job was running randomly. With that change, we did not notice any error on the job so
far.
> However, after testing that configuration on several servers, we had errors reported
and I would like to know what you suggest for that.
> We installed MCF on servers that already have Solr 6.6.X on them. I saw on other threads
on the mailing list that it was OK to use existing ZK installation rather than using a new
ZK instance dedicated to MCF so we use the same ZK for both Solr and MCF.
> After starting MCF and Solr, we noticed some errors on the MCF log for few servers :
Session 0x0 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection
and attempting reconnect
> java.io.IOException: Connection reset by peer
> Then after checking the ZK log we saw this message : "WARN 2017-10-23 08:53:35,431 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181)
- ZooKeeper|ZooKeeper|zookeeper.server.NIOServerCnxnFactory|Too many connections from /127.0.0.1
- max is 60"
> Therefore we changed the parameter maxClientCnxns from 60 in our ZK configuration to
1000 as in the MCF zookeeper.cfg default file and there is no problem anymore.
> I  would just like to know why this parameter needs to be so high for MCF needs and if
other people share their ZK cluster with both Solr and MCF too without any problem. And last
question I had : MCF uses Zookeeper 3.4.8 while Solr 6.6+ uses ZK 3.4.10.  As written above
our ZK cluster version is 3.4.10 and we use MCF on it, is it ok to do that or would it be
best to use a ZK installation with version 3.4.8 only for MCF ?  So far we did not see any
problems using it.
> 
> Thanks,
> Best regards, 
> 
> Olivier TAVARD


Mime
View raw message