lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phil Hoy <p...@friendsreunited.co.uk>
Subject RE: SolrCloud with large synonym files
Date Wed, 02 Nov 2011 16:20:34 GMT
I tried adding the property but it did not seem to improve things. I did however get it working
by noticing that the ZkSolrResourceLoader has a fall back to load resources from the shared
lib, this worked for me. 

Thanks for getting back to me.
Phil

-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com] 
Sent: 02 November 2011 15:06
To: solr-user@lucene.apache.org
Subject: Re: SolrCloud with large synonym files


On Nov 2, 2011, at 7:47 AM, Phil Hoy wrote:

> Hi,
> 
> I am running solrcloud and a file in the Dbootstrap_confdir is a large large synonym
file (~50mb ) used by a SynonymFilterFactory configured in the schema.xml. When i start solr
I get a zookeeper exception presumably because the file size is too large. 

> 
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /configs/recordsets_conf/firstnames.csv
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> 	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1038)
> 
> Is there a way to either increase the limit in zookeeper or perhaps configure the SynonymFilterFactory
differently to get the file from somewhere external to Dbootstrap_confdir?
> 
> Phil  


As a workaround you can try:

(Java system property:* jute.maxbuffer*)


    This option can only be set as a Java system property. There is no
    zookeeper prefix on it. It specifies the maximum size of the data
    that can be stored in a znode. The default is 0xfffff, or just under
    1M. If this option is changed, the system property must be set on
    all servers and clients otherwise problems will arise. This is
    really a sanity check. ZooKeeper is designed to store data on the
    order of kilobytes in size.

Eventually there are other ways to solve this that we may offer...

Optional compression of files
Store a file across multiple zk nodes transparently when size is too large

- Mark Miller
lucidimagination.com












______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________

Mime
View raw message