Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of phil@friendsreunited.co.uk
 designates 195.245.231.130 as permitted sender)
From: Phil Hoy <phil@friendsreunited.co.uk>
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Subject: RE: SolrCloud with large synonym files
Thread-Topic: SolrCloud with large synonym files
Thread-Index: AcyZVTHfCWXn2eghTOOEBxBiAvkKuwAG7JGAAAIOH/A=
Date: Wed, 2 Nov 2011 16:20:34 +0000
Message-ID: 
 <B34B86C16C24CA4284A487BF5BF090DE1188D79F@bsol-dun-mbx4.scotlandonline.local>
References: 
 <B34B86C16C24CA4284A487BF5BF090DE1188D65C@bsol-dun-mbx4.scotlandonline.local>
 <F4841E9B-C739-43A6-9B7F-CA5139569F08@gmail.com>
In-Reply-To: <F4841E9B-C739-43A6-9B7F-CA5139569F08@gmail.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I tried adding the property but it did not seem to improve things. I did ho=
wever get it working by noticing that the ZkSolrResourceLoader has a fall b=
ack to load resources from the shared lib, this worked for me.=20

Thanks for getting back to me.
Phil

-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]=20
Sent: 02 November 2011 15:06
To: solr-user@lucene.apache.org
Subject: Re: SolrCloud with large synonym files


On Nov 2, 2011, at 7:47 AM, Phil Hoy wrote:

> Hi,
>=20
> I am running solrcloud and a file in the Dbootstrap_confdir is a large la=
rge synonym file (~50mb ) used by a SynonymFilterFactory configured in the =
schema.xml. When i start solr I get a zookeeper exception presumably becaus=
e the file size is too large.=20

>=20
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: =
KeeperErrorCode =3D ConnectionLoss for /configs/recordsets_conf/firstnames.=
csv
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> 	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1038)
>=20
> Is there a way to either increase the limit in zookeeper or perhaps confi=
gure the SynonymFilterFactory differently to get the file from somewhere ex=
ternal to Dbootstrap_confdir?
>=20
> Phil =20


As a workaround you can try:

(Java system property:* jute.maxbuffer*)


    This option can only be set as a Java system property. There is no
    zookeeper prefix on it. It specifies the maximum size of the data
    that can be stored in a znode. The default is 0xfffff, or just under
    1M. If this option is changed, the system property must be set on
    all servers and clients otherwise problems will arise. This is
    really a sanity check. ZooKeeper is designed to store data on the
    order of kilobytes in size.

Eventually there are other ways to solve this that we may offer...

Optional compression of files
Store a file across multiple zk nodes transparently when size is too large

- Mark Miller
lucidimagination.com


______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Power=
ed by MessageLabs
______________________________________________________________________