lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Willem Basson <willem.bas...@gmail.com>
Subject Weird issues when upgrading from 1.4 to 3.4
Date Fri, 30 Sep 2011 09:20:38 GMT
Hi there

We are currently upgrading from solr 1.4 to 3.4 and have seen some issues
with our specific use-case.
As background we drop the whole index and then add all our documents in one
big build before we commit and then optimise.
This way we can revert the build if there are any issues and this won't be
replicated out to our slave instances.

We use the following flags: -Xmx2g -Xms2g
With 1.4 it works pretty well, and the process doesn't use much more than
2GB of memory as the index is being built. Garbage collector kicks in quite
a bit but performance are pretty decent throughout the build.
For 3.4 though the process builds up to use a lot more memory, about 20 to
30 GB and it starts swapping and grinding to a bit of a halt. This means it
takes about 10 times longer to complete the build, but it does complete.
We have about a 100k documents and they aren't massive, but they aren't
small either. They do have a lot of fields though, some have 6000+ fields.

Monitoring the heap size I can see that it stays under 2GB and garbage
collector seems to kick in just like with 1.4

While we could increase the memory and do a few other things to make this
less of an issue I would really like to know if anyone has any idea what the
problem could be and what we could do to try and change the behaviour in
config. Our solrconfig.xml file is the same for 1.4 and 3.4, we haven't made
any changes.

Thanks

Willem Basson

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message