lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Willem Basson <willem.bas...@gmail.com>
Subject Re: Weird issues when upgrading from 1.4 to 3.4
Date Fri, 30 Sep 2011 12:44:16 GMT
Just to clarify, I'm not worried about the virtual memory getting bigger,
the issue is that after doing a lot of adds without a commit the performance
dramatically decreases until we do the commit.
This didn't use to be a problem with 1.4

Willem

On Fri, Sep 30, 2011 at 10:20 AM, Willem Basson <willem.basson@gmail.com>wrote

> Hi there
>
> We are currently upgrading from solr 1.4 to 3.4 and have seen some issues
> with our specific use-case.
> As background we drop the whole index and then add all our documents in one
> big build before we commit and then optimise.
> This way we can revert the build if there are any issues and this won't be
> replicated out to our slave instances.
>
> We use the following flags: -Xmx2g -Xms2g
> With 1.4 it works pretty well, and the process doesn't use much more than
> 2GB of memory as the index is being built. Garbage collector kicks in quite
> a bit but performance are pretty decent throughout the build.
> For 3.4 though the process builds up to use a lot more memory, about 20 to
> 30 GB and it starts swapping and grinding to a bit of a halt. This means it
> takes about 10 times longer to complete the build, but it does complete.
> We have about a 100k documents and they aren't massive, but they aren't
> small either. They do have a lot of fields though, some have 6000+ fields.
>
> Monitoring the heap size I can see that it stays under 2GB and garbage
> collector seems to kick in just like with 1.4
>
> While we could increase the memory and do a few other things to make this
> less of an issue I would really like to know if anyone has any idea what the
> problem could be and what we could do to try and change the behaviour in
> config. Our solrconfig.xml file is the same for 1.4 and 3.4, we haven't made
> any changes.
>
> Thanks
>
> Willem Basson
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message