lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <s...@elyograg.org>
Subject Re: What is the best approach to send lots of XML Messages to Solr to build index?
Date Sun, 15 Jun 2014 16:00:43 GMT
On 6/15/2014 2:54 AM, Floyd Wu wrote:
> Thank you Alex.
> I'm doing commit every 100 fiels.
> Maybe there is a better way to do this job, something like DIH(possible?)
> Sometimes i have much bigger xml file (2MB) and post to SOLR(jetty enabled)
> may encounter slow or exceed limitation.

If you are getting OOM exceptions on your Solr server, then you need to
increase your Java heap size.  I have never seen the "pending too many"
error that you mentioned at the beginning of the thread, and Google
didn't turn up anything useful, so I don't know what needs to be done
for that.  If you can post the entire exception stacktrace for this
error, perhaps we can figure it out.  We would also need the exact Solr
version.

Solr has a default 2MB limit on POST requests.  This can be increased
with the formdataUploadLimitInKB parameter on the requestDispatcher tag
in solrconfig.xml -- assuming that you're running 4.1 or later.
Previous versions required changing the request size in the servlet
container config, but there was a bug in the example Jetty included in
4.0.0 that made it impossible to change the size.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130
https://issues.apache.org/jira/browse/SOLR-4223

Regarding the advice to disable the updateLog: I never like to do this,
but if you are sending a very large number of updates in a single
request, it might be advisable until indexing is complete, so that Solr
restart times are not excessive.

Thanks,
Shawn


Mime
View raw message