lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: OutOfMemory when batchupdating from SolrJ
Date Fri, 19 Feb 2016 13:51:08 GMT
And if it is on Solr side, please increase the heap size on Solr side
https://cwiki.apache.org/confluence/display/solr/JVM+Settings

On Fri, Feb 19, 2016 at 8:42 AM, Susheel Kumar <susheel2777@gmail.com>
wrote:

> When you run your SolrJ Client Indexing program, can you increase heap
> size similar below.  I guess it may be on your client side you are running
> int OOM... or please share the exact error if below doesn't work/is the
> issue.
>
>  java -Xmx4096m ....
>
>
> Thanks,
>
> Susheel
>
> On Fri, Feb 19, 2016 at 6:25 AM, Clemens Wyss DEV <clemensdev@mysign.ch>
> wrote:
>
>> Guessing on ;) :
>> must I commit after every "batch", in order to force a flushing of
>> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream et al?
>>
>> OTH it is propagated to NOT "commit" from a (SolrJ) client
>>
>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>> 'Be very careful committing from the client! In fact, don’t do it'
>>
>> I would not want to commit "just to flush a client side buffer" ...
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>> Gesendet: Freitag, 19. Februar 2016 11:09
>> An: solr-user@lucene.apache.org
>> Betreff: AW: OutOfMemory when batchupdating from SolrJ
>>
>> The char[] which occupies 180MB has the following "path to root"
>>
>> char[87690841] @ 0x7940ba658  <add><doc boost="1.0"><field
>> name="_my_id">shopproducts#<CUT>...
>> |- <Java Local> java.lang.Thread @ 0x7321d9b80  SolrUtil executorService
>> |for core 'fust-1-fr_CH_1' -3-thread-1 Thread
>> |- value java.lang.String @ 0x79e804110  <add><doc boost="1.0"><field
>> name="_my_id">shopproducts#<CUT>...
>> |  '- str org.apache.solr.common.util.ContentStreamBase$StringStream @
>> 0x77fd84680
>> |     |- <Java Local> java.lang.Thread @ 0x7321d9b80  SolrUtil
>> executorService for core 'fust-1-fr_CH_1' -3-thread-1
>> |     |- contentStream
>> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream @
>> 0x77fd846a0
>> |     |  |- <Java Local> java.lang.Thread @ 0x7321d9b80  SolrUtil
>> executorService for core 'fust-1-fr_CH_1' -3-thread-1 Thread
>> |     |  |- [0] org.apache.solr.common.util.ContentStream[1] @ 0x79e802fb8
>> |     |  |  '- <Java Local> java.lang.Thread @ 0x7321d9b80  SolrUtil
>> |executorService for core 'fust-1-fr_CH_1' -3-thread-1 Thread
>>
>> And there is another byte[] with 260MB.
>>
>> The logic is somewhat this:
>>
>> SolrClient solrClient = new HttpSolrClient( coreUrl ); while ( got more
>> elements to index ) {
>>   batch = create 100 SolrInputDocuments
>>   solrClient.add( batch )
>>  }
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Clemens Wyss DEV [mailto:clemensdev@mysign.ch]
>> Gesendet: Freitag, 19. Februar 2016 09:07
>> An: solr-user@lucene.apache.org
>> Betreff: OutOfMemory when batchupdating from SolrJ
>>
>> Environment: Solr 5.4.1
>>
>> I am facing OOMs when batchupdating SolrJ. I am seeing approx 30'000(!)
>> SolrInputDocument instances, although my batchsize is 100. I.e. I call
>> solrClient.add( documents ) for every 100 documents only. So I'd expect to
>> see at most 100 SolrInputDocument's in memory at any moment UNLESS
>> a) solrClient.add is "asynchronous" in its nature. Then QueryResponse
>> would be an async-result?
>> or
>> b) SolrJ is spooling the documents in client-side
>>
>> What might be going wrong?
>>
>> Thx for your advices
>> Clemens
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message