lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clemens Wyss DEV <clemens...@mysign.ch>
Subject AW: AW: OutOfMemory when batchupdating from SolrJ
Date Mon, 22 Feb 2016 08:55:15 GMT
Find attached the relevant part of the batch-update:
...
SolrClient solrClient = getSolrClient( coreName, true );
Collection<SolrInputDocument> batch = new ArrayList<SolrInputDocument>();
while ( elements.hasNext() )
{
  IIndexableElement elem = elements.next();
  SolrInputDocument doc = createSolrDocForElement( elem, provider, locale ); // [1]
  if ( doc != null )
  {
    batch.add( doc );
    if ( batch.size() == 100 )
    {
      solrClient.add( documents ); // [2]
      batch = new ArrayList<SolrInputDocument>(); // [3]
    }
  }
}
if ( !batch.isEmpty() )
{
  addMultipleDocuments( uniqueProviderName, solrClient, batch );
  batch = null;
}
...

IIndexableElement is part of our index/search framework.

[1] creating a single SolrInputDocument

[2] handing the SIDs to SolrJ/SolrClient

[3] creating a new batch, i.e. releasing the SolrInputDocuments

The above code is being executed in an ExecutorService handed in as a lambda. I.e.:
executorService.submit( () -> {
<the code>
} );

Thanks for any advices. If needed, I can also provide the OOM-heapdump ...

-----Urspr√ľngliche Nachricht-----
Von: Shawn Heisey [mailto:apache@elyograg.org] 
Gesendet: Freitag, 19. Februar 2016 18:59
An: solr-user@lucene.apache.org
Betreff: Re: AW: OutOfMemory when batchupdating from SolrJ

On 2/19/2016 3:08 AM, Clemens Wyss DEV wrote:
> The logic is somewhat this:
>
> SolrClient solrClient = new HttpSolrClient( coreUrl ); while ( got 
> more elements to index ) {
>   batch = create 100 SolrInputDocuments
>   solrClient.add( batch )
>  }

How much data is going into each of those SolrInputDocument objects?

If the amount of data is very small (a few kilobytes), then this sounds like your program
has a memory leak.  Can you provide more code detail? 
Ideally, you would make the entire code available by placing it on the Internet somewhere
and providing a URL.  If there's anything sensitive in the code, like passwords or public
IP addresses, feel free to redact it, but try not to remove anything that affects how the
code operates.

Thanks,
Shawn

Mime
View raw message