lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noble Paul നോബിള്‍ नोब्ळ् <noble.p...@gmail.com>
Subject Re: performance tests with DataImportHandler and full-import
Date Wed, 01 Apr 2009 10:24:55 GMT
I guess Solr itself is hogging more memory.
May be you can try reloading the core before each import.

On Wed, Apr 1, 2009 at 3:19 PM, Marc Sturlese <marc.sturlese@gmail.com> wrote:
>
> Hey there,
> I am doing performance tests with full-import command from
> DataImportHandler. I have configured 20 cores with 1 Gig index each (about a
> milion docs per index). If I start doing full-imports indexing from a mysql
> table SEQUENCIALY with cron jobs as frequently as possible it will work fine
> for about 17 or 20 full-import jobs in total. After that it starts taking
> much much longer for each full import... (from 30 min at the begining to 4
> hours for the las full-import before crashing) until I get an
> OutOfMemoryError java Heap Space.
>
> Apr  1 10:30:47 indexer-03 solr: 63534480 [Thread-536] ERROR
> org.apache.solr.handler.dataimport.DataImporter - Full Import failed
> Apr  1 10:34:04 indexer-03 solr: 63581229 [http-8080-Processor92] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr  1 10:34:23 indexer-03 solr: 63562764 [Thread-546] WARN
> org.apache.solr.handler.dataimport.DocBuilder -  GC overhead limit exceeded
> Apr  1 10:36:11 indexer-03 solr: 63739903 [http-8080-Processor99] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr  1 10:36:11 indexer-03 solr: 63876821 [http-8080-Processor95] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr  1 10:36:20 indexer-03 solr: 63787787 [Thread-546] ERROR
> org.apache.solr.handler.dataimport.DataImporter - Full Import failed
> Apr  1 10:40:02 indexer-03 solr: 64073790 [http-8080-Processor100] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
> Apr  1 10:40:45 indexer-03 solr: 63991575 [http-8080-Processor96] ERROR
> org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError:
> Java heap space
>
> Even I do indexations sequencially I have tryed using a the
> ConcurrentGarvageCollector: -XX:+UseConcMarkSweepGC but nothing seems to
> change (it's quite logic as I don't index concurrently)
> I am runing on
> Debian 2.6.26-1-amd64
> mysql for database
> java version:
> java version "1.6.0_12"
> Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)
>
> Any idea why this could be happening?
>
>
>
>
> --
> View this message in context: http://www.nabble.com/performance-tests-with-DataImportHandler-and-full-import-tp22823145p22823145.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
--Noble Paul

Mime
View raw message