lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernd Fehling <bernd.fehl...@uni-bielefeld.de>
Subject Re: howto run CheckIndex on huge index size
Date Wed, 15 Aug 2012 12:07:12 GMT
Hi Uwe,

index size is:
-rw-r--r-- 1 solr users  82G 15. Aug 07:50 _2rhe.fdt
-rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
-rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
-rw-r--r-- 1 solr users  39G 15. Aug 09:04 _2rhe.frq
-rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
-rw-r--r-- 1 solr users  47G 15. Aug 09:04 _2rhe.prx
-rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
-rw-r--r-- 1 solr users  25G 15. Aug 09:04 _2rhe.tis
-rw-r--r-- 1 solr users  300 15. Aug 09:05 segments_5u
-rw-r--r-- 1 solr users   20 15. Aug 09:05 segments.gen

Setting ulimit -v and ulimit -m to unlimited solved this and I can
run CheckIndex without any java options.

The setting with ulimit should be pinned on lucene and solr pages!

Thanks for help,
Bernd


Am 15.08.2012 13:34, schrieb Uwe Schindler:
> You don't get a heap-related OOM in your stack trace, it is "Map failed" -
> caused by MMapDirectory. You don't have enough virtual memory to map the
> index into address space. I think your heap is way too mch (-Xmx25g is way
> too big for any existing index and drives GC crazy). How big is your index?
> 
> See http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
>> -----Original Message-----
>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>> Sent: Wednesday, August 15, 2012 1:25 PM
>> To: java-user@lucene.apache.org
>> Subject: howto run CheckIndex on huge index size
>>
>>
>> I'm trying to run CheckIndex as seperate tool on a large index to get nice
> infos
>> about number of terms, number of tokens, ... but always get OOM exception.
>>
>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
>>
>>
>> Any idea how to use CheckIndex on huge index size?
>>
>>
>>
>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
>> Segments file=segments_5u numSegments=1 version=3.6.1
>> format=FORMAT_3_1 [Lucene 3.1+]
>>   1 of 1: name=_2rhe docCount=37821132
>>     compound=false
>>     hasProx=true
>>     numFiles=8
>>     size (MB)=182.992,35
>>     diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, os=Linux,
>> lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 14:08:0 6,
>> source=merge, os.arch=amd64, mergeMaxNumSegments=1,
>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
>>     no deletions
>>     test: open reader.........FAILED
>>     WARNING: fixIndex() would remove reference to this segment; full
>> exception:
>> java.io.IOException: Map failed
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>         at
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirect
>> ory.java:271)
>>         at
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
>>         at
>> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
>>         at
>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.jav
>> a:83)
>>         at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>>         at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
>>         at
> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
>>         at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>         ... 8 more
>> WARNING: 1 broken segments (containing 37821132 documents) detected
>> WARNING: would write new segments file, and 37821132 documents would be
>> lost, if -fix were specified
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

-- 
*************************************************************
Bernd Fehling                Universit├Ątsbibliothek Bielefeld
Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
Universit├Ątsstr. 25                     und Wissensmanagement
33615 Bielefeld
Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de

BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message