lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernd Fehling <bernd.fehl...@uni-bielefeld.de>
Subject Re: howto run CheckIndex on huge index size
Date Wed, 15 Aug 2012 13:04:16 GMT

I guess that ulimit could be a default setting of XenServer when it was first time setup.
We started with about 27G.
I already raised ulimit -n when setting up XenServer because this was also limited.

By the way, am I the only one getting this nasty DELIVERY FAILURE message from one
on this list?

Bernd




Am 15.08.2012 14:10, schrieb Uwe Schindler:
> So my blog post, last section, helped? I think the ulimits came from there.
> What distribution do you use that ulimit was actually limited - or was it
> some sysadmin doing this? :-)
> 
> We should maybe refer to this blog post from docs or create a copy of the
> page inside lucene's distribution!
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
>> -----Original Message-----
>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>> Sent: Wednesday, August 15, 2012 2:07 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: howto run CheckIndex on huge index size
>>
>> Hi Uwe,
>>
>> index size is:
>> -rw-r--r-- 1 solr users  82G 15. Aug 07:50 _2rhe.fdt
>> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
>> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
>> -rw-r--r-- 1 solr users  39G 15. Aug 09:04 _2rhe.frq
>> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
>> -rw-r--r-- 1 solr users  47G 15. Aug 09:04 _2rhe.prx
>> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
>> -rw-r--r-- 1 solr users  25G 15. Aug 09:04 _2rhe.tis
>> -rw-r--r-- 1 solr users  300 15. Aug 09:05 segments_5u
>> -rw-r--r-- 1 solr users   20 15. Aug 09:05 segments.gen
>>
>> Setting ulimit -v and ulimit -m to unlimited solved this and I can run
> CheckIndex
>> without any java options.
>>
>> The setting with ulimit should be pinned on lucene and solr pages!
>>
>> Thanks for help,
>> Bernd
>>
>>
>> Am 15.08.2012 13:34, schrieb Uwe Schindler:
>>> You don't get a heap-related OOM in your stack trace, it is "Map
>>> failed" - caused by MMapDirectory. You don't have enough virtual
>>> memory to map the index into address space. I think your heap is way
>>> too mch (-Xmx25g is way too big for any existing index and drives GC
> crazy).
>> How big is your index?
>>>
>>> See
>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm
>>> l
>>>
>>> -----
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>> -----Original Message-----
>>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>>>> Sent: Wednesday, August 15, 2012 1:25 PM
>>>> To: java-user@lucene.apache.org
>>>> Subject: howto run CheckIndex on huge index size
>>>>
>>>>
>>>> I'm trying to run CheckIndex as seperate tool on a large index to get
>>>> nice
>>> infos
>>>> about number of terms, number of tokens, ... but always get OOM
>> exception.
>>>>
>>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
>>>>
>>>>
>>>> Any idea how to use CheckIndex on huge index size?
>>>>
>>>>
>>>>
>>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
>>>> Segments file=segments_5u numSegments=1 version=3.6.1
>>>> format=FORMAT_3_1 [Lucene 3.1+]
>>>>   1 of 1: name=_2rhe docCount=37821132
>>>>     compound=false
>>>>     hasProx=true
>>>>     numFiles=8
>>>>     size (MB)=182.992,35
>>>>     diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
>>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
>>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
>>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
>>>>     no deletions
>>>>     test: open reader.........FAILED
>>>>     WARNING: fixIndex() would remove reference to this segment; full
>>>> exception:
>>>> java.io.IOException: Map failed
>>>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>>>         at
>>>>
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
>>>> t
>>>> ory.java:271)
>>>>         at
>>>>
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
>>>>         at
>>>>
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
>>>>         at
>>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
>>>> jav
>>>> a:83)
>>>>         at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>>>>         at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
>>>>         at
>>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
>>>>         at
>>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
>>>> Caused by: java.lang.OutOfMemoryError: Map failed
>>>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>>>         ... 8 more
>>>> WARNING: 1 broken segments (containing 37821132 documents) detected
>>>> WARNING: would write new segments file, and 37821132 documents would
>>>> be lost, if -fix were specified
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> --
>> *************************************************************
>> Bernd Fehling                Universitätsbibliothek Bielefeld
>> Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
>> Universitätsstr. 25                     und Wissensmanagement
>> 33615 Bielefeld
>> Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
>>
>> BASE - Bielefeld Academic Search Engine - www.base-search.net
>> *************************************************************
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

-- 
*************************************************************
Bernd Fehling                Universitätsbibliothek Bielefeld
Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
Universitätsstr. 25                     und Wissensmanagement
33615 Bielefeld
Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de

BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message