I guess that ulimit could be a default setting of XenServer when it was first time setup.
We started with about 27G.
I already raised ulimit -n when setting up XenServer because this was also limited.
By the way, am I the only one getting this nasty DELIVERY FAILURE message from one
on this list?
Bernd
Am 15.08.2012 14:10, schrieb Uwe Schindler:
> So my blog post, last section, helped? I think the ulimits came from there.
> What distribution do you use that ulimit was actually limited - or was it
> some sysadmin doing this? :-)
>
> We should maybe refer to this blog post from docs or create a copy of the
> page inside lucene's distribution!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>> Sent: Wednesday, August 15, 2012 2:07 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: howto run CheckIndex on huge index size
>>
>> Hi Uwe,
>>
>> index size is:
>> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
>> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
>> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
>> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
>> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
>> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
>> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
>> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
>> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
>> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
>>
>> Setting ulimit -v and ulimit -m to unlimited solved this and I can run
> CheckIndex
>> without any java options.
>>
>> The setting with ulimit should be pinned on lucene and solr pages!
>>
>> Thanks for help,
>> Bernd
>>
>>
>> Am 15.08.2012 13:34, schrieb Uwe Schindler:
>>> You don't get a heap-related OOM in your stack trace, it is "Map
>>> failed" - caused by MMapDirectory. You don't have enough virtual
>>> memory to map the index into address space. I think your heap is way
>>> too mch (-Xmx25g is way too big for any existing index and drives GC
> crazy).
>> How big is your index?
>>>
>>> See
>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm
>>> l
>>>
>>> -----
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>> -----Original Message-----
>>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>>>> Sent: Wednesday, August 15, 2012 1:25 PM
>>>> To: java-user@lucene.apache.org
>>>> Subject: howto run CheckIndex on huge index size
>>>>
>>>>
>>>> I'm trying to run CheckIndex as seperate tool on a large index to get
>>>> nice
>>> infos
>>>> about number of terms, number of tokens, ... but always get OOM
>> exception.
>>>>
>>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
>>>>
>>>>
>>>> Any idea how to use CheckIndex on huge index size?
>>>>
>>>>
>>>>
>>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
>>>> Segments file=segments_5u numSegments=1 version=3.6.1
>>>> format=FORMAT_3_1 [Lucene 3.1+]
>>>> 1 of 1: name=_2rhe docCount=37821132
>>>> compound=false
>>>> hasProx=true
>>>> numFiles=8
>>>> size (MB)=182.992,35
>>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
>>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
>>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
>>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
>>>> no deletions
>>>> test: open reader.........FAILED
>>>> WARNING: fixIndex() would remove reference to this segment; full
>>>> exception:
>>>> java.io.IOException: Map failed
>>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>>> at
>>>>
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
>>>> t
>>>> ory.java:271)
>>>> at
>>>>
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
>>>> at
>>>>
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
>>>> at
>>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
>>>> jav
>>>> a:83)
>>>> at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>>>> at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
>>>> at
>>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
>>>> at
>>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
>>>> Caused by: java.lang.OutOfMemoryError: Map failed
>>>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>>> ... 8 more
>>>> WARNING: 1 broken segments (containing 37821132 documents) detected
>>>> WARNING: would write new segments file, and 37821132 documents would
>>>> be lost, if -fix were specified
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> --
>> *************************************************************
>> Bernd Fehling Universitätsbibliothek Bielefeld
>> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
>> Universitätsstr. 25 und Wissensmanagement
>> 33615 Bielefeld
>> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
>>
>> BASE - Bielefeld Academic Search Engine - www.base-search.net
>> *************************************************************
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
*************************************************************
Bernd Fehling Universitätsbibliothek Bielefeld
Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
Universitätsstr. 25 und Wissensmanagement
33615 Bielefeld
Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
|