lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: howto run CheckIndex on huge index size
Date Wed, 15 Aug 2012 13:12:31 GMT
I got is, too. As a moderator of this list, I will look into finding the
root cause and forcefully unsubscribe the failing address!

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> Sent: Wednesday, August 15, 2012 3:04 PM
> To: java-user@lucene.apache.org
> Subject: Re: howto run CheckIndex on huge index size
> 
> 
> I guess that ulimit could be a default setting of XenServer when it was
first time
> setup.
> We started with about 27G.
> I already raised ulimit -n when setting up XenServer because this was also
> limited.
> 
> By the way, am I the only one getting this nasty DELIVERY FAILURE message
> from one on this list?
> 
> Bernd
> 
> 
> 
> 
> Am 15.08.2012 14:10, schrieb Uwe Schindler:
> > So my blog post, last section, helped? I think the ulimits came from
there.
> > What distribution do you use that ulimit was actually limited - or was
> > it some sysadmin doing this? :-)
> >
> > We should maybe refer to this blog post from docs or create a copy of
> > the page inside lucene's distribution!
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> >> Sent: Wednesday, August 15, 2012 2:07 PM
> >> To: java-user@lucene.apache.org
> >> Subject: Re: howto run CheckIndex on huge index size
> >>
> >> Hi Uwe,
> >>
> >> index size is:
> >> -rw-r--r-- 1 solr users  82G 15. Aug 07:50 _2rhe.fdt
> >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
> >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
> >> -rw-r--r-- 1 solr users  39G 15. Aug 09:04 _2rhe.frq
> >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
> >> -rw-r--r-- 1 solr users  47G 15. Aug 09:04 _2rhe.prx
> >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
> >> -rw-r--r-- 1 solr users  25G 15. Aug 09:04 _2rhe.tis
> >> -rw-r--r-- 1 solr users  300 15. Aug 09:05 segments_5u
> >> -rw-r--r-- 1 solr users   20 15. Aug 09:05 segments.gen
> >>
> >> Setting ulimit -v and ulimit -m to unlimited solved this and I can
> >> run
> > CheckIndex
> >> without any java options.
> >>
> >> The setting with ulimit should be pinned on lucene and solr pages!
> >>
> >> Thanks for help,
> >> Bernd
> >>
> >>
> >> Am 15.08.2012 13:34, schrieb Uwe Schindler:
> >>> You don't get a heap-related OOM in your stack trace, it is "Map
> >>> failed" - caused by MMapDirectory. You don't have enough virtual
> >>> memory to map the index into address space. I think your heap is way
> >>> too mch (-Xmx25g is way too big for any existing index and drives GC
> > crazy).
> >> How big is your index?
> >>>
> >>> See
> >>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.h
> >>> tm
> >>> l
> >>>
> >>> -----
> >>> Uwe Schindler
> >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >>> eMail: uwe@thetaphi.de
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> >>>> Sent: Wednesday, August 15, 2012 1:25 PM
> >>>> To: java-user@lucene.apache.org
> >>>> Subject: howto run CheckIndex on huge index size
> >>>>
> >>>>
> >>>> I'm trying to run CheckIndex as seperate tool on a large index to get
> >>>> nice
> >>> infos
> >>>> about number of terms, number of tokens, ... but always get OOM
> >> exception.
> >>>>
> >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
> >>>>
> >>>>
> >>>> Any idea how to use CheckIndex on huge index size?
> >>>>
> >>>>
> >>>>
> >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> >>>> Segments file=segments_5u numSegments=1 version=3.6.1
> >>>> format=FORMAT_3_1 [Lucene 3.1+]
> >>>>   1 of 1: name=_2rhe docCount=37821132
> >>>>     compound=false
> >>>>     hasProx=true
> >>>>     numFiles=8
> >>>>     size (MB)=182.992,35
> >>>>     diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
> >>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
> >>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
> >>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
> >>>>     no deletions
> >>>>     test: open reader.........FAILED
> >>>>     WARNING: fixIndex() would remove reference to this segment; full
> >>>> exception:
> >>>> java.io.IOException: Map failed
> >>>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> >>>>         at
> >>>>
> >>
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
> >>>> t
> >>>> ory.java:271)
> >>>>         at
> >>>>
> >>
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> >>>>         at
> >>>>
> > org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
> >>>>         at
> >>>>
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
> >>>> jav
> >>>> a:83)
> >>>>         at
> >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> >>>>         at
> >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> >>>>         at
> >>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
> >>>>         at
> >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> >>>> Caused by: java.lang.OutOfMemoryError: Map failed
> >>>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
> >>>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> >>>>         ... 8 more
> >>>> WARNING: 1 broken segments (containing 37821132 documents) detected
> >>>> WARNING: would write new segments file, and 37821132 documents
> would
> >>>> be lost, if -fix were specified
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>
> >> --
> >> *************************************************************
> >> Bernd Fehling                Universitätsbibliothek Bielefeld
> >> Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
> >> Universitätsstr. 25                     und Wissensmanagement
> >> 33615 Bielefeld
> >> Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
> >>
> >> BASE - Bielefeld Academic Search Engine - www.base-search.net
> >> *************************************************************
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> --
> *************************************************************
> Bernd Fehling                Universitätsbibliothek Bielefeld
> Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
> Universitätsstr. 25                     und Wissensmanagement
> 33615 Bielefeld
> Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
> 
> BASE - Bielefeld Academic Search Engine - www.base-search.net
> *************************************************************
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message