Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of uwe@thetaphi.de designates
 188.138.97.18 as permitted sender)
From: "Uwe Schindler" <uwe@thetaphi.de>
To: <java-user@lucene.apache.org>
References: 
 <11264_1345029880_ZZi022IV69XHm.00_502B86F7.7030708@uni-bielefeld.de>
 <000c01cd7ad9$f5593ad0$e00bb070$@thetaphi.de>
 <29776_1345032456_ZZi126IbejWt1.00_502B90F0.9070503@uni-bielefeld.de>
 <002401cd7ade$f4aab910$de002b30$@thetaphi.de>
 <21709_1345035856_ZZi025IhsXiUv.00_502B9E50.4030301@uni-bielefeld.de>
 <003a01cd7ae7$a109f1f0$e31dd5d0$@thetaphi.de>
In-Reply-To: <003a01cd7ae7$a109f1f0$e31dd5d0$@thetaphi.de>
Subject: RE: howto run CheckIndex on huge index size
Date: Wed, 15 Aug 2012 16:26:14 +0200
Message-ID: <001801cd7af1$ed836250$c88a26f0$@thetaphi.de>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Thread-Index: 
 AQGzo6HAYd+EMgxGv5NvkoY02X6qXQKChb4eAqQSekcCEjv+twIlVyGdAcvvW7qXNUuEMA==
Content-Language: de

I hope the problem is fixed now; this mail is just to check! It was hard =
to
unsubscribe because of the strange eMail. Have no idea at all...

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, August 15, 2012 3:13 PM
> To: java-user@lucene.apache.org
> Subject: RE: howto run CheckIndex on huge index size
>=20
> I got is, too. As a moderator of this list, I will look into finding =
the
root cause
> and forcefully unsubscribe the failing address!
>=20
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>=20
>=20
> > -----Original Message-----
> > From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > Sent: Wednesday, August 15, 2012 3:04 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: howto run CheckIndex on huge index size
> >
> >
> > I guess that ulimit could be a default setting of XenServer when it
> > was
> first time
> > setup.
> > We started with about 27G.
> > I already raised ulimit -n when setting up XenServer because this =
was
> > also limited.
> >
> > By the way, am I the only one getting this nasty DELIVERY FAILURE
> > message from one on this list?
> >
> > Bernd
> >
> >
> >
> >
> > Am 15.08.2012 14:10, schrieb Uwe Schindler:
> > > So my blog post, last section, helped? I think the ulimits came =
from
> there.
> > > What distribution do you use that ulimit was actually limited - or
> > > was it some sysadmin doing this? :-)
> > >
> > > We should maybe refer to this blog post from docs or create a copy
> > > of the page inside lucene's distribution!
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > >> -----Original Message-----
> > >> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > >> Sent: Wednesday, August 15, 2012 2:07 PM
> > >> To: java-user@lucene.apache.org
> > >> Subject: Re: howto run CheckIndex on huge index size
> > >>
> > >> Hi Uwe,
> > >>
> > >> index size is:
> > >> -rw-r--r-- 1 solr users  82G 15. Aug 07:50 _2rhe.fdt
> > >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
> > >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
> > >> -rw-r--r-- 1 solr users  39G 15. Aug 09:04 _2rhe.frq
> > >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
> > >> -rw-r--r-- 1 solr users  47G 15. Aug 09:04 _2rhe.prx
> > >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
> > >> -rw-r--r-- 1 solr users  25G 15. Aug 09:04 _2rhe.tis
> > >> -rw-r--r-- 1 solr users  300 15. Aug 09:05 segments_5u
> > >> -rw-r--r-- 1 solr users   20 15. Aug 09:05 segments.gen
> > >>
> > >> Setting ulimit -v and ulimit -m to unlimited solved this and I =
can
> > >> run
> > > CheckIndex
> > >> without any java options.
> > >>
> > >> The setting with ulimit should be pinned on lucene and solr =
pages!
> > >>
> > >> Thanks for help,
> > >> Bernd
> > >>
> > >>
> > >> Am 15.08.2012 13:34, schrieb Uwe Schindler:
> > >>> You don't get a heap-related OOM in your stack trace, it is "Map
> > >>> failed" - caused by MMapDirectory. You don't have enough virtual
> > >>> memory to map the index into address space. I think your heap is =
way
> > >>> too mch (-Xmx25g is way too big for any existing index and =
drives GC
> > > crazy).
> > >> How big is your index?
> > >>>
> > >>> See
> > >>> =
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.h
> > >>> tm
> > >>> l
> > >>>
> > >>> -----
> > >>> Uwe Schindler
> > >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > >>> eMail: uwe@thetaphi.de
> > >>>
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > >>>> Sent: Wednesday, August 15, 2012 1:25 PM
> > >>>> To: java-user@lucene.apache.org
> > >>>> Subject: howto run CheckIndex on huge index size
> > >>>>
> > >>>>
> > >>>> I'm trying to run CheckIndex as seperate tool on a large index =
to
get
> > >>>> nice
> > >>> infos
> > >>>> about number of terms, number of tokens, ... but always get OOM
> > >> exception.
> > >>>>
> > >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
> > >>>>
> > >>>>
> > >>>> Any idea how to use CheckIndex on huge index size?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> > >>>> Segments file=3Dsegments_5u numSegments=3D1 version=3D3.6.1
> > >>>> format=3DFORMAT_3_1 [Lucene 3.1+]
> > >>>>   1 of 1: name=3D_2rhe docCount=3D37821132
> > >>>>     compound=3Dfalse
> > >>>>     hasProx=3Dtrue
> > >>>>     numFiles=3D8
> > >>>>     size (MB)=3D182.992,35
> > >>>>     diagnostics =3D {mergeFactor=3D1, =
os.version=3D2.6.32.23-0.3-xen,
> > >>>> os=3DLinux, lucene.version=3D3.6.1-SNAPSHOT 1325634 - solr - =
2012-04-13
> > >>>> 14:08:0 6, source=3Dmerge, os.arch=3Damd64, =
mergeMaxNumSegments=3D1,
> > >>>> java.version=3D1.6.0_29, java.vendor=3DSun Microsystems Inc.}
> > >>>>     no deletions
> > >>>>     test: open reader.........FAILED
> > >>>>     WARNING: fixIndex() would remove reference to this segment;
full
> > >>>> exception:
> > >>>> java.io.IOException: Map failed
> > >>>>         at =
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> > >>>>         at
> > >>>>
> > >>
> >
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
> > >>>> t
> > >>>> ory.java:271)
> > >>>>         at
> > >>>>
> > >>
> >
> =
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> > >>>>         at
> > >>>>
> > >
> =
org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
> > >>>>         at
> > >>>>
> > =
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
> > >>>> jav
> > >>>> a:83)
> > >>>>         at
> > >>> =
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> > >>>>         at
> > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> > >>>>         at
> > >>> =
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
> > >>>>         at
> > >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> > >>>> Caused by: java.lang.OutOfMemoryError: Map failed
> > >>>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > >>>>         at =
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> > >>>>         ... 8 more
> > >>>> WARNING: 1 broken segments (containing 37821132 documents)
> detected
> > >>>> WARNING: would write new segments file, and 37821132 documents
> > would
> > >>>> be lost, if -fix were specified
> > >>>>
> > >>>>
> > >>>>
> > >>>>
---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>>> For additional commands, e-mail: =
java-user-help@lucene.apache.org
> > >>>
> > >>>
> > >>>
---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>> For additional commands, e-mail: =
java-user-help@lucene.apache.org
> > >>>
> > >>
> > >> --
> > >>
> *************************************************************
> > >> Bernd Fehling                Universit=E4tsbibliothek Bielefeld
> > >> Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
> > >> Universit=E4tsstr. 25                     und Wissensmanagement
> > >> 33615 Bielefeld
> > >> Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
> > >>
> > >> BASE - Bielefeld Academic Search Engine - www.base-search.net
> > >>
> *************************************************************
> > >>
> > >> =
---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > > =
---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > --
> > *************************************************************
> > Bernd Fehling                Universit=E4tsbibliothek Bielefeld
> > Dipl.-Inform. (FH)            LibTec - Bibliothekstechnologie
> > Universit=E4tsstr. 25                     und Wissensmanagement
> > 33615 Bielefeld
> > Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de
> >
> > BASE - Bielefeld Academic Search Engine - www.base-search.net
> > *************************************************************
> >
> > =
---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>=20
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org