Return-Path: 
 <users-return-589-apmail-jackrabbit-users-archive=jackrabbit.apache.org@jackrabbit.apache.org>
Delivered-To: apmail-jackrabbit-users-archive@locus.apache.org
Received: (qmail 1380 invoked from network); 10 Aug 2006 07:12:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
  by minotaur.apache.org with SMTP; 10 Aug 2006 07:12:19 -0000
Received: (qmail 15336 invoked by uid 500); 10 Aug 2006 07:12:19 -0000
Delivered-To: apmail-jackrabbit-users-archive@jackrabbit.apache.org
Received: (qmail 15172 invoked by uid 500); 10 Aug 2006 07:12:18 -0000
Mailing-List: contact users-help@jackrabbit.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@jackrabbit.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@jackrabbit.apache.org>
List-Post: <mailto:users@jackrabbit.apache.org>
List-Id: <users.jackrabbit.apache.org>
Reply-To: users@jackrabbit.apache.org
Delivered-To: mailing list users@jackrabbit.apache.org
Received: (qmail 15163 invoked by uid 99); 10 Aug 2006 07:12:18 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Aug 2006 00:12:18 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (asf.osuosl.org: domain of marcel.reutegger@gmx.net
 designates 213.165.64.20 as permitted sender)
Received: from [213.165.64.20] (HELO mail.gmx.net) (213.165.64.20)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 10 Aug 2006 00:12:18 -0700
Received: (qmail invoked by alias); 10 Aug 2006 07:11:56 -0000
Received: from bsl-rtr.day.com (EHLO [10.0.0.70]) [212.249.34.130]
  by mail.gmx.net (mp019) with SMTP; 10 Aug 2006 09:11:56 +0200
X-Authenticated: #894343
Message-ID: <44DADC3B.8040509@gmx.net>
Date: Thu, 10 Aug 2006 09:11:55 +0200
From: Marcel Reutegger <marcel.reutegger@gmx.net>
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To: users@jackrabbit.apache.org
Subject: Re: Problems with re-index a huge repository
References: <F186AC080E44C146BBE4A33472CFA79F9A7765@mxs01.tirol.local>
In-Reply-To: <F186AC080E44C146BBE4A33472CFA79F9A7765@mxs01.tirol.local>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-Y-GMX-Trusted: 0
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N

The mergeFactor is way to high. With this setup index merging will 
only take place after 1000 index segments have been created. That's 
also the reason why there are so many directories in the index folder. 
The default value of 10 is usually a good choice and should only be 
changed in rare cases.

Can you please try a re-index with a mergeFactor of 10 and if you 
still run into an out of memory error file a jira issue?

Thanks

regards
  marcel

K�LL Claus wrote:
> i made some performance tests with a repository that has about 2 Million differend files (doc,xls, txt and ppt)
> i am very satisfied with the performace ...
> but now i made a test to re-index the whole repository to handle a scenario if there are some problems with the index at run time.
> i have deleted the index folder an restart the repository
>  
> my test pc configuration (windows 2003/4gb ram/150Gb hard disk) 
>  
> i run always in a outofmemory exception while index creation at startup from the repository
> i have set the /3Gb flag into the boot.ini to get more inital heap size
>  
> the current java start parameters are 
> -Xms1550m -Xmx3000m
> the workspace.xml file has these parameters
> <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>     <param name="path" value="${wsp.home}/index"/>
>     <param name="textFilterClasses"         value="org.apache.jackrabbit.core.query.lucene.TextPlainTextFilter,org.apache.jackrabbit.core.query.MsExcelTextFilter,org.apache.jackrabbit.core.query.MsPowerPointTextFilter,org.apache.jackrabbit.core.query.MsWordTextFilter,org.apache.jackrabbit.core.query.PdfTextFilter,org.apache.jackrabbit.core.query.HTMLTextFilter,org.apache.jackrabbit.core.query.XMLTextFilter,org.apache.jackrabbit.core.query.RTFTextFilter,org.apache.jackrabbit.core.query.OpenOfficeTextFilter"/>
>     <param name="useCompoundFile" value="true" />
>     <param name="minMergeDocs" value="1000" />
>     <param name="mergeFactor" value="1000" />
>     <param name="cacheSize" value="1000"/>
>     <param name="respectDocumentOrder" value="false" />
>     <param name="autoRepair" value="true"/>
>     <param name="forceConsistencyCheck" value="false"/>
> </SearchIndex>
> 
> for me its strange that during the index process lucene creates about 600 - 700 directories under the  
> index folder in the workspace directory and the redo.log is about 25Mb and then i get a outofmemoryexception.
> at the time of initial filling of the repository the merge of the index folders/files works fine
> but now it seems that the merger does not work.
> 
> if i restart the repository after the exception occurs the index folders/files will be merged into about 20-30 folders but
> the repository is not indexed whole.
> 
> thanks for help
> 
> claus
> 
>  
>