lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leon Chaddock" <leonchadd...@macranet.co.uk>
Subject Re: Size + memory restrictions
Date Wed, 15 Feb 2006 18:45:40 GMT
Hi Chris,
Thanks, when I quoted segment I meant index file.
So if we have 10 seperate index files are you saying we should have one 
indexSearcher for the index collectively, or one per index file

Thanks

Leon


----- Original Message ----- 
From: "Chris Hostetter" <hossman_lucene@fucit.org>
To: <java-user@lucene.apache.org>
Sent: Wednesday, February 15, 2006 6:40 PM
Subject: Re: Size + memory restrictions


>: We may have many different segments of our index, and it seems below we 
>are
> : using one
> : IndexSearcher per segment. Could this explain why we run out of memory 
> when
> : using more than 2/3 segments?
> : Anyone else have any comments on the below?
>
> terminology is a big issue hwere .. when you use the word "segment" it
> seems like you are talking about a segment of your data, which is in a
> self contained index in it's own right.  My point in in the comment you
> quoted was that for a given index, you don't need more then one active
> IndexSearcher open at a time, any more then that can waste resources.
>
> I don't know what kind of memory overhead there is in a MultiSearcher, but
> besides that you should also be looking at the other issues in the message
> you quoted from:   who/when is calling your getSearcher() method? ... is
> it getting called more often then the underlying indexes change?  who is
> closing the old searchers when you open new ones?
>
> :
> : Many thanks
> :
> : Leon
> : ps. At the moment I think it is set to only look at 2 segements
> :
> : private Searcher getSearcher() throws IOException {
> :   if (mSearcher == null) {
> :    synchronized (Monitor) {
> :     Searcher[] srs = new IndexSearcher[SearchersDir.size()];
> :     int maxI = 2;
> :    // Searcher[] srs = new IndexSearcher[maxI];
> :     int i = 0;
> :     for (Iterator iter = SearchersDir.iterator(); iter.hasNext() && 
> i<maxI;
> : i++) {
> :      String dir = (String) iter.next();
> :      try {
> :       srs[i] = new IndexSearcher(IndexDir+dir);
> :      } catch (IOException e) {
> :       log.error(ClassTool.getClassNameOnly(e) + ": " + e.getMessage(), 
> e);
> :      }
> :     }
> :     mSearcher = new MultiSearcher(srs);
> :     changeTime = System.currentTimeMillis();
> :    }
> :   }
> :   return mSearcher;
> :  }
> : ----- Original Message -----
> : From: "Leon Chaddock" <leonchaddock@macranet.co.uk>
> : To: <java-user@lucene.apache.org>
> : Sent: Wednesday, February 15, 2006 9:28 AM
> : Subject: Re: Size + memory restrictions
> :
> :
> : > Hi Greg,
> : > Thanks. We are actually running against 4 segments of 4gb so about 20
> : > million docs. We cant merge the segments as their seems to be problems
> : > with out linux box , with having files over about 4gb. Not sure why 
> that
> : > is.
> : >
> : > If I was to upgrade to 8gb of ram does it seem likely this will double 
> the
> : > amount of docs we can handle, or would this provide an exponential
> : > increase?
> : >
> : > Thanks
> : >
> : > Leon
> : > ----- Original Message -----
> : > From: "Greg Gershman" <greggersh@yahoo.com>
> : > To: <java-user@lucene.apache.org>
> : > Sent: Wednesday, February 15, 2006 12:41 AM
> : > Subject: Re: Size + memory restrictions
> : >
> : >
> : >> You may consider incrementally adding documents to
> : >> your index; I'm not sure why there would be problems
> : >> adding to an existing index, but you can always add
> : >> additional documents.  You can optimize later to get
> : >> everything back into a single segment.
> : >>
> : >> Querying is a different story; if you are using the
> : >> Sort API, you will need enough memory to store a full
> : >> sorting of your documents in memory.  If you're trying
> : >> to sort on a string or anything other than an int or
> : >> float, this could require a lot of memory.
> : >>
> : >> I've used indices much bigger than 5 mil. docs/3.5 gb
> : >> with less than 4GB of RAM and had no problems.
> : >>
> : >> Greg
> : >>
> : >>
> : >> --- Leon Chaddock <leonchaddock@macranet.co.uk> wrote:
> : >>
> : >>> Hi,
> : >>> we are having tremendous problems building a large
> : >>> lucene index and querying
> : >>> it.
> : >>>
> : >>> The programmers are telling me that when the index
> : >>> file reaches 3.5 gb or 5
> : >>> million docs the index file can no longer grow any
> : >>> larger.
> : >>>
> : >>> To rectify this they have built index files in
> : >>> multiple directories. Now
> : >>> apparently my 4gb memory is not enough to query.
> : >>>
> : >>> Does this seem right to people or does anyone have
> : >>> any experience on largish
> : >>> scale projects.
> : >>>
> : >>> I am completely tearing my hair out here and dont
> : >>> know what to do.
> : >>>
> : >>> Thanks
> : >>>
> : >>
> : >>
> : >> __________________________________________________
> : >> Do You Yahoo!?
> : >> Tired of spam?  Yahoo! Mail has the best spam protection around
> : >> http://mail.yahoo.com
> : >>
> : >> ---------------------------------------------------------------------
> : >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : >> For additional commands, e-mail: java-user-help@lucene.apache.org
> : >>
> : >>
> : >>
> : >>
> : >>
> : >> --
> : >> Internal Virus Database is out-of-date.
> : >> Checked by AVG Free Edition.
> : >> Version: 7.1.375 / Virus Database: 267.15.0/248 - Release Date:
> : >> 01/02/2006
> : >>
> : >>
> : >
> : >
> : > ---------------------------------------------------------------------
> : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : > For additional commands, e-mail: java-user-help@lucene.apache.org
> : >
> : >
> : >
> : >
> : >
> : > --
> : > Internal Virus Database is out-of-date.
> : > Checked by AVG Free Edition.
> : > Version: 7.1.375 / Virus Database: 267.15.0/248 - Release Date: 
> 01/02/2006
> : >
> : >
> :
> :
> : ---------------------------------------------------------------------
> : To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : For additional commands, e-mail: java-user-help@lucene.apache.org
> :
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> -- 
> Internal Virus Database is out-of-date.
> Checked by AVG Free Edition.
> Version: 7.1.375 / Virus Database: 267.15.0/248 - Release Date: 01/02/2006
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message