lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject RE: Indexing Growth
Date Wed, 02 Apr 2003 16:02:17 GMT
Rob,

Here are some ideas.

1. Which version of Lucene are you using?  HEAD from CVS?  If so,
FSDirectory was patched recently (renameSomething method), make sure
there is no bug in it.

2. IndexReader/IndexWriter allow you to set the stream for logging 
(can't remember the method name now).  Those classes will then print
out what they are doing to that stream.

3. Write a simple class that opens your index before any queries are
made and prints out the number of documents, the number of terms, etc.
Then run your queries, and run the application again and see if the
numbers are different.

4. Do as in 3, but run 1 single query.  Observe difference between pre
and post-query numbers.  Run another query.  Observe the difference in
numbers again.  Is there a pattern?

5. To make sure it's not some background process adding documents, run
your application (from 3.) twice, but without querying the index, just
waiting some time between runs.

Otis



--- Rob Outar <routar@ideorlando.org> wrote:
> Hi all,
> 
> 	This is too odd and I do not even know where to start.  We built a
> Windows
> Explorer type tool that indexes all files in a "sabdboxed" file
> system.
> Each Lucene document contains stuff like path, parent directory, last
> modified date, file_lock etc..  When we display the files in a given
> directory through the tool we query the index about 5 times for each
> file in
> the repository, this is done so we can display all attributes in the
> index
> about that file.  So for example if there are 5 files in the
> directory, each
> file has 6 attributes that means about 30 term queries are executed. 
> The
> initial index when build it about 10.4megs, after accessing about 3
> or 4
> directories the index size increased to over 100megs, and we did not
> add
> anything!!  All we are doing is querying!!  Yesterday after querying
> became
> ungodly slow, we looked at the index size it had grown from 10megs to
> 1.5GB
> (granted we tested the tool all morning).  But I have no idea why the
> index
> is growing like this.  ANY help would be greatly appreciated.
> 
> 
> Thanks,
> 
> Rob
> 
> 
> -----Original Message-----
> From: Rob Outar [mailto:routar@ideorlando.org]
> Sent: Tuesday, April 01, 2003 3:32 PM
> To: Lucene Users List; lists@lissus.com
> Subject: RE: Indexing Growth
> 
> 
> I reuse the same searcher, analyzer and Query object I don't think
> that
> should cause the problem.
> 
> Thanks,
> 
> Rob
> 
> 
> -----Original Message-----
> From: Alex Murzaku [mailto:lists@lissus.com]
> Sent: Tuesday, April 01, 2003 3:22 PM
> To: 'Lucene Users List'
> Subject: RE: Indexing Growth
> 
> 
> I don't know if I remember this correctly: I think for every query
> (term) is created a file but the file should disappear after the
> query
> is completed.
> 
> -----Original Message-----
> From: Rob Outar [mailto:routar@ideorlando.org]
> Sent: Tuesday, April 01, 2003 3:13 PM
> To: Lucene Users List
> Subject: RE: Indexing Growth
> 
> 
> Dang I must be doing something crazy cause all my client app does is
> search and the index size increases.  I do not add anything.
> 
> Thanks,
> 
> Rob
> 
> 
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, April 01, 2003 3:07 PM
> To: Lucene Users List
> Subject: Re: Indexing Growth
> 
> 
> Only when you add new documents to it.
> 
> Otis
> 
> --- Rob Outar <routar@ideorlando.org> wrote:
> > Hi all,
> >
> > 	Will the index grow based on queries alone?  I build my index,
> then
> > run several queries against it and afterwards I check the size of
> the
> > index and
> > in some cases it has grown quite a bit although I did not add
> > anything???
> >
> > Anyhow please let me know the cases when the index will grow.
> >
> > Thanks,
> >
> > Rob
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
> 
> 
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Tax Center - File online, calculators, forms, and more
> http://platinum.yahoo.com
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 


__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
http://tax.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message