lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Serebrennikov <dmit...@earthlink.net>
Subject Re: Homogeneous vs Heterogeneous indexes (was: FileNotFoundException)
Date Wed, 01 May 2002 20:33:59 GMT

Subject:

Re: Homogeneous vs Heterogeneous indexes (was: FileNotFoundException)
From:

petite_abeille <petite_abeille@mac.com>
Date:

Wed, 1 May 2002 08:37:51 +0200

To:

"Lucene Users List" <lucene-user@jakarta.apache.org>


    On Wednesday, May 1, 2002, at 12:41 AM, Dmitry Serebrennikov wrote:

>     - the number of files that Lucene uses depends on the number of
>     segments in the index and the number of *stored* fields
>     - if your fields are not stored but only indexed, they do not
>     require separate files. Otherwise, an .fnn file is created for
>     each field.


    Ok. That's good as all my fields are indexed but not stored in
    Lucene. Only one field is stored in any one index: the uuid of an
    object (as a Keyword).

>     - if at least one document uses a given field name in an index,
>     that index requires the .fnn file for that field


    Ok. So, in theory, more homogeneous index should use less files all
    things being equal?

I think so... I guess you have many kinds of documents that have some 
fields in common and some unique? Yes, then having the same kinds of 
documents in a given index will reduce the total number of files. 
Personally, I don't have experience with this since all of my documents 
have the same fields.


>     - index segments are created when documents are added to the
>     index. For each 10 docs you get a new segment.
>     - optimizing the index removes all segments are replaces them with
>     one new segment that contains all of the documents
>     - optimization is done periodically as more documents are added
>     (controlled by IndexWriter.mergeFactor), but can be done manually
>     whenever needed


    Ok. When doing the optimization, are there any temporary files
    getting created?

Nope, just the files for the new segment. Well, I think the segments and 
deleted files might have "segments.new" and "deleted.new" while they are 
being modified, with the old ones removed and new ones renamed afterwards.

>     Some additional info: there is a field on IndexWriter called
>     infoStream. If this is set to a PrintStream (such as System.out),
>     various diagnostic messages about the merging process will be
>     printed to that stream.


    Yep. I guess I overlooked that.

>     You might find this helpful in tuning the merge parameters.


    Just to make sure: using a small merge factor (eg 2) will reduce the
    number of files or just optimize (aka merge) the index more often?

It will optimize more often and, since optimization replaces all 
segments with one, the number of files will drop. However, the old files 
will stay around until they are no longer in use by pre-existing 
IndexReader instances, so that may be another catch.

Dmitry.



--
To unsubscribe, e-mail:   <mailto:lucene-user-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:lucene-user-help@jakarta.apache.org>


Mime
View raw message