lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject svn commit: r1332650 - in /lucene/dev/trunk/lucene: core/src/java/org/apache/lucene/codecs/lucene40/package.html site/html/fileformats.html site/xsl/index.xsl
Date Tue, 01 May 2012 13:42:29 GMT
Author: rmuir
Date: Tue May  1 13:42:29 2012
New Revision: 1332650

LUCENE-2946: new fileformats page (WIP)


Modified: lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html
--- lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html (original)
+++ lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html Tue
May  1 13:42:29 2012
@@ -20,6 +20,360 @@
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
-Default codec for Lucene 4.0 indexes.
+Lucene 4.0 file format.
+<h1>Apache Lucene - Index File Formats</h1>
+<li><a href="#Index_File_Formats">Index File Formats</a></li>
+<li><a href="#Definitions">Definitions</a>
+<li><a href="#Inverted_Indexing">Inverted Indexing</a></li>
+<li><a href="#Types_of_Fields">Types of Fields</a></li>
+<li><a href="#Segments">Segments</a></li>
+<li><a href="#Document_Numbers">Document Numbers</a></li>
+<li><a href="#Overview">Overview</a></li>
+<li><a href="#File_Naming">File Naming</a></li>
+<li><a href="#file-names">Summary of File Extensions</a></li>
+<li><a href="#Lock_File">Lock File</a></li>
+<li><a href="#Limitations">Limitations</a></li>
+<a name="Index_File_Formats"></a>
+<h2>Index File Formats</h2>
+<p>This document defines the index file formats used in this version of Lucene.
+If you are using a different version of Lucene, please consult the copy of
+<code>docs/</code> that was distributed with
+the version you are using.</p>
+<p>Apache Lucene is written in Java, but several efforts are underway to write
+<a href="">versions of
+Lucene in other programming languages</a>. If these versions are to remain
+compatible with Apache Lucene, then a language-independent definition of the
+Lucene index format is required. This document thus attempts to provide a
+complete and independent definition of the Apache Lucene file formats.</p>
+<p>As Lucene evolves, this document should evolve. Versions of Lucene in
+different programming languages should endeavor to agree on file formats, and
+generate new versions of this document.</p>
+<p>Compatibility notes are provided in this document, describing how file
+formats have changed from prior versions.</p>
+<p>In version 2.1, the file format was changed to allow lock-less commits (ie,
+no more commit lock). The change is fully backwards compatible: you can open a
+pre-2.1 index for searching or adding/deleting of docs. When the new segments
+file is saved (committed), it will be written in the new file format (meaning
+no specific "upgrade" process is needed). But note that once a commit has
+occurred, pre-2.1 Lucene will not be able to read the index.</p>
+<p>In version 2.3, the file format was changed to allow segments to share a
+single set of doc store (vectors &amp; stored fields) files. This allows for
+faster indexing in certain cases. The change is fully backwards compatible (in
+the same way as the lock-less commits change in 2.1).</p>
+<p>In version 2.4, Strings are now written as true UTF-8 byte sequence, not
+Java's modified UTF-8. See issue LUCENE-510 for details.</p>
+<p>In version 2.9, an optional opaque Map&lt;String,String&gt; CommitUserData
+may be passed to IndexWriter's commit methods (and later retrieved), which is
+recorded in the segments_N file. See issue LUCENE-1382 for details. Also,
+diagnostics were added to each segment written recording details about why it
+was written (due to flush, merge; which OS/JRE was used; etc.). See issue
+LUCENE-1654 for details.</p>
+<p>In version 3.0, compressed fields are no longer written to the index (they
+can still be read, but on merge the new segment will write them, uncompressed).
+See issue LUCENE-1960 for details.</p>
+<p>In version 3.1, segments records the code version that created them. See
+LUCENE-2720 for details. Additionally segments track explicitly whether or not
+they have term vectors. See LUCENE-2811 for details.</p>
+<p>In version 3.2, numeric fields are written as natively to stored fields
+file, previously they were stored in text format only.</p>
+<p>In version 3.4, fields can omit position data while still indexing term
+<p>In version 4.0, the format of the inverted index became extensible via
+the {@link org.apache.lucene.codecs.Codec Codec} api. Fast per-document storage
+({@link org.apache.lucene.index.DocValues DocValues}) was introduced. Normalization
+factors need no longer be a single byte, they can be any DocValues 
+{@link org.apache.lucene.index.DocValues.Type type}. Terms need not be unicode
+strings, they can be any byte sequence. Term offsets can optionally be indexed 
+into the postings lists.</p>
+<a name="Definitions" id="Definitions"></a>
+<p>The fundamental concepts in Lucene are index, document, field and term.</p>
+<p>An index contains a sequence of documents.</p>
+<p>A document is a sequence of fields.</p>
+<p>A field is a named sequence of terms.</p>
+<li>A term is a sequence of bytes.</li>
+<p>The same sequence of bytes in two different fields is considered a different 
+term. Thus terms are represented as a pair: the string naming the field, and the
+bytes within the field.</p>
+<a name="Inverted_Indexing"></a>
+<h3>Inverted Indexing</h3>
+<p>The index stores statistics about terms in order to make term-based search
+more efficient. Lucene's index falls into the family of indexes known as an
+<i>inverted index.</i> This is because it can list, for a term, the documents
+that contain it. This is the inverse of the natural relationship, in which
+documents list terms.</p>
+<a name="Types_of_Fields"></a>
+<h3>Types of Fields</h3>
+<p>In Lucene, fields may be <i>stored</i>, in which case their text is
+in the index literally, in a non-inverted manner. Fields that are inverted are
+called <i>indexed</i>. A field may be both stored and indexed.</p>
+<p>The text of a field may be <i>tokenized</i> into terms to be indexed,
or the
+text of a field may be used literally as a term to be indexed. Most fields are
+tokenized, but sometimes it is useful for certain identifier fields to be
+indexed literally.</p>
+<p>See the {@link org.apache.lucene.document.Field Field}
+java docs for more information on Fields.</p>
+<a name="Segments" id="Segments"></a>
+<p>Lucene indexes may be composed of multiple sub-indexes, or <i>segments</i>.
+Each segment is a fully independent index, which could be searched separately.
+Indexes evolve by:</p>
+<p>Creating new segments for newly added documents.</p>
+<p>Merging existing segments.</p>
+<p>Searches may involve multiple segments and/or multiple indexes, each index
+potentially composed of a set of segments.</p>
+<a name="Document_Numbers"></a>
+<h3>Document Numbers</h3>
+<p>Internally, Lucene refers to documents by an integer <i>document number</i>.
+The first document added to an index is numbered zero, and each subsequent
+document added gets a number one greater than the previous.</p>
+<p>Note that a document's number may change, so caution should be taken when
+storing these numbers outside of Lucene. In particular, numbers may change in
+the following situations:</p>
+<p>The numbers stored in each segment are unique only within the segment, and
+must be converted before they can be used in a larger context. The standard
+technique is to allocate each segment a range of values, based on the range of
+numbers used in that segment. To convert a document number from a segment to an
+external value, the segment's <i>base</i> document number is added. To convert
+an external value back to a segment-specific value, the segment is identified
+by the range that the external value is in, and the segment's base value is
+subtracted. For example two five document segments might be combined, so that
+the first segment has a base value of zero, and the second of five. Document
+three from the second segment would have an external value of eight.</p>
+<p>When documents are deleted, gaps are created in the numbering. These are
+eventually removed as the index evolves through merging. Deleted documents are
+dropped when segments are merged. A freshly-merged segment thus has no gaps in
+its numbering.</p>
+<a name="Overview" id="Overview"></a>
+<p>Each segment index maintains the following:</p>
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40FieldInfosFormat Field names}.

+   This contains the set of field names used in the index.</p>
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40StoredFieldsFormat Stored Field
+This contains, for each document, a list of attribute-value pairs, where the attributes 
+are field names. These are used to store auxiliary information about the document, such as

+its title, url, or an identifier to access a database. The set of stored fields are what
+returned for each hit when searching. This is keyed by document number.</p>
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term dictionary}.

+A dictionary containing all of the terms used in all of the
+indexed fields of all of the documents. The dictionary also contains the number
+of documents which contain the term, and pointers to the term's frequency and
+proximity data.</p>
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Frequency data}.

+For each term in the dictionary, the numbers of all the
+documents that contain that term, and the frequency of the term in that
+document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)</p>
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Proximity data}.

+For each term in the dictionary, the positions that the
+term occurs in each document. Note that this will not exist if all fields in
+all documents omit position data.</p>
+<p>Normalization factors. For each field in each document, a value is stored
+that is multiplied into the score for hits on that field.</p>
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat Term Vectors}.

+For each field in each document, the term vector (sometimes
+called document vector) may be stored. A term vector consists of term text and
+term frequency. To add Term Vectors to your index see the 
+{@link org.apache.lucene.document.Field Field} constructors</p>
+<p>Per-document values. Like stored values, these are also keyed by document
+number, but are generally intended to be loaded into main memory for fast
+access. Whereas stored values are generally intended for summary results from
+searches, per-document values are useful for things like scoring factors.
+<p>{@link org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat Deleted documents}.

+An optional file indicating which documents are deleted.</p>
+<p>Details on each of these are provided in their linked pages.</p>
+<a name="File_Naming"></a>
+<h2>File Naming</h2>
+<p>All files belonging to a segment have the same name with varying extensions.
+The extensions correspond to the different file formats described below. When
+using the Compound File format (default in 1.4 and greater) these files are
+collapsed into a single .cfs file (see below for details)</p>
+<p>Typically, all segments in an index are stored in a single directory,
+although this is not required.</p>
+<p>As of version 2.1 (lock-less commits), file names are never re-used (there
+is one exception, "segments.gen", see below). That is, when any file is saved
+to the Directory it is given a never before used filename. This is achieved
+using a simple generations approach. For example, the first segments file is
+segments_1, then segments_2, etc. The generation is a sequential long integer
+represented in alpha-numeric (base 36) form.</p>
+<a name="file-names" id="file-names"></a>
+<h2>Summary of File Extensions</h2>
+<p>The following table summarizes the names and extensions of the files in
+<table cellspacing="1" cellpadding="4">
+<th>Brief Description</th>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40SegmentInfosFormat Segments File}</td>
+<td>segments.gen, segments_N</td>
+<td>Stores information about segments</td>
+<td><a href="#Lock_File">Lock File</a></td>
+<td>The Write lock prevents multiple IndexWriters from writing to the same
+<td>{@link Compound File}</td>
+<td>.cfs, .cfe</td>
+<td>An optional "virtual" file consisting of all the other index files for
+systems that frequently run out of file handles.</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40FieldInfosFormat Fields}</td>
+<td>Stores information about the fields</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40StoredFieldsFormat Field Index}</td>
+<td>Contains pointers to field data</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40StoredFieldsFormat Field Data}</td>
+<td>The stored fields for documents</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Dictionary}</td>
+<td>The term dictionary, stores term info</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Index}</td>
+<td>The index into the Term Dictionary</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Frequencies}</td>
+<td>Contains the list of docs which contain each term along with frequency</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Positions}</td>
+<td>Stores position information about where a term occurs in the index</td>
+<td>Encodes length and boost factors for docs and fields</td>
+<td>Per-Document Values</td>
+<td>Encodes additional scoring factors or other per-document information.</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat Term Vector
+<td>Stores offset into the document data file</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat Term Vector
+<td>Contains information about each document that has term vectors</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat Term Vector
+<td>The field level info about term vectors</td>
+<td>{@link org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat Deleted Documents}</td>
+<td>Info about what files are deleted</td>
+<td>{@link org.apache.lucene.codecs.perfield.PerFieldPostingsFormat Field formats}</td>
+<td>Enables per-field {@link org.apache.lucene.codecs.PostingsFormat PostingsFormat}
+<a name="Lock_File" id="Lock_File"></a>
+<h2>Lock File</h2>
+The write lock, which is stored in the index directory by default, is named
+"write.lock". If the lock directory is different from the index directory then
+the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix
+derived from the full path to the index directory. When this file is present, a
+writer is currently modifying the index (adding or removing documents). This
+lock file ensures that only one writer is modifying the index at a time.</p>
+<a name="Limitations" id="Limitations"></a>
+<p>When referring to term numbers, Lucene's current implementation uses a Java
+<code>int</code> to hold the term index, which means the
+maximum number of unique terms in any single index segment is ~2.1 billion
+times the term index interval (default 128) = ~274 billion. This is technically
+not a limitation of the index file format, just of Lucene's current
+<p>Similarly, Lucene uses a Java <code>int</code> to refer to
+document numbers, and the index file format uses an <code>Int32</code>
+on-disk to store document numbers. This is a limitation
+of both the index file format and the current implementation. Eventually these
+should be replaced with either <code>UInt64</code> values, or
+better yet, <code>VInt</code> values which have no limit.</p>

Modified: lucene/dev/trunk/lucene/site/xsl/index.xsl
--- lucene/dev/trunk/lucene/site/xsl/index.xsl (original)
+++ lucene/dev/trunk/lucene/site/xsl/index.xsl Tue May  1 13:42:29 2012
@@ -65,7 +65,7 @@
             <li><a href="changes/Changes.html">Changes</a>: List of changes
in this release.</li>
             <li><a href="MIGRATE.html">Migration Guide</a>: What changed
in Lucene 4; how to migrate code from Lucene 3.x.</li>
             <li><a href="JRE_VERSION_MIGRATION.html">JRE Version Migration</a>:
Information about upgrading between major JRE versions.</li>
-            <li><a href="fileformats.html">File Formats</a>: Guide to the
index format used by Lucene.</li>
+            <li><a href="core/org/apache/lucene/codecs/lucene40/package-summary.html#package_description">File
Formats</a>: Guide to the index format used by Lucene.</li>
             <li><a href="core/org/apache/lucene/search/package-summary.html#package_description">Search
and Scoring in Lucene</a>: Introduction to how Lucene scores documents.</li>
             <li><a href="core/org/apache/lucene/search/similarities/TFIDFSimilarity.html">Classic
Scoring Formula</a>: Formula of Lucene's classic <a href="">Vector
Space</a> implementation. (look <a href="core/org/apache/lucene/search/similarities/package-summary.html#package_description">here</a>
for other models)</li>
             <li><a href="queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description">Classic
QueryParser Syntax</a>: Overview of the Classic QueryParser's syntax and features.</li>

View raw message