lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rm...@apache.org
Subject svn commit: r1332652 - /lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html
Date Tue, 01 May 2012 13:57:11 GMT
Author: rmuir
Date: Tue May  1 13:57:11 2012
New Revision: 1332652

URL: http://svn.apache.org/viewvc?rev=1332652&view=rev
Log:
LUCENE-2946: restructure page and cleanup formatting

Modified:
    lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html

Modified: lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html?rev=1332652&r1=1332651&r2=1332652&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html (original)
+++ lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/codecs/lucene40/package.html Tue
May  1 13:57:11 2012
@@ -25,7 +25,7 @@ Lucene 4.0 file format.
 <h1>Apache Lucene - Index File Formats</h1>
 <div>
 <ul>
-<li><a href="#Index_File_Formats">Index File Formats</a></li>
+<li><a href="#Introduction">Introduction</a></li>
 <li><a href="#Definitions">Definitions</a>
 <ul>
 <li><a href="#Inverted_Indexing">Inverted Indexing</a></li>
@@ -34,16 +34,17 @@ Lucene 4.0 file format.
 <li><a href="#Document_Numbers">Document Numbers</a></li>
 </ul>
 </li>
-<li><a href="#Overview">Overview</a></li>
+<li><a href="#Overview">Index Structure Overview</a></li>
 <li><a href="#File_Naming">File Naming</a></li>
 <li><a href="#file-names">Summary of File Extensions</a></li>
 <ul>
 <li><a href="#Lock_File">Lock File</a></li>
+<li><a href="#History">History</a></li>
 <li><a href="#Limitations">Limitations</a></li>
 </ul>
 </div>
-<a name="Index_File_Formats"></a>
-<h2>Index File Formats</h2>
+<a name="Introduction"></a>
+<h2>Introduction</h2>
 <div>
 <p>This document defines the index file formats used in this version of Lucene.
 If you are using a different version of Lucene, please consult the copy of
@@ -58,43 +59,6 @@ complete and independent definition of t
 <p>As Lucene evolves, this document should evolve. Versions of Lucene in
 different programming languages should endeavor to agree on file formats, and
 generate new versions of this document.</p>
-<p>Compatibility notes are provided in this document, describing how file
-formats have changed from prior versions.</p>
-<p>In version 2.1, the file format was changed to allow lock-less commits (ie,
-no more commit lock). The change is fully backwards compatible: you can open a
-pre-2.1 index for searching or adding/deleting of docs. When the new segments
-file is saved (committed), it will be written in the new file format (meaning
-no specific "upgrade" process is needed). But note that once a commit has
-occurred, pre-2.1 Lucene will not be able to read the index.</p>
-<p>In version 2.3, the file format was changed to allow segments to share a
-single set of doc store (vectors &amp; stored fields) files. This allows for
-faster indexing in certain cases. The change is fully backwards compatible (in
-the same way as the lock-less commits change in 2.1).</p>
-<p>In version 2.4, Strings are now written as true UTF-8 byte sequence, not
-Java's modified UTF-8. See issue LUCENE-510 for details.</p>
-<p>In version 2.9, an optional opaque Map&lt;String,String&gt; CommitUserData
-may be passed to IndexWriter's commit methods (and later retrieved), which is
-recorded in the segments_N file. See issue LUCENE-1382 for details. Also,
-diagnostics were added to each segment written recording details about why it
-was written (due to flush, merge; which OS/JRE was used; etc.). See issue
-LUCENE-1654 for details.</p>
-<p>In version 3.0, compressed fields are no longer written to the index (they
-can still be read, but on merge the new segment will write them, uncompressed).
-See issue LUCENE-1960 for details.</p>
-<p>In version 3.1, segments records the code version that created them. See
-LUCENE-2720 for details. Additionally segments track explicitly whether or not
-they have term vectors. See LUCENE-2811 for details.</p>
-<p>In version 3.2, numeric fields are written as natively to stored fields
-file, previously they were stored in text format only.</p>
-<p>In version 3.4, fields can omit position data while still indexing term
-frequencies.</p>
-<p>In version 4.0, the format of the inverted index became extensible via
-the {@link org.apache.lucene.codecs.Codec Codec} api. Fast per-document storage
-({@link org.apache.lucene.index.DocValues DocValues}) was introduced. Normalization
-factors need no longer be a single byte, they can be any DocValues 
-{@link org.apache.lucene.index.DocValues.Type type}. Terms need not be unicode
-strings, they can be any byte sequence. Term offsets can optionally be indexed 
-into the postings lists.</p>
 </div>
 <a name="Definitions" id="Definitions"></a>
 <h2>Definitions</h2>
@@ -102,12 +66,8 @@ into the postings lists.</p>
 <p>The fundamental concepts in Lucene are index, document, field and term.</p>
 <p>An index contains a sequence of documents.</p>
 <ul>
-<li>
-<p>A document is a sequence of fields.</p>
-</li>
-<li>
-<p>A field is a named sequence of terms.</p>
-</li>
+<li>A document is a sequence of fields.</li>
+<li>A field is a named sequence of terms.</li>
 <li>A term is a sequence of bytes.</li>
 </ul>
 <p>The same sequence of bytes in two different fields is considered a different 
@@ -137,12 +97,8 @@ java docs for more information on Fields
 Each segment is a fully independent index, which could be searched separately.
 Indexes evolve by:</p>
 <ol>
-<li>
-<p>Creating new segments for newly added documents.</p>
-</li>
-<li>
-<p>Merging existing segments.</p>
-</li>
+<li>Creating new segments for newly added documents.</li>
+<li>Merging existing segments.</li>
 </ol>
 <p>Searches may involve multiple segments and/or multiple indexes, each index
 potentially composed of a set of segments.</p>
@@ -151,7 +107,6 @@ potentially composed of a set of segment
 <p>Internally, Lucene refers to documents by an integer <i>document number</i>.
 The first document added to an index is numbered zero, and each subsequent
 document added gets a number one greater than the previous.</p>
-<p><br></p>
 <p>Note that a document's number may change, so caution should be taken when
 storing these numbers outside of Lucene. In particular, numbers may change in
 the following situations:</p>
@@ -177,60 +132,60 @@ its numbering.</p>
 </ul>
 </div>
 <a name="Overview" id="Overview"></a>
-<h2>Overview</h2>
+<h2>Index Structure Overview</h2>
 <div>
 <p>Each segment index maintains the following:</p>
 <ul>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40FieldInfosFormat Field names}.

-   This contains the set of field names used in the index.</p>
+{@link org.apache.lucene.codecs.lucene40.Lucene40FieldInfosFormat Field names}. 
+   This contains the set of field names used in the index.
 </li>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40StoredFieldsFormat Stored Field
values}. 
+{@link org.apache.lucene.codecs.lucene40.Lucene40StoredFieldsFormat Stored Field values}.

 This contains, for each document, a list of attribute-value pairs, where the attributes 
 are field names. These are used to store auxiliary information about the document, such as

 its title, url, or an identifier to access a database. The set of stored fields are what
is 
-returned for each hit when searching. This is keyed by document number.</p>
+returned for each hit when searching. This is keyed by document number.
 </li>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term dictionary}.

+{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term dictionary}. 
 A dictionary containing all of the terms used in all of the
 indexed fields of all of the documents. The dictionary also contains the number
 of documents which contain the term, and pointers to the term's frequency and
-proximity data.</p>
+proximity data.
 </li>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Frequency data}.

+{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Frequency data}. 
 For each term in the dictionary, the numbers of all the
 documents that contain that term, and the frequency of the term in that
-document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)</p>
+document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
 </li>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Proximity data}.

+{@link org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat Term Proximity data}. 
 For each term in the dictionary, the positions that the
 term occurs in each document. Note that this will not exist if all fields in
-all documents omit position data.</p>
+all documents omit position data.
 </li>
 <li>
-<p>Normalization factors. For each field in each document, a value is stored
-that is multiplied into the score for hits on that field.</p>
+Normalization factors. For each field in each document, a value is stored
+that is multiplied into the score for hits on that field.
 </li>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat Term Vectors}.

+{@link org.apache.lucene.codecs.lucene40.Lucene40TermVectorsFormat Term Vectors}. 
 For each field in each document, the term vector (sometimes
 called document vector) may be stored. A term vector consists of term text and
 term frequency. To add Term Vectors to your index see the 
-{@link org.apache.lucene.document.Field Field} constructors</p>
+{@link org.apache.lucene.document.Field Field} constructors
 </li>
 <li>
-<p>Per-document values. Like stored values, these are also keyed by document
+Per-document values. Like stored values, these are also keyed by document
 number, but are generally intended to be loaded into main memory for fast
 access. Whereas stored values are generally intended for summary results from
 searches, per-document values are useful for things like scoring factors.
 </li>
 <li>
-<p>{@link org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat Deleted documents}.

-An optional file indicating which documents are deleted.</p>
+{@link org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat Deleted documents}. 
+An optional file indicating which documents are deleted.
 </li>
 </ul>
 <p>Details on each of these are provided in their linked pages.</p>
@@ -359,6 +314,47 @@ the write lock will be named "XXXX-write
 derived from the full path to the index directory. When this file is present, a
 writer is currently modifying the index (adding or removing documents). This
 lock file ensures that only one writer is modifying the index at a time.</p>
+<a name="History"></a>
+<h2>History</h2>
+<p>Compatibility notes are provided in this document, describing how file
+formats have changed from prior versions:</p>
+<ul>
+<li>In version 2.1, the file format was changed to allow lock-less commits (ie,
+no more commit lock). The change is fully backwards compatible: you can open a
+pre-2.1 index for searching or adding/deleting of docs. When the new segments
+file is saved (committed), it will be written in the new file format (meaning
+no specific "upgrade" process is needed). But note that once a commit has
+occurred, pre-2.1 Lucene will not be able to read the index.</li>
+<li>In version 2.3, the file format was changed to allow segments to share a
+single set of doc store (vectors &amp; stored fields) files. This allows for
+faster indexing in certain cases. The change is fully backwards compatible (in
+the same way as the lock-less commits change in 2.1).</li>
+<li>In version 2.4, Strings are now written as true UTF-8 byte sequence, not
+Java's modified UTF-8. See issue LUCENE-510 for details.</li>
+<li>In version 2.9, an optional opaque Map&lt;String,String&gt; CommitUserData
+may be passed to IndexWriter's commit methods (and later retrieved), which is
+recorded in the segments_N file. See issue LUCENE-1382 for details. Also,
+diagnostics were added to each segment written recording details about why it
+was written (due to flush, merge; which OS/JRE was used; etc.). See issue
+LUCENE-1654 for details.</li>
+<li>In version 3.0, compressed fields are no longer written to the index (they
+can still be read, but on merge the new segment will write them, uncompressed).
+See issue LUCENE-1960 for details.</li>
+<li>In version 3.1, segments records the code version that created them. See
+LUCENE-2720 for details. Additionally segments track explicitly whether or not
+they have term vectors. See LUCENE-2811 for details.</li>
+<li>In version 3.2, numeric fields are written as natively to stored fields
+file, previously they were stored in text format only.</li>
+<li>In version 3.4, fields can omit position data while still indexing term
+frequencies.</li>
+<li>In version 4.0, the format of the inverted index became extensible via
+the {@link org.apache.lucene.codecs.Codec Codec} api. Fast per-document storage
+({@link org.apache.lucene.index.DocValues DocValues}) was introduced. Normalization
+factors need no longer be a single byte, they can be any DocValues 
+{@link org.apache.lucene.index.DocValues.Type type}. Terms need not be unicode
+strings, they can be any byte sequence. Term offsets can optionally be indexed 
+into the postings lists.</li>
+</ul>
 <a name="Limitations" id="Limitations"></a>
 <h2>Limitations</h2>
 <div>



Mime
View raw message