Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5CF6E200BB4 for ; Tue, 1 Nov 2016 19:00:14 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 5B6E4160AF7; Tue, 1 Nov 2016 18:00:14 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2A8B7160ADA for ; Tue, 1 Nov 2016 19:00:13 +0100 (CET) Received: (qmail 41951 invoked by uid 500); 1 Nov 2016 18:00:12 -0000 Mailing-List: contact commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list commits@lucene.apache.org Received: (qmail 41942 invoked by uid 99); 1 Nov 2016 18:00:12 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Nov 2016 18:00:12 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 390F7E0BB1; Tue, 1 Nov 2016 18:00:12 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: mikemccand@apache.org To: commits@lucene.apache.org Message-Id: <25b02c459cf04876aa15884cfdca074e@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: lucene-solr:branch_6x: LUCENE-7532: add back lost codec file format documentation Date: Tue, 1 Nov 2016 18:00:12 +0000 (UTC) archived-at: Tue, 01 Nov 2016 18:00:14 -0000 Repository: lucene-solr Updated Branches: refs/heads/branch_6x 8c42045f2 -> 830a3fdfb LUCENE-7532: add back lost codec file format documentation Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/830a3fdf Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/830a3fdf Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/830a3fdf Branch: refs/heads/branch_6x Commit: 830a3fdfbf12553d47f8f6c320c862aa88fdc48b Parents: 8c42045 Author: Mike McCandless Authored: Tue Nov 1 14:00:00 2016 -0400 Committer: Mike McCandless Committed: Tue Nov 1 14:00:00 2016 -0400 ---------------------------------------------------------------------- lucene/CHANGES.txt | 6 + .../lucene/codecs/lucene62/package-info.java | 400 ++++++++++++++++++- 2 files changed, 402 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/830a3fdf/lucene/CHANGES.txt ---------------------------------------------------------------------- diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt index 34b62d3..0bfb1ad 100644 --- a/lucene/CHANGES.txt +++ b/lucene/CHANGES.txt @@ -6,6 +6,12 @@ http://s.apache.org/luceneversions ======================= Lucene 6.4.0 ======================= (No Changes) +Improvements + +* LUCENE-7532: Add back lost codec file format documentation + (Shinichiro Abe via Mike McCandless) + + ======================= Lucene 6.3.0 ======================= API Changes http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/830a3fdf/lucene/core/src/java/org/apache/lucene/codecs/lucene62/package-info.java ---------------------------------------------------------------------- diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene62/package-info.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene62/package-info.java index 2fe2dc7..170b7bf 100644 --- a/lucene/core/src/java/org/apache/lucene/codecs/lucene62/package-info.java +++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene62/package-info.java @@ -16,9 +16,401 @@ */ /** - * Components from the Lucene 6.2 index format - * See {@link org.apache.lucene.codecs.lucene62} for an overview - * of the index format. + * Lucene 6.2 file format. + * + *

Apache Lucene - Index File Formats

+ * + * + *

Introduction

+ *
+ *

This document defines the index file formats used in this version of Lucene. + * If you are using a different version of Lucene, please consult the copy of + * docs/ that was distributed with + * the version you are using.

+ *

Apache Lucene is written in Java, but several efforts are underway to write + * versions of + * Lucene in other programming languages. If these versions are to remain + * compatible with Apache Lucene, then a language-independent definition of the + * Lucene index format is required. This document thus attempts to provide a + * complete and independent definition of the Apache Lucene file formats.

+ *

As Lucene evolves, this document should evolve. Versions of Lucene in + * different programming languages should endeavor to agree on file formats, and + * generate new versions of this document.

+ *
+ * + *

Definitions

+ *
+ *

The fundamental concepts in Lucene are index, document, field and term.

+ *

An index contains a sequence of documents.

+ *
    + *
  • A document is a sequence of fields.
    • + *
  • A field is a named sequence of terms.
    • + *
  • A term is a sequence of bytes.
    • + *
+ *

The same sequence of bytes in two different fields is considered a different + * term. Thus terms are represented as a pair: the string naming the field, and the + * bytes within the field.

+ * + *

Inverted Indexing

+ *

The index stores statistics about terms in order to make term-based search + * more efficient. Lucene's index falls into the family of indexes known as an + * inverted index. This is because it can list, for a term, the documents + * that contain it. This is the inverse of the natural relationship, in which + * documents list terms.

+ * + *

Types of Fields

+ *

In Lucene, fields may be stored, in which case their text is stored + * in the index literally, in a non-inverted manner. Fields that are inverted are + * called indexed. A field may be both stored and indexed.

+ *

The text of a field may be tokenized into terms to be indexed, or the + * text of a field may be used literally as a term to be indexed. Most fields are + * tokenized, but sometimes it is useful for certain identifier fields to be + * indexed literally.

+ *

See the {@link org.apache.lucene.document.Field Field} + * java docs for more information on Fields.

+ * + *

Segments

+ *

Lucene indexes may be composed of multiple sub-indexes, or segments. + * Each segment is a fully independent index, which could be searched separately. + * Indexes evolve by:

+ *
    + *
  1. Creating new segments for newly added documents.
    2. + *
  2. Merging existing segments.
    3. + *
+ *

Searches may involve multiple segments and/or multiple indexes, each index + * potentially composed of a set of segments.

+ * + *

Document Numbers

+ *

Internally, Lucene refers to documents by an integer document number. + * The first document added to an index is numbered zero, and each subsequent + * document added gets a number one greater than the previous.

+ *

Note that a document's number may change, so caution should be taken when + * storing these numbers outside of Lucene. In particular, numbers may change in + * the following situations:

+ *
    + *
  • + *

    The numbers stored in each segment are unique only within the segment, and + * must be converted before they can be used in a larger context. The standard + * technique is to allocate each segment a range of values, based on the range of + * numbers used in that segment. To convert a document number from a segment to an + * external value, the segment's base document number is added. To convert + * an external value back to a segment-specific value, the segment is identified + * by the range that the external value is in, and the segment's base value is + * subtracted. For example two five document segments might be combined, so that + * the first segment has a base value of zero, and the second of five. Document + * three from the second segment would have an external value of eight.

    + *
    • + *
  • + *

    When documents are deleted, gaps are created in the numbering. These are + * eventually removed as the index evolves through merging. Deleted documents are + * dropped when segments are merged. A freshly-merged segment thus has no gaps in + * its numbering.

    + *
    • + *
+ *
+ * + *

Index Structure Overview

+ *
+ *

Each segment index maintains the following:

+ *
    + *
  • + * {@link org.apache.lucene.codecs.lucene62.Lucene62SegmentInfoFormat Segment info}. + * This contains metadata about a segment, such as the number of documents, + * what files it uses, + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50FieldInfosFormat Field names}. + * This contains the set of field names used in the index. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat Stored Field values}. + * This contains, for each document, a list of attribute-value pairs, where the attributes + * are field names. These are used to store auxiliary information about the document, such as + * its title, url, or an identifier to access a database. The set of stored fields are what is + * returned for each hit when searching. This is keyed by document number. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Term dictionary}. + * A dictionary containing all of the terms used in all of the + * indexed fields of all of the documents. The dictionary also contains the number + * of documents which contain the term, and pointers to the term's frequency and + * proximity data. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Term Frequency data}. + * For each term in the dictionary, the numbers of all the + * documents that contain that term, and the frequency of the term in that + * document, unless frequencies are omitted (IndexOptions.DOCS_ONLY) + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Term Proximity data}. + * For each term in the dictionary, the positions that the + * term occurs in each document. Note that this will not exist if all fields in + * all documents omit position data. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene53.Lucene53NormsFormat Normalization factors}. + * For each field in each document, a value is stored + * that is multiplied into the score for hits on that field. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50TermVectorsFormat Term Vectors}. + * For each field in each document, the term vector (sometimes + * called document vector) may be stored. A term vector consists of term text and + * term frequency. To add Term Vectors to your index see the + * {@link org.apache.lucene.document.Field Field} constructors + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat Per-document values}. + * Like stored values, these are also keyed by document + * number, but are generally intended to be loaded into main memory for fast + * access. Whereas stored values are generally intended for summary results from + * searches, per-document values are useful for things like scoring factors. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat Live documents}. + * An optional file indicating which documents are live. + *
    • + *
  • + * {@link org.apache.lucene.codecs.lucene60.Lucene60PointsFormat Point values}. + * Optional pair of files, recording dimensionally indexed fields, to enable fast + * numeric range filtering and large numeric values like BigInteger and BigDecimal (1D) + * and geographic shape intersection (2D, 3D). + *
    • + *
+ *

Details on each of these are provided in their linked pages.

+ *
+ * + *

File Naming

+ *
+ *

All files belonging to a segment have the same name with varying extensions. + * The extensions correspond to the different file formats described below. When + * using the Compound File format (default in 1.4 and greater) these files (except + * for the Segment info file, the Lock file, and Deleted documents file) are collapsed + * into a single .cfs file (see below for details)

+ *

Typically, all segments in an index are stored in a single directory, + * although this is not required.

+ *

As of version 2.1 (lock-less commits), file names are never re-used. + * That is, when any file is saved + * to the Directory it is given a never before used filename. This is achieved + * using a simple generations approach. For example, the first segments file is + * segments_1, then segments_2, etc. The generation is a sequential long integer + * represented in alpha-numeric (base 36) form.

+ *
+ * + *

Summary of File Extensions

+ *
+ *

The following table summarizes the names and extensions of the files in + * Lucene:

+ * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + * + *
NameExtensionBrief Description
{@link org.apache.lucene.index.SegmentInfos Segments File}segments_NStores information about a commit point
Lock Filewrite.lockThe Write lock prevents multiple IndexWriters from writing to the same + * file.
{@link org.apache.lucene.codecs.lucene62.Lucene62SegmentInfoFormat Segment Info}.siStores metadata about a segment
{@link org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat Compound File}.cfs, .cfeAn optional "virtual" file consisting of all the other index files for + * systems that frequently run out of file handles.
{@link org.apache.lucene.codecs.lucene50.Lucene50FieldInfosFormat Fields}.fnmStores information about the fields
{@link org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat Field Index}.fdxContains pointers to field data
{@link org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat Field Data}.fdtThe stored fields for documents
{@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Term Dictionary}.timThe term dictionary, stores term info
{@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Term Index}.tipThe index into the Term Dictionary
{@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Frequencies}.docContains the list of docs which contain each term along with frequency
{@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Positions}.posStores position information about where a term occurs in the index
{@link org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat Payloads}.payStores additional per-position metadata information such as character offsets and user payloads
{@link org.apache.lucene.codecs.lucene53.Lucene53NormsFormat Norms}.nvd, .nvmEncodes length and boost factors for docs and fields
{@link org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat Per-Document Values}.dvd, .dvmEncodes additional scoring factors or other per-document information.
{@link org.apache.lucene.codecs.lucene50.Lucene50TermVectorsFormat Term Vector Index}.tvxStores offset into the document data file
{@link org.apache.lucene.codecs.lucene50.Lucene50TermVectorsFormat Term Vector Documents}.tvdContains information about each document that has term vectors
{@link org.apache.lucene.codecs.lucene50.Lucene50TermVectorsFormat Term Vector Fields}.tvfThe field level info about term vectors
{@link org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat Live Documents}.livInfo about what files are live
{@link org.apache.lucene.codecs.lucene60.Lucene60PointsFormat Point values}.dii, .dimHolds indexed points, if any
+ *
+ * + *

Lock File

+ * The write lock, which is stored in the index directory by default, is named + * "write.lock". If the lock directory is different from the index directory then + * the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix + * derived from the full path to the index directory. When this file is present, a + * writer is currently modifying the index (adding or removing documents). This + * lock file ensures that only one writer is modifying the index at a time. + * + *

History

+ *

Compatibility notes are provided in this document, describing how file + * formats have changed from prior versions:

+ *
    + *
  • In version 2.1, the file format was changed to allow lock-less commits (ie, + * no more commit lock). The change is fully backwards compatible: you can open a + * pre-2.1 index for searching or adding/deleting of docs. When the new segments + * file is saved (committed), it will be written in the new file format (meaning + * no specific "upgrade" process is needed). But note that once a commit has + * occurred, pre-2.1 Lucene will not be able to read the index.
    • + *
  • In version 2.3, the file format was changed to allow segments to share a + * single set of doc store (vectors & stored fields) files. This allows for + * faster indexing in certain cases. The change is fully backwards compatible (in + * the same way as the lock-less commits change in 2.1).
    • + *
  • In version 2.4, Strings are now written as true UTF-8 byte sequence, not + * Java's modified UTF-8. See + * LUCENE-510 for details.
    • + *
  • In version 2.9, an optional opaque Map<String,String> CommitUserData + * may be passed to IndexWriter's commit methods (and later retrieved), which is + * recorded in the segments_N file. See + * LUCENE-1382 for details. Also, + * diagnostics were added to each segment written recording details about why it + * was written (due to flush, merge; which OS/JRE was used; etc.). See issue + * LUCENE-1654 for details.
    • + *
  • In version 3.0, compressed fields are no longer written to the index (they + * can still be read, but on merge the new segment will write them, uncompressed). + * See issue LUCENE-1960 + * for details.
    • + *
  • In version 3.1, segments records the code version that created them. See + * LUCENE-2720 for details. + * Additionally segments track explicitly whether or not they have term vectors. + * See LUCENE-2811 + * for details.
    • + *
  • In version 3.2, numeric fields are written as natively to stored fields + * file, previously they were stored in text format only.
    • + *
  • In version 3.4, fields can omit position data while still indexing term + * frequencies.
    • + *
  • In version 4.0, the format of the inverted index became extensible via + * the {@link org.apache.lucene.codecs.Codec Codec} api. Fast per-document storage + * ({@code DocValues}) was introduced. Normalization factors need no longer be a + * single byte, they can be any {@link org.apache.lucene.index.NumericDocValues NumericDocValues}. + * Terms need not be unicode strings, they can be any byte sequence. Term offsets + * can optionally be indexed into the postings lists. Payloads can be stored in the + * term vectors.
    • + *
  • In version 4.1, the format of the postings list changed to use either + * of FOR compression or variable-byte encoding, depending upon the frequency + * of the term. Terms appearing only once were changed to inline directly into + * the term dictionary. Stored fields are compressed by default.
    • + *
  • In version 4.2, term vectors are compressed by default. DocValues has + * a new multi-valued type (SortedSet), that can be used for faceting/grouping/joining + * on multi-valued fields.
    • + *
  • In version 4.5, DocValues were extended to explicitly represent missing values.
    • + *
  • In version 4.6, FieldInfos were extended to support per-field DocValues generation, to + * allow updating NumericDocValues fields.
    • + *
  • In version 4.8, checksum footers were added to the end of each index file + * for improved data integrity. Specifically, the last 8 bytes of every index file + * contain the zlib-crc32 checksum of the file.
    • + *
  • In version 4.9, DocValues has a new multi-valued numeric type (SortedNumeric) + * that is suitable for faceting/sorting/analytics. + *
  • In version 5.4, DocValues have been improved to store more information on disk: + * addresses for binary fields and ord indexes for multi-valued fields. + *
  • In version 6.0, Points were added, for multi-dimensional range/distance search. + *
  • In version 6.2, new Segment info format that reads/writes the index sort, to support index sorting. + *
    • + *
+ * + *

Limitations

+ *
+ *

Lucene uses a Java int to refer to + * document numbers, and the index file format uses an Int32 + * on-disk to store document numbers. This is a limitation + * of both the index file format and the current implementation. Eventually these + * should be replaced with either UInt64 values, or + * better yet, {@link org.apache.lucene.store.DataOutput#writeVInt VInt} values which have no limit.

+ *
*/ - package org.apache.lucene.codecs.lucene62;