Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <1650888949.1213094085202.JavaMail.jira@brutus>
Date: Tue, 10 Jun 2008 03:34:45 -0700 (PDT)
From: "Christopher Morris (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-1292) Tag Index
In-Reply-To: <376504639.1211378875716.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603827#action_12603827 ] 

Christopher Morris commented on LUCENE-1292:
--------------------------------------------

Not sure I understand the question. There is no database or in-memory data.

There are three indexes: the static index (s), the pseudo index (p), and the tag index (t). The index p doesn't exist - it is spoofed from t. The user view is of a ParallelReader wrapping s and p.

Example:

Populate index s with four documents. It is a requirement that the documents have a primary key field (a field known to contain a unique integer). The PKs for these documents are 1,2,3,and 4; ascending order from the document with docID 0.

The user tags documents with PK 1, 3, 4 with the Term("tag","foo").

The index p looks like:

Document 0
==========
tag : foo

Document 2
==========
tag : foo

Document 3
==========
tag : foo

The index t looks like the following (where {n} means the term is stored as occuring at term position 'n'; the term position is (ab)used here to store the primary key for use by p.termDocs()) :

Document: 0
==========
tag : foo_ADD{1}, foo_ADD{3}, foo_ADD{4}
PK_tag : 1_foo,3_foo,4_foo

The user now deletes the Term("tag", "foo") from the PK 3.

The index p looks like:

Document 0
==========
tag : foo

Document 3
==========
tag : foo

The index t looks like the following:

Document: 0
==========
tag : foo_ADD{1}, foo_ADD{3}, foo_ADD{4}
PK_tag : 1_foo,3_foo,4_foo

Document: 0
==========
tag : foo_DEL{3}
PK_tag : 3_foo

The method p.docFreq(new Term("tag", "foo")) runs the following block of code (removed some fail-fast tests and tidy up code for clarity):

int docFreq = 0;
TermDocs additions = t.termDocs(new Term("tag","foo_ADD")); // A TermDocs with one doc. The doc has a freq() of 3.
while (additions.next()) // true; once
{
  docFreq += additions.freq(); // +3
}
deletions = t.termDocs(new Term("tag","foo_DEL")); // A TermDocs with one doc. The doc has a freq() of 1.
while(deletions.next()) // true; once
{
  docFreq -= additions.freq(); // -1
}
return docFreq; // 2

I'd hoped to have test results for very dynamic terms, but I forgot to check that the server had any disk space available before I started the test.

> Tag Index
> ---------
>
>                 Key: LUCENE-1292
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1292
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>         Attachments: lucene-1292.patch
>
>
> The problem the tag index solves is slow field cache loading and range queries, and reindexing an entire document to update fields that are not tokenized.  
> The tag index holds untokenized terms with a docfreq of 1 in a term dictionary like index file.  The file also stores the docs per term, similar to LUCENE-1278.  The index also has a transaction log and in memory index for realtime updates to the tags.  The transaction log is periodically merged into the existing tag term dictionary index file.
> The TagIndexReader extends IndexReader and is unified with a regular index by ParallelReader.  There is a doc id to terms skip pointer file for the IndexReader.document method.  This file contains a pointer for looking up the terms for a document.  
> There is a higher level class that encapsulates writing a document with tag fields to IndexWriter and TagIndexWriter.  This requires a hook into IndexWriter to coordinate doc ids and flushing segments to disk.  
> The writer class could be as simple as:
> {code}
> public class TagIndexWriter {
>   
>   public void add(Term term, DocIdSetIterator iterator) {
>   }
>   
>   public void delete(Term term, DocIdSetIterator iterator) {
>   }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org