lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Morris (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1292) Tag Index
Date Tue, 10 Jun 2008 10:34:45 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603827#action_12603827
] 

Christopher Morris commented on LUCENE-1292:
--------------------------------------------

Not sure I understand the question. There is no database or in-memory data.

There are three indexes: the static index (s), the pseudo index (p), and the tag index (t).
The index p doesn't exist - it is spoofed from t. The user view is of a ParallelReader wrapping
s and p.

Example:

Populate index s with four documents. It is a requirement that the documents have a primary
key field (a field known to contain a unique integer). The PKs for these documents are 1,2,3,and
4; ascending order from the document with docID 0.

The user tags documents with PK 1, 3, 4 with the Term("tag","foo").

The index p looks like:

Document 0
==========
tag : foo

Document 2
==========
tag : foo

Document 3
==========
tag : foo

The index t looks like the following (where {n} means the term is stored as occuring at term
position 'n'; the term position is (ab)used here to store the primary key for use by p.termDocs())
:

Document: 0
==========
tag : foo_ADD{1}, foo_ADD{3}, foo_ADD{4}
PK_tag : 1_foo,3_foo,4_foo

The user now deletes the Term("tag", "foo") from the PK 3.

The index p looks like:

Document 0
==========
tag : foo

Document 3
==========
tag : foo

The index t looks like the following:

Document: 0
==========
tag : foo_ADD{1}, foo_ADD{3}, foo_ADD{4}
PK_tag : 1_foo,3_foo,4_foo

Document: 0
==========
tag : foo_DEL{3}
PK_tag : 3_foo

The method p.docFreq(new Term("tag", "foo")) runs the following block of code (removed some
fail-fast tests and tidy up code for clarity):

int docFreq = 0;
TermDocs additions = t.termDocs(new Term("tag","foo_ADD")); // A TermDocs with one doc. The
doc has a freq() of 3.
while (additions.next()) // true; once
{
  docFreq += additions.freq(); // +3
}
deletions = t.termDocs(new Term("tag","foo_DEL")); // A TermDocs with one doc. The doc has
a freq() of 1.
while(deletions.next()) // true; once
{
  docFreq -= additions.freq(); // -1
}
return docFreq; // 2

I'd hoped to have test results for very dynamic terms, but I forgot to check that the server
had any disk space available before I started the test.

> Tag Index
> ---------
>
>                 Key: LUCENE-1292
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1292
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>    Affects Versions: 2.3.1
>            Reporter: Jason Rutherglen
>         Attachments: lucene-1292.patch
>
>
> The problem the tag index solves is slow field cache loading and range queries, and reindexing
an entire document to update fields that are not tokenized.  
> The tag index holds untokenized terms with a docfreq of 1 in a term dictionary like index
file.  The file also stores the docs per term, similar to LUCENE-1278.  The index also has
a transaction log and in memory index for realtime updates to the tags.  The transaction log
is periodically merged into the existing tag term dictionary index file.
> The TagIndexReader extends IndexReader and is unified with a regular index by ParallelReader.
 There is a doc id to terms skip pointer file for the IndexReader.document method.  This file
contains a pointer for looking up the terms for a document.  
> There is a higher level class that encapsulates writing a document with tag fields to
IndexWriter and TagIndexWriter.  This requires a hook into IndexWriter to coordinate doc ids
and flushing segments to disk.  
> The writer class could be as simple as:
> {code}
> public class TagIndexWriter {
>   
>   public void add(Term term, DocIdSetIterator iterator) {
>   }
>   
>   public void delete(Term term, DocIdSetIterator iterator) {
>   }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message