lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Li" <>
Subject Re: Payloads
Date Thu, 21 Dec 2006 21:58:33 GMT
> 1. Make the index format extensible by adding user-implementable reader
> and writer interfaces for postings.
> ...
> Here's a very rough, sketchy, first draft of a type (1) proposal.


In approach 1, what is the best abstraction of a flexible index format
for Lucene?

The draft proposal seems to suggest the following (roughly):
  A dictionary entry is <Term, FilePointer>.
  A posting entry for a term in a document is <Doc, PostingContent>.
Classes which implement PostingFormat decide the format of PostingContent.

Storing all the posting content, e.g. frequencies and positions, in a
single file greatly simplifies things. However, this could cause some
performance penalty. For example, boolean query 'Apache AND Lucene'
would have to paw through positions. But position indexing for Apache
and Lucene is necessary to support phrase query '"Apache Lucene"'.

Is it a good idea to allow PostingFormat to decide whether and how to
store posting content in multiple files?
  A dictionary entry is <Term, <FilePointer>+>.
  A posting entry for a term in a document is <Doc, <PostingContent>+>.
Each PostingContent is stored in a separate file.

Or is a two-file abstraction good enough? It supports all formats in
approaches 2 and 3.
  A dictionary entry is <Term, FreqPointer, ProxPointer>.
  A posting entry for a term in a document is <Doc,
PerDocPostingContent, <Position, PerPositionPostingContent>+>.
Doc and PerDocPostingContent are stored in a .frq file.
Position and PerPositionPostingContent are stored in a .prx file.

What Michael called Payload can be viewed as PerPositionPostingContent here.

> I'm not sure this is the best approach: it's just the first one that
> comes to my mind.  Perhaps instead Tokens should have a list of aspects,
> each of which implement a TokenAspect interface, or somesuch.

Making Token have a list of aspects would work. A particular analyzer
would add certain types of aspects to the tokens it emits. For
example, one analyzer adds a TextEmphasis aspect to a token. Another
analyzer adds a PartOfSpeech aspect to the same token. A particular
posting implementation would expect certain types of aspects. For
example, one may require a TextEmphasis aspect and a PartOfSpeech
aspect. The posting implementation generates posting content (payload)
by encoding the values of both aspects.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message