lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Balmain" <>
Subject Re: Global field semantics
Date Mon, 10 Jul 2006 01:44:17 GMT
On 7/10/06, Chuck Williams <> wrote:
> Marvin Humphrey wrote on 07/08/2006 11:13 PM:
> >
> > On Jul 8, 2006, at 9:46 AM, Chuck Williams wrote:
> >
> >> Many things would be cleaner in Lucene if fields had a global semantics,
> >> i.e., if properties like text vs. binary, Index, Store, TermVector, the
> >> appropriate Analyzer, the assignment of Directory in ParallelReader (or
> >> ParallelWriter), etc. were a function of just the field name and the
> >> index.
> >
> > In June, Dave Balmain and I discussed the issue extensively on the
> > Ferret list.  It might have been nice to use the Lucy list, since a
> > lot of the discussion was about Lucy, but the Lucy lists didn't exist
> > at the time.
> >
> >
> >
> I think there are a number of problems with that proposal and hope it
> was not adopted.

Hi Chuck,

Actually, it was adopted and I'm quite happy with the solution. I'd be
very interested to hear what the number of problems are, besides the
example you've already given. Even if you never use Ferret, it can
only help me improve my software.

I'll start by covering your term-vector example. By adding fixed
index-wide field properties to Ferret I was able to obtain up to a
huge speed improvement during indexing. I believe Marvin has had
similar success using his own merge model and with fixed field
properties in KinoSearch. With the CPU time I gain in Ferret I could
easily re-analyze large fields and build term vectors for them
separately. It's a little more work for less common use cases like
yours but in the end, everyone benifits in terms of performance.

> As my earlier example showed, there is at least one
> valid use case where storing a term vector is not an invariant property
> of a field; specifically, when using term vectors to optimize excerpt
> generation, it is best to store them only for fields that have long
> values.  This is even a counter-example to Karl's proposal, since a
> single Document may have multiple fields of the same name, some with
> long values and others with short values; multiple fields of the same
> name may legitimately have different TermVector settings even on a
> single Document.

I think you'll find if you look at the DocumentWriter#writePostings
method that it's "one in, all in" in terms of storing term vectors for
a field. That is, if you have 5 "content" fields and only one of those
is set to store term vectors, then all of the fields will store term

> As another counter-example from my own app which I'd forgotten
> yesterday, an important case where the Analyzer will vary across
> documents is for i18n, where different languages require different
> analyzers.  Refuting again my own argument about this not being
> consistent with query parsing, the language of the query is a distinct
> property from the languages of various documents in the collection.  In
> my app, I let the user specify the language of the query, while the
> language of each Document is determined automatically.  So, analyzers
> vary for both queries and documents, but independently.

Ferret doesn't record any details about analysis in the field
properties. I definitely agree with you here.

> I haven't thought of cases where Index or Store would legitimately vary
> across Fields or Documents, but am less convinced there aren't important
> use cases for these as well.  Similarly, although it is important to
> allow term vectors to be on or off at the field level, I don't see any
> obvious need to vary the type of term vector (positions, offsets or both).

I think Store could definitely legitimately vary across Fields or
Documents for the same reason your term vectors do. Perhaps you are
indexing pages from the web and you want to cache only the smaller

> There are significant benefits to global semantics, as evidenced by the
> fact that several of us independently came to desire this.  However,
> deciding what can be global and what cannot is more subtle.

I agree. I can't see global field semantics making it into Lucene in
the short term. It's a rather large change, particularly if you want
to make full use of the performance benifits it affords.

> Perhaps the best thing at the Lucene level is to have a notion of
> default semantics for a field name.  Whenever a Field of that name is
> constructed, those semantics would be used unless the constructor
> overrides them.  This would allow additional constructors on Field with
> simpler signatures for the common case of invariant Field properties.
> It would also allow applications to access the class that holds the
> default field information for an index.  The application will know which
> properties it can rely on as invariant and whether or not the set of
> fields is closed.
> This approach would preserve upward compatibility and provide, I
> believe, most of the benefits we all seek.
> Thoughts?

If this is all you are going to add, I don't think you'd need to
change Lucene. You could just implement a DocumentFactory in your own
application. Perhaps something like this could go in the contrib
section of Lucene.

Also, you mentioned earlier having a field validating query parser.
You can already use
IndexWriter#getFieldNames(IndexReader.FieldOption.INDEXED) to get all
the indexed fields.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message