lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <>
Subject RE: not indexing analyzed field
Date Mon, 29 Nov 2010 07:41:17 GMT
Hi Bernd,

> Nevertheless I'm in the need of the functionality to change the stored
> during analysis or tokenization or filtering (what ever works).
> Thats how it can be done in FAST FDS/ESP (full processing) compared to
> Lucene/Solr (sparse processing).
> Do you have any idea, how to change the stored value during analysis or
> tokenization or filtering?

This is simply not possible or even wanted. Stored fields in Lucene are for
storing arbitrary values in the index (they don't even need to be strings).
With RTFM I mean, that you need to completely separate them in your head and
look a them differently. It's indeed not the best idea in Lucene (and Solr)
to provide them together and in one API to the user.

As stored fields are not processed at all, you can simply process then
*before* you put them into Lucene! Why do you want to do that in the
processing pipeline?

> Sure I can make a branch of the trunk and enhance the
> "DocInverterPerField.processFields" class/method and change the line
> "boolean hasMoreTokens = stream.incrementToken();"
> but my hope is still that it is possible without touching the basic Lucene

I would never do this, and as noted above, its uneeded.

To e.g. store something different from what was being analyzed in Lucene,
just add the field two times with same name to o.a.l.d.Document. Once as
stored-only, once as indexed-only (internally that's exactly what Lucene is
also doing when you pass a combined stored/indexed field; stored fields and
indexed fields are completely different - and thats why you want to change

If you are using Solr, there are also possibilities to do this, just
implement your own FieldType, that handles analysis and storing different.
E.g. TrieField in Solr works exactly like that (it indexes tokens as binary
terms and also stores them in a different format). Look at methods


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message