lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki>
Subject Analyzer and Fieldable, different stored and indexed values
Date Wed, 27 Aug 2008 14:47:30 GMT
Hi all,

I recently had a situation where I had to pass some metadata information 
to Analyzer. This metadata was specific to a Document instance (short 
story is that the analysis of some fields depended on data coming from 
other fields, and the number of possible values was too big to use 
separate fields for each combination).

It would be nice to have an Analyzer.tokenStream(String fieldName, Field 
f), or even better tokenStream(String fieldName, Document doc) ... but 
probably it's too intrusive to change this. Although I would be happy to 
have tokenStream(String, Fieldable), because then I could provide my own 
Fieldable with metadata.

In the meantime, having neither option, I came up with an idea: I will 
use a subclass of Reader, and attach my metadata there, and then use 
this Reader when creating a Field. However, I quickly discovered that if 
you set a Reader on a Field, this field automatically becomes un-stored 
- not what I wanted ... Field is declared final, so no luck there.

In the end I implemented a Fieldable, which sort of breaks the contract 
for Fieldable - but it works :) . Namely, my Fieldable returns both 
readerValue() and stringValue(). The first method returns my subclass of 
Reader with metadata, and the second returns the value to be stored.

The reason why it works is that DocInverterPerField first checks the 
tokenStreamValue, then the readerValue, and only then the stringValue 
that it converts to a Reader - so in my case it uses the supplied 
readerValue. At the same time, FieldsWriter, which is responsible for 
storing field values, uses just the stringValue (or binaryValue, but 
that wasn't relevant to my case), which is also set to non-null.

So, here are my thoughts on this, and I'd appreciate any comments on this:

* is this a justified use of the API? it works, at least at the moment 
;) and I couldn't find any other way to accomplish this task.

* could we perhaps relax the restriction on Fieldable so that it can 
return non-null values from more than one method, and clearly document 
in what sequence they are processed? This is already hinted at in the 

* I propose to add a new API to Analyzer:

   public TokenStream tokenStream(String fieldName, Fieldable field);

to support use cases like the one I described above. The default 
implementation could be something like this:

   public TokenStream tokenStream(String fieldName, Fieldable field) {
	Reader r = field.readerValue();
	if (r == null) {
		String s = field.stringValue();
		r = new StringReader(s);
	return tokenStream(fieldName, r);

Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration  Contact: info at sigram dot com

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message