lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (Created) (JIRA)" <>
Subject [jira] [Created] (LUCENE-3843) implement PositionLengthAttribute for all tokenstreams where its appropriate
Date Sat, 03 Mar 2012 22:13:57 GMT
implement PositionLengthAttribute for all tokenstreams where its appropriate

                 Key: LUCENE-3843
             Project: Lucene - Java
          Issue Type: Improvement
            Reporter: Robert Muir
             Fix For: 3.6, 4.0

LUCENE-3767 introduces PositionLengthAttribute, which extends the tokenstream API
from a sausage to a real graph. 

Currently tokenstreams such as WordDelimiterFilter and SynonymsFilter theoretically
work at a graph level, but then serialize themselves to a sausage, for example:

wi-fi with WDF creates:
wi(posinc=1), fi(posinc=1), wifi(posinc=0)

So the lossiness is that the 'wifi' is simply stacked ontop of 'fi'

PositionLengthAttribute fixes this by allowing a token to declare how far it "spans",
so we don't lose any information.

While the indexer currently can only support sausages anyway (and for performance reasons,
this is probably just fine!), other tokenstream consumers such as queryparsers and suggesters
such as LUCENE-3842 can actually make use of this information for better behavior.

So I think its ideal if the TokenStream API doesn't reflect the lossiness of the index format,
but instead keeps all information, and after LUCENE-3767 is committed we should fix tokenstreams
to preserve this information for consumers that can use it.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message