lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (Updated) (JIRA)" <>
Subject [jira] [Updated] (LUCENE-3848) basetokenstreamtestcase should fail if tokenstream starts with posinc=0
Date Sun, 04 Mar 2012 19:42:59 GMT


Robert Muir updated LUCENE-3848:

    Attachment: LUCENE-3848.patch

patch fixing the bug in WikipediaTokenizer.

But i think we just dont have good tests for the removers.

Ideally for tests i think we should have a simple 'MockSynonymsFilter' that is juts stupid
and slow and makes certain synonyms (maybe some multi-word) to use in testing.

Then we can write tests to find and fix the bugs in the removingfilter.
> basetokenstreamtestcase should fail if tokenstream starts with posinc=0
> -----------------------------------------------------------------------
>                 Key: LUCENE-3848
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Robert Muir
>             Fix For: 3.6, 4.0
>         Attachments: LUCENE-3848.patch
> This is meaningless for a tokenstream to start with posinc=0,
> Its also caused problems and hairiness in the indexer (LUCENE-1255, LUCENE-1542),
> and it makes senseless tokenstreams. We should add a check and fix any that do this.
> Furthermore the same bug can exist in removing-filters if they have enablePositionIncrements=false.
> I think this option is useful: but it shouldnt mean 'allow broken tokenstream', it just
means we
> don't add gaps. 
> If you remove tokens with enablePositionIncrements=false it should not cause the TS to
start with
> positionincrement=0, and it shouldnt 'restructure' the tokenstream (e.g. moving synonyms
on top of a different word).
> It should just not add any 'holes'.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message