lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Otis Gospodnetic (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-400) NGramFilter -- construct n-grams from a TokenStream
Date Sun, 09 Jul 2006 14:52:30 GMT
    [ http://issues.apache.org/jira/browse/LUCENE-400?page=comments#action_12419913 ] 

Otis Gospodnetic commented on LUCENE-400:
-----------------------------------------

Sebastian, ever figured out the problem?  Also, is there a way to get rid of the Commons Collections?
 Lucene has no run-time dependencies on other libraries.

> NGramFilter -- construct n-grams from a TokenStream
> ---------------------------------------------------
>
>          Key: LUCENE-400
>          URL: http://issues.apache.org/jira/browse/LUCENE-400
>      Project: Lucene - Java
>         Type: Improvement

>   Components: Analysis
>     Versions: unspecified
>  Environment: Operating System: All
> Platform: All
>     Reporter: Sebastian Kirsch
>     Assignee: Lucene Developers
>     Priority: Minor
>  Attachments: NGramAnalyzerWrapper.java, NGramAnalyzerWrapperTest.java, NGramFilter.java,
NGramFilterTest.java
>
> This filter constructs n-grams (token combinations up to a fixed size, sometimes
> called "shingles") from a token stream.
> The filter sets start offsets, end offsets and position increments, so
> highlighting and phrase queries should work.
> Position increments > 1 in the input stream are replaced by filler tokens
> (tokens with termText "_" and endOffset - startOffset = 0) in the output
> n-grams. (Position increments > 1 in the input stream are usually caused by
> removing some tokens, eg. stopwords, from a stream.)
> The filter uses CircularFifoBuffer and UnboundedFifoBuffer from Apache
> Commons-Collections.
> Filter, test case and an analyzer are attached.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message