lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rowe (JIRA)" <>
Subject [jira] Commented: (LUCENE-2749) Co-occurrence filter
Date Sun, 13 Mar 2011 19:06:59 GMT


Steven Rowe commented on LUCENE-2749:

Hi Elmar,

I haven't had a chance to do more than an hour or two of work on this, and that was a while
back, so please feel free to run with it.

You should know, though, that Robert Muir and Yonik Seeley (both Lucene/Solr developers) expressed
skepticism (on #lucene IRC) about whether this filter belongs in Lucene itself, because in
their experience, collocations are used by non-search software, and they believe that Lucene
should remain focused exclusively on search.  

Robert Muir also thinks that components that support Boolean search (i.e., not ranked search)
should go elsewhere.  

I personally disagree with these restrictions in general, and I think that a co-occurrence
filter could directly support search.  See this mailing list discussion
for an example I gave (and one of the reasons I made this issue):
. In this thread, I described a way to solve the original poster's problem using a co-occurrence
filter exactly like the one proposed here.

I mention all this to caution you that work you put in here may never be committed to Lucene

The mailing list thread I mentioned above describes the main limitations a filter like this
will have: combinatoric explosion of generated terms.  I haven't figured out how to manage
this, but it occurs to me that the two-term-collocation case is less problematic in this regard
than the generalized case (whole-field window, all possible combinations).  I had a vague
implementation conception of incrementing a fixed-width integer to iterate over the combinations,
using the integer's bits to include/exclude input terms in the output "termset" tokens.  Using
a 32-bit integer to track combinations would limit the length of an input token stream to
32 tokens, but in the generalized case of all combinations, I'm pretty sure that the number
of bits available would not be the limiting factor, but rather the number of generated terms.
 I guess the question is how to handle cases that produce fewer terms than all combinations
of terms from an input token stream, e.g. the two-term-collocation case, without imposing
the restrictions necessary in the generalized case.

Here are a couple of recent information retrieval papers using "termset" to mean "indexed
token containing multiple input terms":

"TSS: Efficient Term Set Search in Large Peer-to-Peer Textual Collections"

"Termset-based Indexing and Query Processing in P2P Search"

(Sorry, I couldn't find a free public location for the second paper.)

> Co-occurrence filter
> --------------------
>                 Key: LUCENE-2749
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Analysis
>    Affects Versions: 3.1, 4.0
>            Reporter: Steven Rowe
>            Priority: Minor
>             Fix For: 4.0
> The co-occurrence filter to be developed here will output sets of tokens that co-occur
within a given window onto a token stream.  
> These token sets can be ordered either lexically (to allow order-independent matching/counting)
or positionally (e.g. sliding windows of positionally ordered co-occurring terms that include
all terms in the window are called n-grams or shingles). 
> The parameters to this filter will be: 
> * window size: this can be a fixed sequence length, sentence/paragraph context (these
will require sentence/paragraph segmentation, which is not in Lucene yet), or over the entire
token stream (full field width)
> * minimum number of co-occurring terms: >= 2
> * maximum number of co-occurring terms: <= window size
> * token set ordering (lexical or positional)
> One use case for co-occurring token sets is as candidates for collocations.

This message is automatically generated by JIRA.
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message