Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of appy74@dsl.pipex.com designates
 212.74.114.14 as permitted sender)
From: "Martin O'Shea" <appy74@dsl.pipex.com>
To: <java-user@lucene.apache.org>
Subject: Using a Lucene ShingleFilter to extract frequencies of bigrams in
 Lucene
Date: Tue, 4 Sep 2012 17:37:16 +0100
Message-ID: <006c01cd8abb$8b69b310$a23d1930$@dsl.pipex.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_006D_01CD8AC3.ED2FC8C0"
Thread-Index: Ac2Kuv+EJSYLUrh/Q3GHG/kF5QuNEw==
Content-Language: en-gb

------=_NextPart_000_006D_01CD8AC3.ED2FC8C0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

If a Lucene ShingleFilter can be used to tokenize a string into shingles, or
ngrams, of different sizes, e.g.:

 
    "please divide this sentence into shingles"

 
Becomes:

 
    shingles "please divide", "divide this", "this sentence", "sentence
into", and "into shingles"

 
Does anyone know if this can be used in conjunction with other analyzers to
return the frequencies of the bigrams or trigrams found, e.g.:

 
    "please divide this please divide sentence into shingles"

 
Would return 2 for "please divide"?

 
I'm currently using Lucene 3.0.2 to extract frequencies of unigrams from a
string using a combination of a TermVectorMapper and Standard/Snowball
analyzers.

 
I should add that my strings are built up from a database and then indexed
by Lucene in memory and are not persisted beyond this. Use of other products
like Solr is not intended.

 
Thanks

 
Mr Morgan.

 
------=_NextPart_000_006D_01CD8AC3.ED2FC8C0--