lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chad Small" <>
Subject RE: Query syntax on Keyword field question
Date Wed, 24 Mar 2004 05:28:47 GMT
Here is my attempt at a KeywordAnalyzer - although is not working?  Excuse the length of the
message, but wanted to give actual code.
package domain.lucenesearch;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.CharTokenizer;
import org.apache.lucene.analysis.TokenStream;
public class KeywordAnalyzer extends Analyzer
   public TokenStream tokenStream(String s, Reader reader)
      return new KeywordTokenizer(reader);
   private class KeywordTokenizer extends CharTokenizer
      public KeywordTokenizer(Reader in)
       * Collects all characters.
      protected boolean isTokenChar(char c)
         return true;

However, this test: fails
public class KeywordAnalyzerTest extends TestCase
   RAMDirectory directory;
   private IndexSearcher searcher;
   public void setUp() throws Exception
      directory = new RAMDirectory();
      IndexWriter writer = new IndexWriter(directory,
                                           new StandardAnalyzer(),
      Document doc = new Document();
      doc.add(Field.Keyword("category", "HW-NCI_TOPICS"));
      doc.add(Field.Text("description", "Illidium Space Modulator"));
      searcher = new IndexSearcher(directory);
public void testPerFieldAnalyzer() throws Exception
      PerFieldAnalyzerWrapper analyzer = new PerFieldAnalyzerWrapper(new StandardAnalyzer());
      analyzer.addAnalyzer("category", new KeywordAnalyzer());   //|#1
      Query query = QueryParser.parse("category:HW-NCI_TOPICS AND SPACE",
      Hits hits =;
      System.out.println("query.ToString = " + query.toString("description"));
      assertEquals("HW-NCI_TOPICS kept as-is",
                   "category:HW-NCI_TOPICS +space", query.toString("description"));
      assertEquals("doc found!", 1, hits.length());
   private void analyze(String text) throws Exception
      Analyzer[] analyzers = new Analyzer[]{
         new WhitespaceAnalyzer(),
         new SimpleAnalyzer(),
         new StopAnalyzer(),
         new StandardAnalyzer(),
         new KeywordAnalyzer(),
         //new SnowballAnalyzer("English", StopAnalyzer.ENGLISH_STOP_WORDS)
      System.out.println("Analzying \"" + text + "\"");
      for (int i = 0; i < analyzers.length; i++)
         Analyzer analyzer = analyzers[i];
         System.out.println("\t" + analyzer.getClass().getName() + ":");
         TokenStream stream = analyzer.tokenStream("category", new StringReader(text));
         while (true)
            Token token =;
            if (token == null) break;
            System.out.print("[" + token.termText() + "] ");
With this output:
Analzying "HW-NCI_TOPICS"
  [hw] [nci] [topics] 
  [hw] [nci] [topics] 
  [hw] [nci] [topics] 
query.ToString = category:HW -"nci topics" +space

junit.framework.ComparisonFailure: HW-NCI_TOPICS kept as-is 
Expected:+category:HW-NCI_TOPICS +space
Actual  :category:HW -"nci topics" +space
See anything?

	-----Original Message----- 
	From: Chad Small 
	Sent: Tue 3/23/2004 8:48 PM 
	To: Lucene Users List 
	Subject: RE: Query syntax on Keyword field question

	Thanks-you Erik and Incze.  I now understand the issue and I'm trying to create a "KeywordAnalyzer"
as suggested from you book excerpt, Erik:
	However, not being all that familiar with the Analyzer framework, I'm not sure how to implement
the "KeywordAnalyzer" even though it might be "trivial" :)  Any hints, code, or messages to
look at?
	<<from message link above>>
	Ok, here is the section from Lucene in Action.  I'll leave the
	development of KeywordAnalyzer as an exercise for the reader (although
	its implementation is trivial, one of the simplest analyzers possible -
	only emit one token of the entire contents).  I hope this helps.
	thanks again,
	        -----Original Message-----
	        From: Incze Lajos []
	        Sent: Tue 3/23/2004 8:08 PM
	        To: Lucene Users List
	        Subject: Re: Query syntax on Keyword field question
	        On Tue, Mar 23, 2004 at 08:10:15PM -0500, Erik Hatcher wrote:
	        > QueryParser and Field.Keyword fields are a strange mix.  For some
	        > background, check the archives as this has been covered pretty
	        > extensively.
	        > A quick answer is yes you can use MFQP and QP with keyword fields,
	        > however you need to be careful which analyzer you use.
	        > PerFieldAnalyzerWrapper is a good solution - you'll just need to use an
	        > analyzer for your keyword field which simply tokenizes the whole string
	        > as one chunk.  Perhaps such an analyzer should be made part of the
	        > core?
	        >       Erik
	        I've implemented suche an analyzer but it's only partial solution
	        if your keyword field contains spaces, as the QP would split
	        the query, e.g.:
	        NOTTOKNIZED:(term with spaces*)
	        would give you no hit even with an not tokenized field
	        "term with spaces and other useful things". The full solution
	        would be to be able to tell the QP not to split at spaces,
	        either by 'do not split till apos' syntax, or by the good ol'
	        backslash: do\ not\ notice\ these\ spaces.
	        To unsubscribe, e-mail:
	        For additional commands, e-mail:

View raw message