lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yann-Erwan Perio <ye.pe...@gmail.com>
Subject Re: Custom Analyzer Not Called When Indexing
Date Sun, 09 Sep 2012 10:24:35 GMT
Hello Uwe,

It works perfectly, thank you very much for your answer!

Cheers,
Yep.


On Sun, Sep 9, 2012 at 12:06 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
> StringField is not tokenized, see http://lucene.apache.org/core/4_0_0-BETA/core/org/apache/lucene/document/StringField.html
>
> You must use TextField!
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Yann-Erwan Perio [mailto:ye.perio@gmail.com]
>> Sent: Sunday, September 09, 2012 11:55 AM
>> To: java-user@lucene.apache.org
>> Subject: Custom Analyzer Not Called When Indexing
>>
>> Hello,
>>
>> This is my first time writing to the list. I am a Java developer, writing a personal
>> project using Lucene, and so far have been very happy with the library
>> (v4BETA). However, I have recently decided to build and use a custom analyzer,
>> and could not make it work with IndexWriter. I must be missing something
>> obvious, but all my searches on the web and my unit tests were to no avail. I
>> would be thankful if you could point me out to the right direction. You will find
>> below some test classes demonstrating my problem.
>>
>> Basically, I have created a custom analyzer, which tokenizes its input to every
>> two chars, then lower cases them. For instance, the string "Hello World" would
>> be tokenized as "he", "ll", and so forth. When I try to index some document with
>> this analyzer, it seems that Lucene does not take it into account at all. To make
>> sure of that, I have simply added a log entry inside my "createComponents"
>> method, to check when this method is called - and the method is never called
>> during the indexing process. Interestingly, when I use a query parser with my
>> custom analyzer, everything is fine - the createComponents method is called as
>> expected. To me, this means that I have somehow failed to properly attach my
>> analyzer to the index writer.
>>
>> The following classes demonstrate my issue:
>> - IndexUtil: has methods to create the analyzer, open a FSDirectory-based
>> directory writer, and the corresponding reader,
>> - TwoLettersAnalyzer: my custom analyzer, which puts together my custom
>> tokenizer with a LowerCase filter,
>> - TwoLettersTokenizer: my custom tokenizer,
>> - TwoLettersTest: the main test class, with the standard "main()" method.
>>
>> The console output generated by the program is as followed:
>>     Calling writeDocs()    <= I expected a call to createComponents()
>> right after this line
>>     Calling checkDocs()
>>     Calling TwoLettersAnalyzer.createComponents()
>>     Calling TwoLettersTokenizer.incrementToken()
>>     Calling TwoLettersTokenizer.incrementToken()
>>     Have we found our doc? false
>>
>>
>> Thank you very much for your time.
>>
>> Kind regards,
>> Yep.
>>
>>
>>
>>
>> ================================================================
>> =
>> IndexUtil.java
>> ================================================================
>> =
>>
>> package experiments;
>>
>> import org.apache.lucene.analysis.Analyzer;
>> import org.apache.lucene.index.DirectoryReader;
>> import org.apache.lucene.index.IndexReader;
>> import org.apache.lucene.index.IndexWriter;
>> import org.apache.lucene.index.IndexWriterConfig;
>> import org.apache.lucene.store.FSDirectory;
>> import org.apache.lucene.util.Version;
>>
>> import java.io.File;
>> import java.io.IOException;
>>
>> public final class IndexUtil {
>>
>>     private IndexUtil() {
>>
>>     }
>>
>>     private static final File indexDirectory = new
>> File("C:\\Users\\Elegie\\Documents\\_Programmation\\Tests\\LuceneWriterTe
>> st\\index");
>>
>>     public static Analyzer createAnalyzer() {
>>         return new TwoLettersAnalyzer();
>>     }
>>
>>     public static IndexWriter openWriter() throws IOException {
>>         IndexWriterConfig config = new
>> IndexWriterConfig(Version.LUCENE_40, createAnalyzer());
>>         config.setOpenMode(IndexWriterConfig.OpenMode.CREATE);
>>         return new IndexWriter(FSDirectory.open(indexDirectory), config);
>>     }
>>
>>     public static IndexReader openReader() throws IOException {
>>         return DirectoryReader.open(FSDirectory.open(indexDirectory));
>>     }
>>
>> }
>>
>>
>> ================================================================
>> =
>> TwoLettersAnalyzer.java
>> ================================================================
>> =
>>
>> package experiments;
>>
>> import org.apache.lucene.analysis.Analyzer;
>> import org.apache.lucene.analysis.TokenStream;
>> import org.apache.lucene.analysis.Tokenizer;
>> import org.apache.lucene.analysis.core.LowerCaseFilter;
>> import org.apache.lucene.util.Version;
>>
>> import java.io.Reader;
>>
>> public class TwoLettersAnalyzer extends Analyzer {
>>
>>     @Override
>>     protected TokenStreamComponents createComponents(String s, Reader
>> reader) {
>>         System.out.println("Calling TwoLettersAnalyzer.createComponents()");
>>         Tokenizer source = new TwoLettersTokenizer(reader);
>>         TokenStream filter = new LowerCaseFilter(Version.LUCENE_40, source);
>>         return new TokenStreamComponents(source, filter);
>>     }
>>
>> }
>>
>>
>> ================================================================
>> =
>> TwoLettersTokenizer.java
>> ================================================================
>> =
>>
>> package experiments;
>>
>> import org.apache.lucene.analysis.Tokenizer;
>> import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
>>
>> import java.io.IOException;
>> import java.io.Reader;
>>
>> public class TwoLettersTokenizer extends Tokenizer {
>>
>>     private static final int TWO_LETTERS = 2;
>>
>>     private CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
>>
>>     protected TwoLettersTokenizer(Reader input) {
>>         super(input);
>>     }
>>
>>     @Override
>>     public final boolean incrementToken() throws IOException {
>>         System.out.println("Calling TwoLettersTokenizer.incrementToken()");
>>         StringBuilder builder = new StringBuilder();
>>         boolean success = readAhead(builder, TWO_LETTERS);
>>         if (success) {
>>             termAtt.setEmpty();
>>             termAtt.append(builder);
>>         }
>>         return success;
>>     }
>>
>>     private boolean readAhead(StringBuilder builder, int ahead) throws
>> IOException {
>>         if (ahead <= 0) {
>>             return true;
>>         }
>>         int data = input.read();
>>         if (data != -1) {
>>             builder.append((char) data);
>>             return readAhead(builder, ahead - 1);
>>         }
>>         return false;
>>     }
>>
>> }
>>
>> ================================================================
>> =
>> TwoLettersTest.java
>> ================================================================
>> =
>>
>> package experiments;
>>
>> import org.apache.lucene.document.Document;
>> import org.apache.lucene.document.Field; import
>> org.apache.lucene.document.StringField;
>> import org.apache.lucene.index.IndexReader;
>> import org.apache.lucene.index.IndexWriter;
>> import org.apache.lucene.queryparser.classic.ParseException;
>> import org.apache.lucene.queryparser.classic.QueryParser;
>> import org.apache.lucene.search.IndexSearcher;
>> import org.apache.lucene.search.Query;
>> import org.apache.lucene.search.TopDocs; import
>> org.apache.lucene.util.Version;
>>
>> import java.io.IOException;
>>
>> public class TwoLettersTest {
>>
>>     private static final String FIELD_NAME = "content";
>>     private static final String TEST_CONTENT = "Hello, World!";
>>     private static final String TEST_SEARCH = "he"; //llo World!
>>     private static final int EXPECTED_MATCHED_DOCS = 1;
>>
>>     public static void main(String[] args) throws IOException, ParseException {
>>         TwoLettersTest test = new TwoLettersTest();
>>
>>         System.out.println("Calling writeDocs()");
>>         test.writeDocs();
>>
>>         System.out.println("Calling checkDocs()");
>>         test.checkDocs();
>>     }
>>
>>     public void writeDocs() throws IOException {
>>         IndexWriter writer = IndexUtil.openWriter();
>>         writer.addDocument(createDoc(TEST_CONTENT));
>>         writer.close();
>>     }
>>
>>     public void checkDocs() throws ParseException, IOException {
>>         Query query = createQuery();
>>         IndexReader reader = IndexUtil.openReader();
>>         IndexSearcher searcher = new IndexSearcher(reader);
>>         TopDocs results = searcher.search(query, EXPECTED_MATCHED_DOCS);
>>         System.out.println("Have we found our doc? " + (results.scoreDocs.length
>> == EXPECTED_MATCHED_DOCS));
>>         reader.close();
>>     }
>>
>>     private Document createDoc(String content) {
>>         Document doc = new Document();
>>         StringField field = new StringField(FIELD_NAME, content, Field.Store.YES);
>>         doc.add(field);
>>         return doc;
>>     }
>>
>>     private Query createQuery() throws ParseException {
>>         String search = FIELD_NAME + ":" + TEST_SEARCH;
>>         QueryParser parser = new QueryParser(Version.LUCENE_40,
>> FIELD_NAME, IndexUtil.createAnalyzer());
>>         Query query = parser.parse(search);
>>         return query;
>>     }
>>
>> }
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message