Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of weiho@princeton.edu designates
 128.112.128.213 as permitted sender)
Message-ID: <4BD9E616.7030103@princeton.edu>
Date: Thu, 29 Apr 2010 16:03:34 -0400
From: Wei Ho <weiho@princeton.edu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US;
 rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Lucene QueryParser and Analyzer
References: <4BD9E329.5030306@princeton.edu>
 <F8A04958BB57B64096F9BE6B95725643081E3D74@FMD3VS012.fda.gov>
In-Reply-To: <F8A04958BB57B64096F9BE6B95725643081E3D74@FMD3VS012.fda.gov>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

No, there is no whitespace after the comma in Input1

Input1: C1C2,C3C4,C5C6,C7,C8C9C10
Input2: C1C2  C3C4  C5C6  C7  C8C9C10

Input1 is basically one big long word with commas and Chinese characters 
one after the other. Input2 is where I manually separated the string 
into the component terms by replacing the comma with whitespace. My 
confusion stems from the fact that I thought it should not matter since 
the analyzer should be discarding the punctuation anyway? So the 
tokenization process should be the same for both Input1 and Input2? If 
that is not the case, what do I need to change?

Thanks,
Wei Ho

-------- Original Message  --------
Subject: Re: Lucene QueryParser and Analyzer
From: Sudarsan, Sithu D. <Sithu.Sudarsan@fda.hhs.gov>
To: java-user@lucene.apache.org
Date: 4/29/2010 3:54 PM
> Hi,
>
> Is there a whitespace after the comma?
>
>
> Sincerely,
> Sithu D Sudarsan
>
>
> -----Original Message-----
> From: Wei Ho [mailto:weiho@princeton.edu]
> Sent: Thursday, April 29, 2010 3:51 PM
> To: java-user@lucene.apache.org
> Subject: Lucene QueryParser and Analyzer
>
> Hello,
>
> I'm using Lucene to index and search through a collection of Chinese
> documents. However, I'm noticing an odd behavior in query
> parsing/searching.
>
> Given the two queries below:
>
> (Ci refers to Chinese character i)
> Input1: C1C2,C3C4,C5C6,C7,C8C9C10
> Input2: C1C2  C3C4  C5C6  C7  C8C9C10
>
> Input1 returns absolutely nothing, while Input2 (replacing the commas
> with spaces) works as expected. I'm a bit confused why this would be
> happening - it seems that QueryParser uses the Analyzer passed to it to
> tokenize the input query string, so if the Analyzer ignores the
> punctuations, it seems that Input1 and Input2 should return identical
> results. Is there some pre-Analyzer filtering or whatever that
> QueryParser does? I've tried this with the StandardAnalyzer,
> SmartChineseAnalyzer, and an analyzer that I implemented which
> explicitly skips over punctuations and whitespaces in tokenizing the
> query string, but to no avail.
>
> -------sample code-------------
> Analyzer analyzer = new LingPipeAnalyzer();
> Searcher searcher = new IndexSearcher(directory);
> QueryParser qParser = new MultiFieldQueryParser(Version.LUCENE_30,
> SEARCH_FIELDS, analyzer);
> Query query = qParser.parse(queryLine[1]);
> ScoreDoc[] results = searcher.search(query, TOP_N).scoreDocs;
> -----------------------------------
>
> I'm probably just doing something dumb, but any help would be greatly
> appreciated!
>
> Thanks,
> Wei Ho
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org