lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven A Rowe <sar...@syr.edu>
Subject RE: Case where StandardAnalyzer doesn't remove punctuation
Date Fri, 23 Mar 2012 13:31:46 GMT
Hi Colm,

Thanks for bringing the issue up.  The behavior you mention is expected, though - see below
for details.

In the future, you should use the java-user@l.a.o mailing list for questions about Lucene
*usage* -- this mailing list (dev@l.a.o), by contrast, is intended for development discussions.

<rant>
FYI, Nabble.com stripped out all of your code and examples before sending your message to
the mailing list.  My suggestion: stop using Nabble.  (I've described this problem to their
support people a couple of times, and they apparently just don't care, since it still persists,
years later.)
</rant>

StandardTokenizer, the tokenizer included in StandardAnalyzer, implements the Word Break rules
from Unicode 6.0.0 UAX#29 <http://www.unicode.org/reports/tr29/tr29-17.html>.  These
rules are international in nature.

The UAX#29 Word Break rules that prohibit breaking around commas when surrounded by digits
are:

	Do not break within sequences, such as "3.2" or "3,456.789".

	WB11.	Numeric (MidNum | MidNumLet) × Numeric
	WB12.	Numeric × (MidNum | MidNumLet) Numeric

(In these rules, "×" means "do not break".)

>From the table "Word_Break property values" at <http://www.unicode.org/reports/tr29/tr29-17.html#Default_Word_Boundaries>,
you can see that the set of characters that are assigned the Word_Break:MidNum property value
is a superset of the characters assigned the Line_Break:Infix_Numeric property value, which
includes the comma character.  You can see the full set of characters that are assigned the
Line_Break:Infix_Number property value here: <http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3ALine_Break+%3D+Infix_Numeric%3A%5D>
(note that because this utility refers to Unicode 6.1.0, the results may differ slightly from
Lucene v3.5.0 StandardTokenizer, since it uses Unicode 6.0.0).

Steve

-----Original Message-----
From: colm.mchugh [mailto:colm.mchugh@mapflow.com] 
Sent: Thursday, March 22, 2012 9:23 AM
To: dev@lucene.apache.org
Subject: Case where StandardAnalyzer doesn't remove punctuation

I'm using Lucene to search address data, and came across an interesting case where StandardAnalyzer
appears not to remove punctuation (a comma). To illustrate, the following code snippet uses
StandardAnalyzer to analyze an address, printing out each analyzed token. 
 The output of the code snippet is: 
If the code is altered slightly so the String text is initialized as
follows:
 (there's a space between the first comma and the building number) then the output is as follows:I
would expect the output to be the same in both cases based on my understanding. Is this a
known issue? Or am I off on my understanding? It's not a biggie. It caught my attention because
I have a unit test that asserts token text is all lower case or alphanumeric. It can be easily
got around, but I thought it worth posting about.

--
View this message in context: http://lucene.472066.n3.nabble.com/Case-where-StandardAnalyzer-doesn-t-remove-punctuation-tp3848460p3848460.html
Sent from the Lucene - Java Developer mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail:
dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message