lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Newson (JIRA)" <>
Subject [jira] Created: (LUCENE-1438) StandardTokenizer splits host names with hyphens into multiple tokens
Date Tue, 04 Nov 2008 14:18:44 GMT
StandardTokenizer splits host names with hyphens into multiple tokens

                 Key: LUCENE-1438
             Project: Lucene - Java
          Issue Type: Bug
          Components: Analysis
    Affects Versions: 2.4
            Reporter: Robert Newson

StandardTokenizer does not recognize host names with hyphens as a single HOST token. Specifically
"" is tokenized as "www.m" and "", both of "<HOST>" type.

StandardTokenizer should instead output a single HOST token for "", since hyphens
are a legitimate character in DNS host names.

We've a local fix to the grammar file which also required us to significantly simplify the
NUM type to get the behavior we needed for host names.

here's a junit test for the desired behavior;

	public void testWithHyphens() throws Exception {
		final String host = "";
		final StandardTokenizer tokenizer = new StandardTokenizer(
				new StringReader(host));
		final Token token = new Token();;
		assertEquals("<HOST>", token.type());
		assertEquals("", token.term());

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message