Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=lrLISdTcG7sBcwf9FqpMRDwjSmMvp8Dsr3dtHabXclnND1UujeJXvnjYv2ji86hXdWaMGJoYyDdxgTnoy6QxJam8EzmSxG/dzXVUjgq/iBmzM3mh45CqAKbg6CO6eQcjhbumF4zKRnEN9wV5IJpQXf+NQS2SlEb5V0OQm4GoCXU=;
Message-ID: <605791.43324.qm@web52901.mail.re2.yahoo.com>
Date: Fri, 31 Jul 2009 00:26:53 -0700 (PDT)
From: AHMET ARSLAN <iorixxx@yahoo.com>
Subject: Re: Is there a list of "special" characters for standard analyzer?
To: java-user@lucene.apache.org
In-Reply-To: <20090731020249.56MKO.13352.imail@eastrmwml39>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii


> I guess that the obvious question is "Which characters are
> considered 'punctuation characters'?".
 
Punctuation = ("_"|"-"|"/"|"."|",")

> In particular, does the analyzer consider "=" (equal) and
> ":" (colon) to be punctuation characters?

":" is special character at QueryParser (if you are using it). If you want to search it you need to escape it first. At index time this character is ignored. Like the punctuations. The string ahmet:arslan will produce two tokens ahmet and arslan. It also breaks words at "=" character in both query/index time.

If you want to understand the behavior of StandardTokenizer, you need to look at the file StandardTokenizerImpl.jflex. It recognizes the followings as one token: {ALPHANUM}, {APOSTROPHE}, {ACRONYM}, {COMPANY}, {EMAIL} {HOST}, {NUM}, {CJ}, {ACRONYM_DEP} and ignores the rest. There are some definitions of these token types, similar to Regular Expression. You can change behavior of StandardTokenizer by editing this file and generating StandardTokenizerImpl.java from it. There is also another jflex file named WikipediaTokenizerImpl.jflex. By looking it you can understand how new token types can be added. 

Ahmet


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org