Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: Ilya Zavorin <izavorin@caci.com>
To: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
Subject: RE: how to fully preprocess query before fuzzy search?
Thread-Topic: how to fully preprocess query before fuzzy search?
Thread-Index: Ac2U4oBxmT+wE7F/Reu0r8MMWj6EuwAI31qAAAf8pbA=
Date: Mon, 17 Sep 2012 15:08:20 +0000
Message-ID: <A57498EDEC10C64781EA0F7DBA665CEF402F584A@ex2010mb01-2.caci.com>
References: <A57498EDEC10C64781EA0F7DBA665CEF402F381B@ex2010mb01-2.caci.com>
 <CBD9C3E598DF4874BF2E9F7F88F1AD49@JackKrupansky>
In-Reply-To: <CBD9C3E598DF4874BF2E9F7F88F1AD49@JackKrupansky>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks so I do not need to escape the "&" in=20

	"dog & cat"

But I do need to escape the "&&" in=20

	"dog && cat"

correct? And do I escape as "dog \&& cat" or as "dog \&\& cat"?


Ilya


-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]=20
Sent: Monday, September 17, 2012 10:55 AM
To: java-user@lucene.apache.org
Subject: Re: how to fully preprocess query before fuzzy search?

"
Lucene supports escaping special characters that are part of the query synt=
ax. The current list special characters are

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /
"

See:
http://lucene.apache.org/core/4_0_0-ALPHA/queryparser/org/apache/lucene/que=
ryparser/classic/package-summary.html

So, maybe you should escape all special characters, and then add the fuzzy =
query. Note: In 4.0 the fuzzy query is limited to an editing distance of 2.

-- Jack Krupansky

-----Original Message-----
From: Ilya Zavorin
Sent: Monday, September 17, 2012 10:41 AM
To: java-user@lucene.apache.org
Subject: how to fully preprocess query before fuzzy search?

I am processing a bunch of text coming out of OCR, i.e. it's machine-genera=
ted text that contains some errors like garbage characters attached to word=
s, letters replaced with similarly looking characters (e.g.=20
"I" with "1") etc. The text is whitespace-tokenized and I am trying to matc=
h each token against an index using a fuzzy match, so that small amounts of=
 occasional garbage in the tokens do not prevent a match.

Right now I am preprocessing each query as follows:

//term =3D token
Query queryF =3D parser.Parse(term.Replace("~", "") + "~");

However, searcher.Search still throws "can't parse" exceptions for queries =
that contain brackets, quotes and other garbage characters.

So how should I fully preprocess a query to avoid these exceptions?

Looks like I just need to remove a certain set of characters just like the =
tilde is removed above. What is the complete set of such characters? Do I n=
eed to do any other preprocess?

Thanks,

Ilya Zavorin


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org