lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Hall <mh...@informatics.jax.org>
Subject Re: Is there a way for me to handle a multiword synonym correctly?
Date Fri, 07 Aug 2009 14:42:44 GMT
Create a field that is specifically for this type of matches.

What you could then do is at indexing time manipulate your data in such 
a way that it can be matched in a punctuation irrelevant way.

So in this field you would convert all non letter characters into 
spaces, and reduce all white space instances to single ones ("     " 
becomes " ") , you could also likely lowercase it at the same time.

Then at search time perform a special search against this field that 
does the same thing to the query string.  At this point plain old phrase 
queries should work for you.

Our corpus contains remarkably obnoxious items in it like: Rara<^tm3.1Ipc>

So we need to be able to do very similar things as you are describing, 
the above mentioned technique worked like a charm.

Matt

Donna L Gresh wrote:
> I saw some discussion on the board but I'm not sure I've got quite the 
> same problem. As an example, I have a query that might be a technical 
> skill:
>
> SAP EM FIN AM
>
> I would like that to match a document that has *either* SAP.EM.FIN.AM or 
> "SAP EM FIN AM" (in that order and all together, not spread out through 
> the document).
>
> The approach I had tried was at index time if I saw SAP.EM.FIN.AM I would 
> consider "SAP EM FIN AM" a synonym for it, using the Lucene in Action 
> example. Luke shows me that I have two terms in the index for this 
> document: SAP.EM.FIN.AM and "SAP EM FIN AM" (one term). Thus it appears 
> differently in the index than if it had been organically found as just the 
> string of tokens, in which case there would be separate terms for SAP, EM, 
> and so on. 
>
> At query time if I look for "SAP EM FIN AM" it is formed as a phrase query 
> with a slop of 0 which does *not* match the one term version "SAP EM FIN 
> AM". (For that matter a simple boolean query doesn't find it either) Luke 
> confirms the fact that the phrase query does not find my synonym term. The 
> query "SAP EM FIN AM" finds *only* documents that originally had those 
> separated tokens in them.
>
> Is there a way to handle this situation such that at index time I can turn 
> SAP.EM.FIN.AM into something that will be found with a query for "SAP EM 
> FIN AM"?
>
> Thanks for any pointers
>
> Donna 
>
>   


-- 
Matthew Hall
Software Engineer
Mouse Genome Informatics
mhall@informatics.jax.org
(207) 288-6012



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message