Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: Solr Cloud and Multi-word Synonyms :: synonym_edismax parser
From: Steve Rowe <sarowe@gmail.com>
In-Reply-To: <CAFWP_DrtRq35K5fEmKxFjQd8c+V4YnQSqwPdm5UhiH-S9m8sbA@mail.gmail.com>
Date: Fri, 27 May 2016 12:28:26 -0400
Cc: chris@depahelix.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <C580231D-00CA-4942-9FD4-8E75988FFE12@gmail.com>
References: <CAFWP_Do-fJwTvi9rz2qsZfuNS1WTZBjTbUxf-SV72yqmADvhiA@mail.gmail.com> <CAFWP_DpV6EMzNEUhoLGTEcNvXA1TrbuaiVvCXUqjF2nKtTVM8g@mail.gmail.com> <8795A443-5CCA-4E07-B02E-6A3644104611@whitepages.com> <CAFWP_DoQoxxt=dgfPK4R4YnZGCFWRUMhLBuAFXmDL7VaL_eL1w@mail.gmail.com> <902536a1fed24e379aebd8c096ff35e1@depahelix.com> <CAFWP_DrtRq35K5fEmKxFjQd8c+V4YnQSqwPdm5UhiH-S9m8sbA@mail.gmail.com>
To: solr-user@lucene.apache.org
archived-at: Fri, 27 May 2016 16:28:40 -0000

I=E2=80=99m working on addressing problems using multi-term synonyms at =
query time in Lucene and Solr.

I recommend these two blogs for understanding the issues (the second one =
was mentioned earlier in this thread):

=
<http://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.=
html>
<https://nolanlawson.com/2012/10/31/better-synonym-handling-in-solr/>

In addition to the already-mentioned projects, there is also:

<https://issues.apache.org/jira/browse/SOLR-5379>

All of these projects try in various ways to work around the fact that =
Lucene=E2=80=99s QueryParser splits on whitespace before sending text to =
analysis, one token at a time, so in a synonym filter, multi-word =
synonyms can never match and add alternatives.  See =
<https://issues.apache.org/jira/browse/LUCENE-2605>, where I=E2=80=99ve =
posted a patch to directly address that problem - note that it=E2=80=99s =
still a work in progress.

Once LUCENE-2605 has been fixed, there is still work to do getting =
(e)dismax to work with the modified Lucene QueryParser, and addressing =
problems with how queries are constructed from Lucene=E2=80=99s =
=E2=80=9Csausagized=E2=80=9D token stream.

--
Steve
www.lucidworks.com

> On May 26, 2016, at 2:21 PM, John Bickerstaff =
<john@johnbickerstaff.com> wrote:
>=20
> Thanks Chris --
>=20
> The two projects I'm aware of are:
>=20
> https://github.com/healthonnet/hon-lucene-synonyms
>=20
> and the one referenced from the Lucidworks page here:
> =
https://lucidworks.com/blog/2014/07/12/solution-for-multi-term-synonyms-in=
-lucenesolr-using-the-auto-phrasing-tokenfilter/
>=20
> ... which is here : =
https://github.com/LucidWorks/auto-phrase-tokenfilter
>=20
> Is there anything else out there that you would recommend I look at?
>=20
> On Thu, May 26, 2016 at 12:01 PM, Chris Morley <chris@depahelix.com> =
wrote:
>=20
>> Chris Morley here, from Wayfair.  (Depahelix =3D my domain)
>>=20
>> Suyash Sonawane and I have worked on multiple word synonyms at =
Wayfair.
>> We worked mostly off of Ted Sullivan's work and also off of some
>> suggestions from Koorosh Vakhshoori.  We have gotten to a point where =
we
>> have a more sophisticated internal implementation, however, we've =
found
>> that it is very difficult to make it do what you want it to do, and =
also be
>> sufficiently performant.  Watch out for exceptional situations with =
mm
>> (minimum should match).
>>=20
>> Trey Grainger (now at Lucidworks) and Simon Hughes of Dice.com have =
also
>> done work in this area.
>>=20
>> It should be very possible to get this kind of thing working on
>> SolrCloud.  I haven't tried it yet but I think theoretically, it =
should
>> just work.  The synonyms stuff is mostly about doing things at index =
time
>> and query time.  The index time stuff should translate to SolrCloud
>> directly, while the query time stuff might pose some issues, but =
probably
>> not too bad, if there are any issues at all.
>>=20
>> I've had decent luck porting our various plugins from 4.10.x to 5.5.0
>> because a lot of stuff is just Java, and it still works within the =
Jetty
>> context.
>>=20
>> -Chris.
>>=20
>>=20
>>=20
>>=20
>> ----------------------------------------
>> From: "John Bickerstaff" <john@johnbickerstaff.com>
>> Sent: Thursday, May 26, 2016 1:51 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Cloud and Multi-word Synonyms :: synonym_edismax =
parser
>> Hey Jeff (or anyone interested in multi-word synonyms) here are some
>> potentially interesting links...
>>=20
>> http://wiki.apache.org/solr/QueryParser (search the page for
>> synonum_edismax)
>>=20
>> https://nolanlawson.com/2012/10/31/better-synonym-handling-in-solr/ =
(blog
>> post about what became the synonym_edissmax Query Parser)
>>=20
>>=20
>> =
https://lucidworks.com/blog/2014/07/12/solution-for-multi-term-synonyms-in=
-lucenesolr-using-the-auto-phrasing-tokenfilter/
>>=20
>> This last was useful for lots of reasons and contains links to other
>> interesting, related web pages...
>>=20
>> On Thu, May 26, 2016 at 11:45 AM, Jeff Wartes =
<jwartes@whitepages.com>
>> wrote:
>>=20
>>> Oh, interesting. I've certainty encountered issues with multi-word
>>> synonyms, but I hadn't come across this. If you end up using it with =
a
>>> recent solr verison, I'd be glad to hear your experience.
>>>=20
>>> I haven't used it, but I am aware of one other project in this vein =
that
>>> you might be interested in looking at:
>>> https://github.com/LucidWorks/auto-phrase-tokenfilter
>>>=20
>>>=20
>>> On 5/26/16, 9:29 AM, "John Bickerstaff" <john@johnbickerstaff.com>
>> wrote:
>>>=20
>>>> Ahh - for question #3 I may have spoken too soon. This line from =
the
>>>> github repository readme suggests a way.
>>>>=20
>>>> Update: We have tested to run with the jar in $SOLR_HOME/lib as =
well,
>> and
>>>> it works (Jetty).
>>>>=20
>>>> I'll try that and only respond back if that doesn't work.
>>>>=20
>>>> Questions 1 and 2 still stand of course... If anyone on the list =
has
>>>> experience in this area...
>>>>=20
>>>> Thanks.
>>>>=20
>>>> On Thu, May 26, 2016 at 10:25 AM, John Bickerstaff <
>>> john@johnbickerstaff.com
>>>>> wrote:
>>>>=20
>>>>> Hi all,
>>>>>=20
>>>>> I'm creating a Solr Cloud that will index and search medical text.
>>>>> Multi-word synonyms are a pretty important factor.
>>>>>=20
>>>>> I find that there are some challenges around multi-word synonyms =
and I
>>>>> also found on the wiki that there is a recommended 3rd-party =
parser
>>>>> (synonym_edismax parser) created by Nolan Lawson and found here:
>>>>> https://github.com/healthonnet/hon-lucene-synonyms
>>>>>=20
>>>>> Here's the thing - the instructions on the github site involve
>> bringing
>>>>> the jar file into the war file - which is not applicable any =
more...
>> at
>>>>> least I think it's not...
>>>>>=20
>>>>> I have three questions:
>>>>>=20
>>>>> 1. Is this still a good solution for multi-word synonyms (I.e. =
Solr
>>> Cloud
>>>>> doesn't break it in some way)
>>>>> 2. Is there a tool or plug-in out there that the contributors =
would
>>>>> recommend above this one?
>>>>> 3. Assuming 1 =3D yes and 2 =3D no, can anyone tell me an updated
>> procedure
>>>>> for bringing it in to Solr Cloud (I'm running 5.4.x)
>>>>>=20
>>>>> Thanks
>>>>>=20
>>>=20
>>>=20
>>=20
>>=20
>>=20