Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Message-Id: <1432928887.443455.281744505.77A1DD1F@webmail.messagingengine.com>
From: Upayavira <uv@odoko.co.uk>
To: solr-user@lucene.apache.org
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"
In-Reply-To: 
 <CAJ9V1mbcAHqqAccJB2o1d4rC+gA4DV1d1X6Q-WKB4r5bY_E9Zg@mail.gmail.com>
References: 
 <CAJ9V1mauQRgVjx9v1qCfgV9+2P76v73uwss7TdL7nFW_y7w9cg@mail.gmail.com>
 <1432643558.2561023.278264121.51A2AF14@webmail.messagingengine.com>
 <CAB-fSbx_2H=nEwphLF=CgQ4xe3a3VA-Q0G+DPsjSTmSqwN0UUA@mail.gmail.com>
 <CAJ9V1maWdM4N3AE05q5F-FoCLLQiMgGMKg7Af_3unE71NznX+A@mail.gmail.com>
 <CAB-fSbyN7t4b2R3aBHmrO8Oz9+JeYCQjQfTrYNNZJGjdvQ5YeA@mail.gmail.com>
 <CAJ9V1mYx7t2iN7o7f1d0SnOKdAFRAjDxLwQX_N-FuzDY0F6T8w@mail.gmail.com>
 <CAB-fSbwbxME=jeO2iHut9V3VyLsR7J0ST0iHjz3NYG9su8-BQw@mail.gmail.com>
 <CAJ9V1mYfcLyWwAWggYX1_RJk1ro3RKE+4yWgRTqrpbZTgvDKeQ@mail.gmail.com>
 <D51A4FE3F889C3418EF43E229145B57C0CAD2FC1@chapdt3chm01.ad.tiaa-cref.org>
 <CAJ9V1marGZgciLOkYh708oL8HhNBbcDjTtbFiqbGw+T6cCPSQw@mail.gmail.com>
 <D51A4FE3F889C3418EF43E229145B57C0CAD304B@chapdt3chm01.ad.tiaa-cref.org>
 <D51A4FE3F889C3418EF43E229145B57C0CAD30AA@chapdt3chm01.ad.tiaa-cref.org>
 <CAJ9V1mZVsdS=rUG5_TwczsmOM0OcUJ0uyXHHtvG+t_NTqJWutg@mail.gmail.com>
 <D51A4FE3F889C3418EF43E229145B57C0CAD4D8B@chapdt3chm01.ad.tiaa-cref.org>
 <CAJ9V1maFKN86fnSRdNSwDF3zsbODbSgyTrQZ_w5WKdFK772VuQ@mail.gmail.com>
 <CAB-fSbz=TUgp0E8TyF1T0R7D3PGHQF_epnT9yzcauTSnaSSKdg@mail.gmail.com>
 <CAN4YXvc2Q=UoOATxM1z6HvTJY3ShhZ81yv9zghrm5YboCOp9Ag@mail.gmail.com>
 <CAJ9V1mbcAHqqAccJB2o1d4rC+gA4DV1d1X6Q-WKB4r5bY_E9Zg@mail.gmail.com>
Subject: Re: docValues: Can we apply synonym
Date: Fri, 29 May 2015 20:48:07 +0100

Use copyField to clone the field for faceting purposes.

Upayavira

On Fri, May 29, 2015, at 08:06 PM, Aman Tandon wrote:
> Hi Erick,
>=20
> Thanks for suggestion, We are this query parser plugin (
> *SynonymExpandingExtendedDismaxQParserPlugin*) to manage multi-word
> synonym. So it does work slower than edismax that's why it is not in
> contrib right? (I am asking this question because we are using for all
> our
> searches to handle 10 multiword ice cube, icecube etc)
>=20
> *Moreover I thought a solution for this docValue problem*
>=20
> I need to make city field as *multivalued* and by this I mean i will add
> the synonym (*mumbai, bombay*) as an extra value to that field if
> present.
> Now searching operation will work fine as before.
>=20
> >
> > *<field name=3D"city">mumbai</field><field name=3D"city">bombay</field>*
>=20
>=20
> The only prob is if we have to remove the 'city alias/synonym facets'
> when
> we are providing results to the clients.
>=20
> *mumbai, 1000*
>=20
>=20
> With Regards
> Aman Tandon
>=20
> On Fri, May 29, 2015 at 7:26 PM, Erick Erickson <erickerickson@gmail.com>
> wrote:
>=20
> > Do take time for performance testing with that parser. It can be slow
> > depending on your
> > data as I remember. That said it solves the problem it set out to
> > solve so if it meets
> > your SLAs, it can be a life-saver.
> >
> > Best,
> > Erick
> >
> >
> > On Fri, May 29, 2015 at 2:35 AM, Alessandro Benedetti
> > <benedetti.alex85@gmail.com> wrote:
> > > Even if a little bit outdated, that query parser is really really coo=
l to
> > > manage synonyms !
> > > +1 !
> > >
> > > 2015-05-29 1:01 GMT+01:00 Aman Tandon <amantandon.10@gmail.com>:
> > >
> > >> Thanks chris.
> > >>
> > >> Yes we are using it for handling multiword synonym problem.
> > >>
> > >> With Regards
> > >> Aman Tandon
> > >>
> > >> On Fri, May 29, 2015 at 12:38 AM, Reitzel, Charles <
> > >> Charles.Reitzel@tiaa-cref.org> wrote:
> > >>
> > >> > Again, I would recommend using Nolan Lawson's
> > >> > SynonymExpandingExtendedDismaxQParserPlugin.
> > >> >
> > >> > http://nolanlawson.com/2012/10/31/better-synonym-handling-in-solr/
> > >> >
> > >> > -----Original Message-----
> > >> > From: Aman Tandon [mailto:amantandon.10@gmail.com]
> > >> > Sent: Wednesday, May 27, 2015 6:42 PM
> > >> > To: solr-user@lucene.apache.org
> > >> > Subject: Re: docValues: Can we apply synonym
> > >> >
> > >> > Ok and what synonym processor you is talking about maybe it could
> > help ?
> > >> >
> > >> > With Regards
> > >> > Aman Tandon
> > >> >
> > >> > On Thu, May 28, 2015 at 4:01 AM, Reitzel, Charles <
> > >> > Charles.Reitzel@tiaa-cref.org> wrote:
> > >> >
> > >> > > Sorry, my bad.   The synonym processor I mention works different=
ly.
> > >> It's
> > >> > > an extension of the EDisMax query processor and doesn't require
> > field
> > >> > > level synonym configs.
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: Reitzel, Charles [mailto:Charles.Reitzel@tiaa-cref.org]
> > >> > > Sent: Wednesday, May 27, 2015 6:12 PM
> > >> > > To: solr-user@lucene.apache.org
> > >> > > Subject: RE: docValues: Can we apply synonym
> > >> > >
> > >> > > But the query analysis isn't on a specific field, it is applied =
to
> > the
> > >> > > query string.
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: Aman Tandon [mailto:amantandon.10@gmail.com]
> > >> > > Sent: Wednesday, May 27, 2015 6:08 PM
> > >> > > To: solr-user@lucene.apache.org
> > >> > > Subject: Re: docValues: Can we apply synonym
> > >> > >
> > >> > > Hi Charles,
> > >> > >
> > >> > > The problem here is that the docValues works only with primitives
> > data
> > >> > > type only like String, int, etc So how could we apply synonym on
> > >> > > primitive data type.
> > >> > >
> > >> > > With Regards
> > >> > > Aman Tandon
> > >> > >
> > >> > > On Thu, May 28, 2015 at 3:19 AM, Reitzel, Charles <
> > >> > > Charles.Reitzel@tiaa-cref.org> wrote:
> > >> > >
> > >> > > > Is there any reason you cannot apply the synonyms at query tim=
e?
> > >> > > >  Applying synonyms at indexing time has problems, e.g. polluti=
ng
> > the
> > >> > > > term frequency for synonyms added, preventing distance queries,
> > ...
> > >> > > >
> > >> > > > Since city names often have multiple terms, e.g. New York, Den
> > >> > > > Hague, etc., I would recommend using Nolan Lawson's
> > >> > > > SynonymExpandingExtendedDismaxQParserPlugin.   Tastes great, l=
ess
> > >> > > filling.
> > >> > > >
> > >> > > >
> > http://nolanlawson.com/2012/10/31/better-synonym-handling-in-solr/
> > >> > > >
> > >> > > > We found this to fix synonyms like "ny" for "New York" and vice
> > >> versa.
> > >> > > > Haven't tried it with docValues, tho.
> > >> > > >
> > >> > > > -----Original Message-----
> > >> > > > From: Aman Tandon [mailto:amantandon.10@gmail.com]
> > >> > > > Sent: Tuesday, May 26, 2015 11:15 PM
> > >> > > > To: solr-user@lucene.apache.org
> > >> > > > Subject: Re: docValues: Can we apply synonym
> > >> > > >
> > >> > > > Yes it could be :)
> > >> > > >
> > >> > > > Anyway thanks for helping.
> > >> > > >
> > >> > > > With Regards
> > >> > > > Aman Tandon
> > >> > > >
> > >> > > > On Tue, May 26, 2015 at 10:22 PM, Alessandro Benedetti <
> > >> > > > benedetti.alex85@gmail.com> wrote:
> > >> > > >
> > >> > > > > I should investigate that, as usually synonyms are analysis
> > stage.
> > >> > > > > A simple way is to replace the word with all its synonyms (
> > >> > > > > including original word), but simply using this kind of
> > processor
> > >> > > > > will change the token position and offsets, modifying the ac=
tual
> > >> > > > > content of the
> > >> > > > document .
> > >> > > > >
> > >> > > > > " I am from Bombay" will become " I am from Bombay Mumbai" w=
hich
> > >> > > > > can be annoying.
> > >> > > > > So a clever approach must be investigated.
> > >> > > > >
> > >> > > > > 2015-05-26 17:36 GMT+01:00 Aman Tandon <amantandon.10@gmail.=
com
> > >:
> > >> > > > >
> > >> > > > > > Okay So how could I do it with UpdateProcessors?
> > >> > > > > >
> > >> > > > > > With Regards
> > >> > > > > > Aman Tandon
> > >> > > > > >
> > >> > > > > > On Tue, May 26, 2015 at 10:00 PM, Alessandro Benedetti <
> > >> > > > > > benedetti.alex85@gmail.com> wrote:
> > >> > > > > >
> > >> > > > > > > mmm this is different !
> > >> > > > > > > Without any customisation, right now you could :
> > >> > > > > > > - use docValues to provide exact value facets.
> > >> > > > > > > - Than you can use a copy field, with the proper analysi=
s,
> > to
> > >> > > > > > > search
> > >> > > > > > when a
> > >> > > > > > > user click on a filter !
> > >> > > > > > >
> > >> > > > > > > So you will see in your facets :
> > >> > > > > > > Mumbai(3)
> > >> > > > > > > Bombay(2)
> > >> > > > > > >
> > >> > > > > > > And when clicking you see 5 results.
> > >> > > > > > > A little bit misleading for the users =E2=80=A6
> > >> > > > > > >
> > >> > > > > > > On the other hand if you you want to apply the synonyms
> > >> > > > > > > before, the indexing pipeline ( because docValues field =
can
> > >> > > > > > > not be analysed), I
> > >> > > > > think
> > >> > > > > > > you should play with UpdateProcessors.
> > >> > > > > > >
> > >> > > > > > > Cheers
> > >> > > > > > >
> > >> > > > > > > 2015-05-26 17:18 GMT+01:00 Aman Tandon <
> > >> amantandon.10@gmail.com
> > >> > >:
> > >> > > > > > >
> > >> > > > > > > > We are interested in using docValues for better memory
> > >> > > > > > > > utilization
> > >> > > > > and
> > >> > > > > > > > speed.
> > >> > > > > > > >
> > >> > > > > > > > Currently we are faceting the search results on *city.=
 *In
> > >> > > > > > > > city we
> > >> > > > > have
> > >> > > > > > > > also added the synonym for cities like mumbai, bombay
> > (These
> > >> > > > > > > > are
> > >> > > > > Indian
> > >> > > > > > > > cities). So that result of mumbai is also eligible when
> > >> > > > > > > > somebody will applying filter of bombay on search resu=
lts.
> > >> > > > > > > >
> > >> > > > > > > > I need this functionality to apply with docValues enab=
led
> > >> > field.
> > >> > > > > > > >
> > >> > > > > > > > With Regards
> > >> > > > > > > > Aman Tandon
> > >> > > > > > > >
> > >> > > > > > > > On Tue, May 26, 2015 at 9:19 PM, Alessandro Benedetti <
> > >> > > > > > > > benedetti.alex85@gmail.com> wrote:
> > >> > > > > > > >
> > >> > > > > > > > > I checked in the Documentation to be sure, but
> > apparently :
> > >> > > > > > > > >
> > >> > > > > > > > > DocValues are only available for specific field type=
s.
> > The
> > >> > > > > > > > > types
> > >> > > > > > chosen
> > >> > > > > > > > > determine the underlying Lucene docValue type that w=
ill
> > be
> > >> > > used.
> > >> > > > > The
> > >> > > > > > > > > available Solr field types are:
> > >> > > > > > > > >
> > >> > > > > > > > >    - StrField and UUIDField.
> > >> > > > > > > > >    - If the field is single-valued (i.e., multi-valu=
ed
> > is
> > >> > > > > > > > > false),
> > >> > > > > > > Lucene
> > >> > > > > > > > >       will use the SORTED type.
> > >> > > > > > > > >       - If the field is multi-valued, Lucene will use
> > the
> > >> > > > > SORTED_SET
> > >> > > > > > > > type.
> > >> > > > > > > > >    - Any Trie* numeric fields and EnumField.
> > >> > > > > > > > >    - If the field is single-valued (i.e., multi-valu=
ed
> > is
> > >> > > > > > > > > false),
> > >> > > > > > > Lucene
> > >> > > > > > > > >       will use the NUMERIC type.
> > >> > > > > > > > >       - If the field is multi-valued, Lucene will use
> > the
> > >> > > > > SORTED_SET
> > >> > > > > > > > type.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > This means you should not analyse a field where
> > DocValues
> > >> > > > > > > > > is
> > >> > > > > enabled.
> > >> > > > > > > > > Can your explain us your use case ? Why are you
> > interested
> > >> > > > > > > > > in
> > >> > > > > > synonyms
> > >> > > > > > > > > DocValues level ?
> > >> > > > > > > > >
> > >> > > > > > > > > Cheers
> > >> > > > > > > > >
> > >> > > > > > > > > 2015-05-26 13:32 GMT+01:00 Upayavira <uv@odoko.co.uk=
>:
> > >> > > > > > > > >
> > >> > > > > > > > > > To my understanding, docValues are just an uninver=
ted
> > >> > index.
> > >> > > > > > > > > > That
> > >> > > > > > is,
> > >> > > > > > > > it
> > >> > > > > > > > > > contains the terms that are generated at the end o=
f an
> > >> > > > > > > > > > analysis
> > >> > > > > > > chain.
> > >> > > > > > > > > > Therefore, you simply enable docValues and include=
 the
> > >> > > > > > > > > > SynonymFilterFactory in your analysis.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Is that enough, or are you struggling with some ot=
her
> > >> > issue?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Upayavira
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Tue, May 26, 2015, at 12:03 PM, Aman Tandon wro=
te:
> > >> > > > > > > > > > > Hi,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > We have some field *city* in which the docValues=
 are
> > >> > > enabled.
> > >> > > > > We
> > >> > > > > > > need
> > >> > > > > > > > > to
> > >> > > > > > > > > > > add the synonym in that field so how could we do=
 it?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > With Regards
> > >> > > > > > > > > > > Aman Tandon
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > --------------------------
> > >> > > > > > > > >
> > >> > > > > > > > > Benedetti Alessandro
> > >> > > > > > > > > Visiting card : http://about.me/alessandro_benedetti
> > >> > > > > > > > >
> > >> > > > > > > > > "Tyger, tyger burning bright In the forests of the
> > night,
> > >> > > > > > > > > What immortal hand or eye Could frame thy fearful
> > >> symmetry?"
> > >> > > > > > > > >
> > >> > > > > > > > > William Blake - Songs of Experience -1794 England
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > --------------------------
> > >> > > > > > >
> > >> > > > > > > Benedetti Alessandro
> > >> > > > > > > Visiting card : http://about.me/alessandro_benedetti
> > >> > > > > > >
> > >> > > > > > > "Tyger, tyger burning bright
> > >> > > > > > > In the forests of the night,
> > >> > > > > > > What immortal hand or eye
> > >> > > > > > > Could frame thy fearful symmetry?"
> > >> > > > > > >
> > >> > > > > > > William Blake - Songs of Experience -1794 England
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > --------------------------
> > >> > > > >
> > >> > > > > Benedetti Alessandro
> > >> > > > > Visiting card : http://about.me/alessandro_benedetti
> > >> > > > >
> > >> > > > > "Tyger, tyger burning bright
> > >> > > > > In the forests of the night,
> > >> > > > > What immortal hand or eye
> > >> > > > > Could frame thy fearful symmetry?"
> > >> > > > >
> > >> > > > > William Blake - Songs of Experience -1794 England
> > >> > > > >
> > >> > > >
> > >> > > >
> > ********************************************************************
> > >> > > > **
> > >> > > > *** This e-mail may contain confidential or privileged
> > information.
> > >> > > > If you are not the intended recipient, please notify the sender
> > >> > > > immediately and then delete it.
> > >> > > >
> > >> > > > TIAA-CREF
> > >> > > >
> > ********************************************************************
> > >> > > > **
> > >> > > > ***
> > >> > > >
> > >> > >
> > >> > >
> > **********************************************************************
> > >> > > *** This e-mail may contain confidential or privileged informati=
on.
> > >> > > If you are not the intended recipient, please notify the sender
> > >> > > immediately and then delete it.
> > >> > >
> > >> > > TIAA-CREF
> > >> > >
> > **********************************************************************
> > >> > > ***
> > >> > >
> > >> > >
> > **********************************************************************
> > >> > > *** This e-mail may contain confidential or privileged informati=
on.
> > >> > > If you are not the intended recipient, please notify the sender
> > >> > > immediately and then delete it.
> > >> > >
> > >> > > TIAA-CREF
> > >> > >
> > **********************************************************************
> > >> > > ***
> > >> > >
> > >> >
> > >> >
> > ***********************************************************************=
**
> > >> > This e-mail may contain confidential or privileged information.
> > >> > If you are not the intended recipient, please notify the sender
> > >> > immediately and then delete it.
> > >> >
> > >> > TIAA-CREF
> > >> >
> > ***********************************************************************=
**
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > --------------------------
> > >
> > > Benedetti Alessandro
> > > Visiting card : http://about.me/alessandro_benedetti
> > >
> > > "Tyger, tyger burning bright
> > > In the forests of the night,
> > > What immortal hand or eye
> > > Could frame thy fearful symmetry?"
> > >
> > > William Blake - Songs of Experience -1794 England
> >