lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lin Edwin Yeo <edwinye...@gmail.com>
Subject Re: Queries on SynonymFilterFactory
Date Tue, 12 May 2015 02:42:56 GMT
Yes sure, thanks for your advice.

I'm still waiting for my server to come before I can scale up my system and
do the testing. Now the Solr running on my 4GB RAM system will crash if I
try to scale up my system as there's not enough memory to support it.

Regards,
Edwin


On 11 May 2015 at 19:11, Alessandro Benedetti <benedetti.alex85@gmail.com>
wrote:

> 2015-05-11 4:44 GMT+01:00 Zheng Lin Edwin Yeo <edwinyeozl@gmail.com>:
>
> > I've managed to run the synonyms with 10 different synonyms file. Each of
> > the synonym file size is 1MB, which consist of about 1000 tokens, and
> each
> > token has about 40-50 words. These lists of files are more extreme,
> which I
> > probably won't use for the real environment, except now for the testing
> > purpose.
> >
> > The QTime is about 100-200, as compared to about 50 for collection
> without
> > synonyms configured.
> >
> > Is this timing consider fast or slow? Although the synonyms files are
> big,
> > there's not that many index in my collection yet. Just afraid the
> > performance will be affected when more index comes in.
> >
>
> If it's fast or slow it depends on your requirements :)
> For a human waiting for the response, I would say 100ms to be quite fast.
> To understand what happens when the index scale up, you should prototype !
> Anyway there are a lot of solution in Solr to scale up your system !
>
> Cheers
>
> >
> > Regards,
> > Edwin
> >  On 9 May 2015 00:14, "Zheng Lin Edwin Yeo" <edwinyeozl@gmail.com>
> wrote:
> >
> > > Thank you for your suggestions.
> > >
> > > I can't do a proper testing on that yet as I'm currently using a 4GB
> RAM
> > > normal PC machine, and all these probably requires more RAM that what I
> > > have.
> > > I've tried running the setup with 20 synonyms file, and the system went
> > > Out of Memory before I could test anything.
> > >
> > > For your option 2), do you mean that I'll need to download a synonym
> > > database (like the one with over 20MB in size which I have), and index
> > them
> > > into an Ad Hoc Solr Core to manage them?
> > >
> > > I probably can only try them out properly when I can get the server
> > > machine with more RAM.
> > >
> > > Regards,
> > > Edwin
> > >
> > >
> > > On 8 May 2015 at 22:16, Alessandro Benedetti <
> benedetti.alex85@gmail.com
> > >
> > > wrote:
> > >
> > >> This is a quite big Sinonym corpus !
> > >> If it's not feasible to have only 1 big synonym file ( I haven't
> > checked,
> > >> so I assume the 1 Mb limit is true, even if strange)
> > >> I would do an experiment :
> > >> 1) testing query time with a Solr Classic config
> > >> 2) Use an Ad Hoc Solr Core to manage Synonyms ( in this way we can
> keep
> > it
> > >> updated and use it with a custom version of the Sysnonym filter that
> > will
> > >> get the Synonyms directly from another Solr instance).
> > >> 2b) develop a Solr plugin to provide this approach
> > >>
> > >> If the synonym thesaurus is really big, I guess managing them through
> > >> another Solr Core ( or something similar) locally , will be better
> than
> > >> managing it with an external web service.
> > >>
> > >> Cheers
> > >>
> > >> 2015-05-08 12:16 GMT+01:00 Zheng Lin Edwin Yeo <edwinyeozl@gmail.com
> >:
> > >>
> > >> > So it means like having more than 10 or 20 synonym files locally
> will
> > >> still
> > >> > be faster than accessing external service?
> > >> >
> > >> > As I found out that zookeeper only allows the synonym.txt file to
> be a
> > >> > maximum of 1MB, and as my potential synonym file is more than 20MB,
> > I'll
> > >> > need to split the file to more than 20 of them.
> > >> >
> > >> > Regards,
> > >> > Edwin
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> --------------------------
> > >>
> > >> Benedetti Alessandro
> > >> Visiting card : http://about.me/alessandro_benedetti
> > >>
> > >> "Tyger, tyger burning bright
> > >> In the forests of the night,
> > >> What immortal hand or eye
> > >> Could frame thy fearful symmetry?"
> > >>
> > >> William Blake - Songs of Experience -1794 England
> > >>
> > >
> > >
> >
>
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message