lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Benedetti <benedetti.ale...@gmail.com>
Subject Re: Queries on SynonymFilterFactory
Date Mon, 11 May 2015 11:11:15 GMT
2015-05-11 4:44 GMT+01:00 Zheng Lin Edwin Yeo <edwinyeozl@gmail.com>:

> I've managed to run the synonyms with 10 different synonyms file. Each of
> the synonym file size is 1MB, which consist of about 1000 tokens, and each
> token has about 40-50 words. These lists of files are more extreme, which I
> probably won't use for the real environment, except now for the testing
> purpose.
>
> The QTime is about 100-200, as compared to about 50 for collection without
> synonyms configured.
>
> Is this timing consider fast or slow? Although the synonyms files are big,
> there's not that many index in my collection yet. Just afraid the
> performance will be affected when more index comes in.
>

If it's fast or slow it depends on your requirements :)
For a human waiting for the response, I would say 100ms to be quite fast.
To understand what happens when the index scale up, you should prototype !
Anyway there are a lot of solution in Solr to scale up your system !

Cheers

>
> Regards,
> Edwin
>  On 9 May 2015 00:14, "Zheng Lin Edwin Yeo" <edwinyeozl@gmail.com> wrote:
>
> > Thank you for your suggestions.
> >
> > I can't do a proper testing on that yet as I'm currently using a 4GB RAM
> > normal PC machine, and all these probably requires more RAM that what I
> > have.
> > I've tried running the setup with 20 synonyms file, and the system went
> > Out of Memory before I could test anything.
> >
> > For your option 2), do you mean that I'll need to download a synonym
> > database (like the one with over 20MB in size which I have), and index
> them
> > into an Ad Hoc Solr Core to manage them?
> >
> > I probably can only try them out properly when I can get the server
> > machine with more RAM.
> >
> > Regards,
> > Edwin
> >
> >
> > On 8 May 2015 at 22:16, Alessandro Benedetti <benedetti.alex85@gmail.com
> >
> > wrote:
> >
> >> This is a quite big Sinonym corpus !
> >> If it's not feasible to have only 1 big synonym file ( I haven't
> checked,
> >> so I assume the 1 Mb limit is true, even if strange)
> >> I would do an experiment :
> >> 1) testing query time with a Solr Classic config
> >> 2) Use an Ad Hoc Solr Core to manage Synonyms ( in this way we can keep
> it
> >> updated and use it with a custom version of the Sysnonym filter that
> will
> >> get the Synonyms directly from another Solr instance).
> >> 2b) develop a Solr plugin to provide this approach
> >>
> >> If the synonym thesaurus is really big, I guess managing them through
> >> another Solr Core ( or something similar) locally , will be better than
> >> managing it with an external web service.
> >>
> >> Cheers
> >>
> >> 2015-05-08 12:16 GMT+01:00 Zheng Lin Edwin Yeo <edwinyeozl@gmail.com>:
> >>
> >> > So it means like having more than 10 or 20 synonym files locally will
> >> still
> >> > be faster than accessing external service?
> >> >
> >> > As I found out that zookeeper only allows the synonym.txt file to be a
> >> > maximum of 1MB, and as my potential synonym file is more than 20MB,
> I'll
> >> > need to split the file to more than 20 of them.
> >> >
> >> > Regards,
> >> > Edwin
> >> >
> >>
> >>
> >>
> >> --
> >> --------------------------
> >>
> >> Benedetti Alessandro
> >> Visiting card : http://about.me/alessandro_benedetti
> >>
> >> "Tyger, tyger burning bright
> >> In the forests of the night,
> >> What immortal hand or eye
> >> Could frame thy fearful symmetry?"
> >>
> >> William Blake - Songs of Experience -1794 England
> >>
> >
> >
>



-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message