lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <mkhlud...@griddynamics.com>
Subject Re: faceting is unusable slow since upgrade to 5.3.0
Date Wed, 23 Sep 2015 09:48:15 GMT
Despite docvalues provide NRT faceting with a great performance (since
5.4), enum method is still really important for edge cases (many docs,
small num of terms).
Also, Solr's UnIvertedField had a really smart BigTerms strategy, when
fattest terms were counted by enum and remaining ones with fc. DocValues
needs to evolve much until it's capable to provide such level of
flexibility. I might confuse something again, but I a little bit worry
about existence BigTerms/UnIvertedField in recent versions.

On Wed, Sep 23, 2015 at 12:44 PM, Alessandro Benedetti <
benedetti.alex85@gmail.com> wrote:

> Well done Mikhail,
> curious to see the performance!
>
> Apart the disk usage ( of course building docValues will cost more space),
> taking in consideration the Field cardinality, in the past when the field
> cardinality was low ( few unique values in the field), the enum approach
> was suggested ( so DocValues were non necessary strictly talking about
> faceting).
>
> After this improvement, in your opinion, what is the current status ? after
> the 5.4 release, will enum faceting approach make sense anymore ?
> Has anyone benchmarked this ?
>
> Cheers
>
> 2015-09-23 9:02 GMT+01:00 Mikhail Khludnev <mkhludnev@griddynamics.com>:
>
> > Uwe,
> >
> > I'm sorry for confusion https://issues.apache.org/jira/browse/SOLR-7730
> > goes in 5.4 only. Hence, to get fast DV facets you need to apply patch
> > (it's pretty small).
> > Accelerating non-DV facets is not so clear so far. Please show profiler
> > snapshot for non-DV facets if you wish to go this way.
> >
> >
> > On Tue, Sep 22, 2015 at 5:36 PM, Uwe Reh <reh@hebis.uni-frankfurt.de>
> > wrote:
> >
> > > The exact version as shown by the UI is:
> > > - solr-impl   5.3.0 1696229 - noble - 2015-08-17 17:10:43
> > > - lucene-impl 5.3.0 1696229 - noble - 2015-08-17 16:59:03
> > >
> > > Unfortunately my skills in debugging are limited. So I'm not sure
> about a
> > > 'deeper caller stack'.
> > > Did you mean the attached snapshot from VirtualVM, a stack trace like
> > > below or something else? Please give me a hint.
> > >
> > > uwe
> > >
> > > "qtp1734853116-68" #68 prio=5 os_prio=64 tid=0x00000000117fd800
> nid=0x77
> > >> runnable [0xfffffd7f991fc000]
> > >>    java.lang.Thread.State: RUNNABLE
> > >>         at java.util.HashMap.resize(HashMap.java:734)
> > >>         at java.util.HashMap.putVal(HashMap.java:662)
> > >>         at java.util.HashMap.put(HashMap.java:611)
> > >>         at
> > >>
> >
> org.apache.lucene.index.FieldInfos$Builder.addOrUpdateInternal(FieldInfos.java:344)
> > >>         at
> > >> org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:366)
> > >>         at
> > >> org.apache.lucene.index.FieldInfos$Builder.add(FieldInfos.java:304)
> > >>         at
> > >>
> >
> org.apache.lucene.index.MultiFields.getMergedFieldInfos(MultiFields.java:245)
> > >>         at
> > >>
> >
> org.apache.lucene.index.SlowCompositeReaderWrapper.getFieldInfos(SlowCompositeReaderWrapper.java:237)
> > >>         at
> > >>
> >
> org.apache.lucene.index.SlowCompositeReaderWrapper.getSortedSetDocValues(SlowCompositeReaderWrapper.java:174)
> > >>         at
> > >>
> >
> org.apache.solr.request.DocValuesFacets.getCounts(DocValuesFacets.java:72)
> > >>         at
> > >>
> > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:492)
> > >>         at
> > >>
> > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:385)
> > >>         at
> > >> org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:628)
> > >>         at
> > >> org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:619)
> > >>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > >>         at
> > >> org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:573)
> > >>         at
> > >>
> >
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:644)
> > >>         at
> > >>
> >
> org.apache.solr.handler.component.FacetComponent.getFacetCounts(FacetComponent.java:294)
> > >>         at
> > >>
> >
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:256)
> > >>         at
> > >>
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:285)
> > >>         at
> > >>
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> > >>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2068)
> > >>         at
> > >> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:669)
> > >>         at
> > >> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:462)
> > >>         at
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:210)
> > >>         at
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> > >>         at
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> > >>         at
> > >>
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > >>         at
> > >>
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> > >>         at
> > >>
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> > >>         at org.eclipse.jetty.server.Server.handle(Server.java:499)
> > >>         at
> > >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> > >>         at
> > >>
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> > >>         at org.eclipse.jetty.io
> > >> .AbstractConnection$2.run(AbstractConnection.java:540)
> > >>         at
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> > >>         at
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> > >>         at java.lang.Thread.run(Thread.java:745)
> > >>
> > >
> > >
> > >
> > > Am 22.09.2015 um 12:56 schrieb Mikhail Khludnev:
> > >
> > >> It's quite strange
> > >> https://issues.apache.org/jira/browse/SOLR-7730 significantly
> optimized
> > >> DV
> > >> facets at 5.3.0 exactly by avoiding FileInfo merge.
> > >> Would you mind to provide deeper caller stack for
> > >>
> org.apache.lucene.index.FileInfos.MultibleFields.getMergedFieldInfos()?
> > >> Or a time spend in SlowCompositeReaderWrapper, DocValuesFacets,
> > >> MultiDocValues and their hot methods.
> > >> Which version you exactly on? and how do you know that?
> > >> Thanks
> > >>
> > >>
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mkhludnev@griddynamics.com>
> >
>
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card - http://about.me/alessandro_benedetti
> Blog - http://alexbenedetti.blogspot.co.uk
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mkhludnev@griddynamics.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message