lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sushant Vengurlekar <svengurle...@curvolabs.com>
Subject Re: How to split index more than 2GB in size
Date Wed, 20 Jun 2018 17:58:16 GMT
Thank you for the detailed response Eric. Very much appreciated. The reason
I am looking into splitting the index into two is because it’s much faster
to search across a smaller index than a larger one.

On Wed, Jun 20, 2018 at 10:46 AM Erick Erickson <erickerickson@gmail.com>
wrote:

> You still haven't answered _why_ you think splitting even a 20G index
> is desirable. We regularly see 200G+ indexes per replica in the field,
> so what's the point? Have you measured different setups to see if it's
> a good idea? A 200G index needs some beefy hardware admittedly....
>
> If you have adequate response times with a 20G index and need to
> increase the QPS rate, just add more replicas. Having more than one
> shard inevitably adds overhead which may (or may not) be made up for
> by parallelizing some of the work. It's nearly always better to use
> only one shard if it meets your response time requirements.
>
> Best,
> Erick
>
> On Wed, Jun 20, 2018 at 10:39 AM, Sushant Vengurlekar
> <svengurlekar@curvolabs.com> wrote:
> > The index size is small because this is my local development copy.  The
> > production index is more than 20GB. So I am working on getting the index
> > split and replicated on different nodes. Our current instance on prod is
> > single instance solr 6 which we are working on moving towards solrcloud 7
> >
> > On Wed, Jun 20, 2018 at 10:30 AM Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> Use the indexupgrader tool or optimize your index before using
> splitshard.
> >>
> >> Since this is a small index (< 5G), optimizing will not create an
> >> overly-large segment, so that pitfall is avoided.
> >>
> >> You haven't yet explained why you think splitting the index would be
> >> beneficial. Splitting an index this small is unlikely to improve query
> >> performance appreciably. This feels a lot like an "XY" problem, you're
> >> asking how to do X thinking it will solve Y but not telling us what Y
> >> is.
> >>
> >> Best,
> >> Erick
> >>
> >> On Wed, Jun 20, 2018 at 9:40 AM, Sushant Vengurlekar
> >> <svengurlekar@curvolabs.com> wrote:
> >> > How can I resolve this error?
> >> >
> >> > On Wed, Jun 20, 2018 at 9:11 AM, Alexandre Rafalovitch <
> >> arafalov@gmail.com>
> >> > wrote:
> >> >
> >> >> This seems more related to an old index upgraded to latest Solr
> rather
> >> than
> >> >> the split itself.
> >> >>
> >> >> Regards,
> >> >>     Alex
> >> >>
> >> >> On Wed, Jun 20, 2018, 12:07 PM Sushant Vengurlekar, <
> >> >> svengurlekar@curvolabs.com> wrote:
> >> >>
> >> >> > Thanks for the reply Alessandro! Appreciate it.
> >> >> >
> >> >> > Below is the full request and the error received
> >> >> >
> >> >> > curl '
> >> >> >
> >> >> > http://localhost:8081/solr/admin/collections?action=
> >> >> SPLITSHARD&collection=dev-transactions&shard=shard1
> >> >> > '
> >> >> >
> >> >> > {
> >> >> >
> >> >> >   "responseHeader":{
> >> >> >
> >> >> >     "status":500,
> >> >> >
> >> >> >     "QTime":7920},
> >> >> >
> >> >> >   "success":{
> >> >> >
> >> >> >     "solr-1:8081_solr":{
> >> >> >
> >> >> >       "responseHeader":{
> >> >> >
> >> >> >         "status":0,
> >> >> >
> >> >> >         "QTime":1190},
> >> >> >
> >> >> >       "core":"dev-transactions_shard1_0_replica_n3"},
> >> >> >
> >> >> >     "solr-1:8081_solr":{
> >> >> >
> >> >> >       "responseHeader":{
> >> >> >
> >> >> >         "status":0,
> >> >> >
> >> >> >         "QTime":1047},
> >> >> >
> >> >> >       "core":"dev-transactions_shard1_1_replica_n4"},
> >> >> >
> >> >> >     "solr-1:8081_solr":{
> >> >> >
> >> >> >       "responseHeader":{
> >> >> >
> >> >> >         "status":0,
> >> >> >
> >> >> >         "QTime":6}},
> >> >> >
> >> >> >     "solr-1:8081_solr":{
> >> >> >
> >> >> >       "responseHeader":{
> >> >> >
> >> >> >         "status":0,
> >> >> >
> >> >> >         "QTime":1009}}},
> >> >> >
> >> >> >   "failure":{
> >> >> >
> >> >> >
> >> >> >
> "solr-1:8081_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$
> >> >> RemoteSolrException:Error
> >> >> > from server at http://solr-1:8081/solr:
> >> >> > java.lang.IllegalArgumentException:
> >> >> > Cannot merge a segment that has been created with major version
6
> into
> >> >> this
> >> >> > index which has been created by major version 7"},
> >> >> >
> >> >> >   "Operation splitshard caused
> >> >> >
> >> >> > exception:":"org.apache.solr.common.SolrException:org.
> >> >> apache.solr.common.SolrException:
> >> >> > SPLITSHARD failed to invoke SPLIT core admin command",
> >> >> >
> >> >> >   "exception":{
> >> >> >
> >> >> >     "msg":"SPLITSHARD failed to invoke SPLIT core admin command",
> >> >> >
> >> >> >     "rspCode":500},
> >> >> >
> >> >> >   "error":{
> >> >> >
> >> >> >     "metadata":[
> >> >> >
> >> >> >       "error-class","org.apache.solr.common.SolrException",
> >> >> >
> >> >> >       "root-error-class","org.apache.solr.common.SolrException"],
> >> >> >
> >> >> >     "msg":"SPLITSHARD failed to invoke SPLIT core admin command",
> >> >> >
> >> >> >     "trace":"org.apache.solr.common.SolrException: SPLITSHARD
> failed
> >> to
> >> >> > invoke SPLIT core admin command\n\tat
> >> >> >
> >> >> > org.apache.solr.client.solrj.SolrResponse.getException(
> >> >> SolrResponse.java:53)\n\tat
> >> >> >
> >> >> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(
> >> >> CollectionsHandler.java:258)\n\tat
> >> >> >
> >> >> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(
> >> >> CollectionsHandler.java:230)\n\tat
> >> >> >
> >> >> > org.apache.solr.handler.RequestHandlerBase.handleRequest(
> >> >> RequestHandlerBase.java:195)\n\tat
> >> >> >
> >> >> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(
> >> >> HttpSolrCall.java:736)\n\tat
> >> >> >
> >> >> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
> >> >> HttpSolrCall.java:717)\n\tat
> >> >> >
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:498)\n\tat
> >> >> >
> >> >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> >> >> SolrDispatchFilter.java:384)\n\tat
> >> >> >
> >> >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> >> >> SolrDispatchFilter.java:330)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> >> >> doFilter(ServletHandler.java:1629)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(
> >> >> ServletHandler.java:533)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> >> >> ScopedHandler.java:143)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.security.SecurityHandler.handle(
> >> >> SecurityHandler.java:548)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> >> >> HandlerWrapper.java:132)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ScopedHandler.
> >> >> nextHandle(ScopedHandler.java:190)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.session.SessionHandler.
> >> >> doHandle(SessionHandler.java:1595)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ScopedHandler.
> >> >> nextHandle(ScopedHandler.java:188)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ContextHandler.
> >> >> doHandle(ContextHandler.java:1253)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ScopedHandler.
> >> >> nextScope(ScopedHandler.java:168)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.servlet.ServletHandler.doScope(
> >> >> ServletHandler.java:473)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.session.SessionHandler.
> >> >> doScope(SessionHandler.java:1564)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ScopedHandler.
> >> >> nextScope(ScopedHandler.java:166)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ContextHandler.
> >> >> doScope(ContextHandler.java:1155)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> >> >> ScopedHandler.java:141)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> >> >> ContextHandlerCollection.java:219)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.HandlerCollection.
> >> >> handle(HandlerCollection.java:126)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> >> >> HandlerWrapper.java:132)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> >> >> RewriteHandler.java:335)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> >> >> HandlerWrapper.java:132)\n\tat
> >> >> > org.eclipse.jetty.server.Server.handle(Server.java:530)\n\tat
> >> >> >
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.server.HttpConnection.onFillable(
> >> >> HttpConnection.java:256)\n\tat
> >> >> > org.eclipse.jetty.io
> >> >> >
> >> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)\
> >> >> n\tat
> >> >> > org.eclipse.jetty.io
> >> .FillInterest.fillable(FillInterest.java:102)\n\tat
> >> >> > org.eclipse.jetty.io
> >> .ChannelEndPoint$2.run(ChannelEndPoint.java:124)\n\
> >> >> tat
> >> >> >
> >> >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.
> >> >> doProduce(EatWhatYouKill.java:247)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.
> >> >> produce(EatWhatYouKill.java:140)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(
> >> >> EatWhatYouKill.java:131)\n\tat
> >> >> >
> >> >> >
> >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(
> >> >> ReservedThreadExecutor.java:382)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> >> >> QueuedThreadPool.java:708)\n\tat
> >> >> >
> >> >> > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> >> >> QueuedThreadPool.java:626)\n\tat
> >> >> > java.lang.Thread.run(Thread.java:748)\n",
> >> >> >
> >> >> >     "code":500}}
> >> >> >
> >> >> >
> >> >> > On Wed, Jun 20, 2018 at 2:25 AM, Alessandro Benedetti <
> >> >> > a.benedetti@sease.io>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi,
> >> >> > > in the first place, why do you want to split 2 Gb indexes
?
> >> >> > > Nowadays is a fairly small index.
> >> >> > >
> >> >> > > Secondly what you reported is incomplete.
> >> >> > > I would expect a Caused By section in the stacktrace.
> >> >> > >
> >> >> > > This are generic recommendations, always spend time in analysing
> the
> >> >> > > problem
> >> >> > > you had scrupulously.
> >> >> > > - SolrCloud problems often involve more than one node. Be
sure to
> >> check
> >> >> > the
> >> >> > > logs of all the nodes possibly involved.
> >> >> > > - Report the full stack trace to the community
> >> >> > > - Report your full request which provoked the exception
> >> >> > >
> >> >> > > Help is much easier this way :)
> >> >> > >
> >> >> > > Regards
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > -----
> >> >> > > ---------------
> >> >> > > Alessandro Benedetti
> >> >> > > Search Consultant, R&D Software Engineer, Director
> >> >> > > Sease Ltd. - www.sease.io
> >> >> > > --
> >> >> > > Sent from:
> >> http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >> >> > >
> >> >> >
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message