lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From spoonerk <john.spoo...@gmail.com>
Subject Re: The Impact of the Number of Collections on Indexing Performance in Solr 6.0
Date Mon, 12 Mar 2018 12:56:07 GMT
I have tried emailing to.unsubscribe.  I have tried disrupting threads
hoping to anger the admin into getting me out of the spam list.  All I get
is arrogant emails about headers

On Mar 12, 2018 1:15 AM, "苗海泉" <mseaspring@gmail.com> wrote:

> Thanks  Erick and  Shawn , Thank you for your patience. I said that the
> above phenomenon was caused by the IO, cpu, memory, and network io. The
> swap was turned off and the machine's memory was sufficient. When the speed
> of indexing is declining, QTime is found to take 3 seconds to 4 seconds to
> reload the index, so it can be guessed that it is more likely to be a Solr
> problem than a jetty. It is worth mentioning that when the speed of the
> index under construction dropped sharply, the Solr used only about 5% of
> the CPU, and when it was normal, the CPU usage was 200 percent, and the
> overall system's CPU usage was 100 percent. About twenty.
>
> Basic information:
> 1) The data volume of each collection is between 2 billion and 3 billion.
> 2) The configuration of the machine is 24 cpu and 128G memory.
> 3) The disk usage per copy is about 10G.
>
> In addition, I noticed that the work of zookeeper is normal and there is no
> error or warning message.
>
> So all these phenomena make me think that the internal specific mechanism
> of solr may lead to a sharp drop in the index construction speed. At
> present, it seems that our solr's machine resources are sufficient.
>
> As for the reduction of the number of collections that you said, we also
> have this plan, and we are looking for ways to reform it. Are there any
> other suggestions?
>
>
> Best .
> miaohq
>
> 2018-03-11 10:15 GMT+08:00 spoonerk <john.spooner@gmail.com>:
>
> > Wow thanks.  Just trying to unsubscribe.  Most email lists let u do that
> >
> > On Mar 10, 2018 2:36 PM, "Erick Erickson" <erickerickson@gmail.com>
> wrote:
> >
> > > Spoonerk:
> > >
> > > You say you've tried "many times", but you haven't provided  full
> > > header as described in the "problems" link at the link below. You
> > > haven't e-mailed the list owner as suggested in the "problems" link.
> > > You haven't, in short, provided any of the information that's
> > > necessary to actually unsubscribe you.
> > >
> > > Please follow the instructions here:
> > > http://lucene.apache.org/solr/community.html#mailing-lists-irc. In
> > > particular look at the "problems" link.
> > >
> > > You must use the _exact_ same e-mail as you used to subscribe.
> > >
> > > If the initial try doesn't work and following the suggestions at the
> > > "problems" link doesn't work for you, let us know. But note you need
> > > to show us the _entire_ return header to allow anyone to diagnose the
> > > problem.
> > >
> > > Best,
> > > Erick
> > >
> > > On Sat, Mar 10, 2018 at 1:03 PM, spoonerk <john.spooner@gmail.com>
> > wrote:
> > > > I have manually unsubscribed many times.  But I still get emails from
> > the
> > > > list.  Can some admin please unsubscribe me?
> > > >
> > > > On Mar 9, 2018 9:52 PM, "苗海泉" <mseaspring@gmail.com> wrote:
> > > >
> > > >> hello,We found a problem. In solr 6.0, the indexing speed of solr
is
> > > >> influenced by the number of solr collections. The speed is normal
> > before
> > > >> the limit is reached. If the limit is reached, the indexing speed
> will
> > > >> decrease by 50 times.
> > > >>
> > > >> In our environment, there are 49 solr nodes. If each collection has
> 25
> > > >> shards, you can maintain high-speed indexing until the total number
> of
> > > >> collections is about 900. To reduce the number of collections to the
> > > limit,
> > > >> the speed will increase. Go up.
> > > >> If each collection is 49 shards, the total number of collections can
> > > only
> > > >> be about 700, exceeding this value will cause the index to drop
> > > >> dramatically.
> > > >> In the explanation, we are single copies, and multiple copies will
> > cause
> > > >> serious stability problems in the large solr cluster environment.
> > > >>
> > > >> At first I suspect that it was due to too many thread submissions,
> and
> > > >> there are still problems with this method, so I'm inclined to
> > > >> searcherExecutor thread pool thread. This is just my guess, I want
> to
> > > know
> > > >> the real reason. Can someone know if I can help?
> > > >>
> > > >> Also, I noticed that the searcherExecutor thread and solr
> collection's
> > > >> shards basically correspond to each other. How can I reduce the
> number
> > > of
> > > >> threads or even close it? Although there are many collections in our
> > > >> environment, there are few queries and it is not necessary to keep
> the
> > > >> threads open to provide queries. This is too wasteful.
> > > >>
> > > >> thank you .
> > > >>
> > >
> >
>
>
>
> --
> ==============================
> 联创科技
> 知行如一
> ==============================
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message