lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr Queries are very slow - Suggestions needed
Date Mon, 14 Mar 2016 17:00:07 GMT
Try shards.info=true, but pinging the shard directly is the most certain.


Best,
Erick

On Mon, Mar 14, 2016 at 9:48 AM, Anil <anilklce@gmail.com> wrote:
> HI Erik,
>
> we have used document routing to balance the shards load and for
> expand/collapse. it is mainly used for main_collection which holds one to
> many relationship records. In file_collection, it is only for load
> distribution.
>
> 25GB for entire solr service. each machine will act as shard for some
> collections.
>
> we have not stress tested our servers at least for solr service. i have
> read the the link you have shared, i will do something on it. thanks for
> sharing.
>
> i have checked other collections, where index size is max 90GB and 5 M as
> max number of documents. but for the particular file_collection_2014 , i
> see total index size across replicas is 147 GB.
>
> Can we get any hints if we run the query with debugQuery=true ?  what is
> the effective way of load distribution ? Please advice.
>
> Regards,
> Anil
>
> On 14 March 2016 at 20:32, Erick Erickson <erickerickson@gmail.com> wrote:
>
>> bq: The slowness is happening for file_collection. though it has 3 shards,
>> documents are available in 2 shards. shard1 - 150M docs and shard2 has 330M
>> docs , shard3 is empty.
>>
>> Well, this collection terribly balanced. Putting 330M docs on a single
>> shard is
>> pushing the limits, the only time I've seen that many docs on a shard,
>> particularly
>> with 25G of ram, they were very small records. My guess is that you will
>> find
>> the queries you send to that shard substantially slower than the 150M
>> shard,
>> although 150M could also be pushing your limits. You can measure this
>> by sending the query to the specific core (something like
>>
>> solr/files_shard1_replica1/query?(your queryhere)&distrib=false
>>
>> My bet is that your QTime will be significantly different with the two
>> shards.
>>
>> It also sounds like you're using implicit routing where you control where
>> the
>> files go, it's easy to have unbalanced shards in that case, why did you
>> decide
>> to do it this way? There are valid reasons, but...
>>
>> In short, my guess is that you've simply overloaded your shard with
>> 330M docs. It's
>> not at all clear that even 150 will give you satisfactory performance,
>> have you stress
>> tested your servers? Here's the long form of sizing:
>>
>>
>> https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
>>
>> Best,
>> Erick
>>
>> On Mon, Mar 14, 2016 at 7:05 AM, Susheel Kumar <susheel2777@gmail.com>
>> wrote:
>> > For each of the solr machines/shards you have.  Thanks.
>> >
>> > On Mon, Mar 14, 2016 at 10:04 AM, Susheel Kumar <susheel2777@gmail.com>
>> > wrote:
>> >
>> >> Hello Anil,
>> >>
>> >> Can you go to Solr Admin Panel -> Dashboard and share all 4 memory
>> >> parameters under System / share the snapshot. ?
>> >>
>> >> Thanks,
>> >> Susheel
>> >>
>> >> On Mon, Mar 14, 2016 at 5:36 AM, Anil <anilklce@gmail.com> wrote:
>> >>
>> >>> HI Toke and Jack,
>> >>>
>> >>> Please find the details below.
>> >>>
>> >>> * How large are your 3 shards in bytes? (total index across replicas)
>> >>>           --  *146G. i am using CDH (cloudera), not sure how to check
>> the
>> >>> index size of each collection on each shard*
>> >>> * What storage system do you use (local SSD, local spinning drives,
>> remote
>> >>> storage...)? *Local (hdfs) spinning drives*
>> >>> * How much physical memory does your system have? *we have 15 data
>> nodes.
>> >>> multiple services installed on each data node (252 GB RAM for each data
>> >>> node). 25 gb RAM allocated for solr service.*
>> >>> * How much memory is free for disk cache? *i could not find.*
>> >>> * How many concurrent queries do you issue? *very less. i dont see any
>> >>> concurrent queries to this file_collection for now.*
>> >>> * Do you update while you search? *Yes.. its very less.*
>> >>> * What does a full query (rows, faceting, grouping, highlighting,
>> >>> everything) look like? *for the file_collection, rows - 100,
>> highlights =
>> >>> false, no facets, expand = false.*
>> >>> * How many documents does a typical query match (hitcount)? *it varies
>> >>> with
>> >>> each file. i have sort on int field to order commands in the query.*
>> >>>
>> >>> we have two sets of collections on solr cluster ( 17 data nodes)
>> >>>
>> >>> 1. main_collection - collection created per year. each collection uses
>> 8
>> >>> shards 2 replicas ex: main_collection_2016, main_collection_2015 etc
>> >>>
>> >>> 2. file_collection (where files having commands are indexed) -
>> collection
>> >>> created per 2 years. it uses 3 shards and 2 replicas. ex :
>> >>> file_collection_2014, file_collection_2016
>> >>>
>> >>> The slowness is happening for file_collection. though it has 3 shards,
>> >>> documents are available in 2 shards. shard1 - 150M docs and shard2 has
>> >>> 330M
>> >>> docs , shard3 is empty.
>> >>>
>> >>> main_collection is looks good.
>> >>>
>> >>> please let me know if you need any additional details.
>> >>>
>> >>> Regards,
>> >>> Anil
>> >>>
>> >>>
>> >>> On 13 March 2016 at 21:48, Anil <anilklce@gmail.com> wrote:
>> >>>
>> >>> > Thanks Toke and Jack.
>> >>> >
>> >>> > Jack,
>> >>> >
>> >>> > Yes. it is 480 million :)
>> >>> >
>> >>> > I will share the additional details soon. thanks.
>> >>> >
>> >>> >
>> >>> > Regards,
>> >>> > Anil
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > On 13 March 2016 at 21:06, Jack Krupansky <jack.krupansky@gmail.com>
>> >>> > wrote:
>> >>> >
>> >>> >> (We should have a wiki/doc page for the "usual list of suspects"
>> when
>> >>> >> queries are/appear slow, rather than need to repeat the same
>> mantra(s)
>> >>> for
>> >>> >> every inquiry on this topic.)
>> >>> >>
>> >>> >>
>> >>> >> -- Jack Krupansky
>> >>> >>
>> >>> >> On Sun, Mar 13, 2016 at 11:29 AM, Toke Eskildsen <
>> >>> te@statsbiblioteket.dk>
>> >>> >> wrote:
>> >>> >>
>> >>> >> > Anil <anilklce@gmail.com> wrote:
>> >>> >> > > i have indexed a data (commands from files) with
10 fields and
>> 3 of
>> >>> >> them
>> >>> >> > is
>> >>> >> > > text fields. collection is created with 3 shards
and 2
>> replicas. I
>> >>> >> have
>> >>> >> > > used document routing as well.
>> >>> >> >
>> >>> >> > > Currently collection holds 47,80,01,405 records.
>> >>> >> >
>> >>> >> > ...480 million, right? Funny digit grouping in India.
>> >>> >> >
>> >>> >> > > text search against text field taking around 5 sec.
solr is
>> query
>> >>> just
>> >>> >> > and
>> >>> >> > > of two terms with fl as 7 fields
>> >>> >> >
>> >>> >> > > fileId:"file unique id" AND command_text:(system
login)
>> >>> >> >
>> >>> >> > While not an impressive response time, it might just be
that your
>> >>> >> hardware
>> >>> >> > is not enough to handle that amount of documents. The
usual
>> culprit
>> >>> is
>> >>> >> IO
>> >>> >> > speed, so chances are you have a system with spinning
drives and
>> not
>> >>> >> enough
>> >>> >> > RAM: Switch to SSD and/or add more RAM.
>> >>> >> >
>> >>> >> > To give better advice, we need more information.
>> >>> >> >
>> >>> >> > * How large are your 3 shards in bytes?
>> >>> >> > * What storage system do you use (local SSD, local spinning
>> drives,
>> >>> >> remote
>> >>> >> > storage...)?
>> >>> >> > * How much physical memory does your system have?
>> >>> >> > * How much memory is free for disk cache?
>> >>> >> > * How many concurrent queries do you issue?
>> >>> >> > * Do you update while you search?
>> >>> >> > * What does a full query (rows, faceting, grouping, highlighting,
>> >>> >> > everything) look like?
>> >>> >> > * How many documents does a typical query match (hitcount)?
>> >>> >> >
>> >>> >> > - Toke Eskildsen
>> >>> >> >
>> >>> >>
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>>

Mime
View raw message