From solr-user-return-142037-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Fri Jun 22 16:13:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B53BA180647 for ; Fri, 22 Jun 2018 16:13:04 +0200 (CEST) Received: (qmail 60815 invoked by uid 500); 22 Jun 2018 14:13:02 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 60643 invoked by uid 99); 22 Jun 2018 14:13:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jun 2018 14:13:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id AD5A21A1085 for ; Fri, 22 Jun 2018 14:13:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.102 X-Spam-Level: X-Spam-Status: No, score=-0.102 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=elyograg.org Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Y_4iToRFmq2V for ; Fri, 22 Jun 2018 14:12:59 +0000 (UTC) Received: from frodo.elyograg.org (frodo.elyograg.org [166.70.79.217]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 54FFB5F35A for ; Fri, 22 Jun 2018 14:12:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by frodo.elyograg.org (Postfix) with ESMTP id ADCB326F for ; Fri, 22 Jun 2018 08:12:56 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=elyograg.org; h= content-language:content-transfer-encoding:content-type :content-type:in-reply-to:mime-version:user-agent:date:date :message-id:from:from:references:subject:subject:received :received; s=mail; t=1529676776; bh=UY004R8r6eXsrQwdzKhYnoWfn2uu fK2/xb0BqxXtOR8=; b=iCP4zL+58FBzztZdnLW+IFO9FQegzZUJmhZT1iMSiGLr 7FuwUiJgs31uGwb9qVYyr5pKzxH+n/faUW0hLKgjfHQww/MAFoqR0qzPd5QaG8Pw QCLWapEP1MqfEtu4VjI6FXtj5L4bWx3lOAYI/JPMFhvHYjBJaCT3LySLrhAFouA= X-Virus-Scanned: Debian amavisd-new at frodo.elyograg.org Received: from frodo.elyograg.org ([127.0.0.1]) by localhost (frodo.elyograg.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id qqWIBT7whicx for ; Fri, 22 Jun 2018 08:12:56 -0600 (MDT) Received: from [192.168.1.114] (114.int.elyograg.org [192.168.1.114]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: elyograg@elyograg.org) by frodo.elyograg.org (Postfix) with ESMTPSA id 60EF04A for ; Fri, 22 Jun 2018 08:12:56 -0600 (MDT) Subject: Re: top 10 query overall vs shard To: solr-user@lucene.apache.org References: From: Shawn Heisey Message-ID: Date: Fri, 22 Jun 2018 08:12:58 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US On 6/22/2018 6:50 AM, Arturas Mazeika wrote: > I grabbed the 2.7.1 version of solr, created a 4 core setup with > replication factor 2 on windows using [1], I've restarted the setup with > 2GB for each node [2], inserted the html docs from the german wikipedia > archive [3], and obtained top 10 terms for the whole collection vs one > specific shard: > http://localhost:9999/solr/de_wiki_all/terms?terms.limit=10&terms.fl=text&wt=json > { > "responseHeader":{ > "zkConnected":true, > > "status":0, > "QTime":5287}, > "terms":{ > "text":[ > "8",670564, > "application",670564, > "articles",670564, > "charset",670564, > "de",670564, > "f",670564, > "utf",670564, > "wiki",670564, > "xhtml",670564, > "xml",670564]}} > > http://localhost:9999/solr/de_wiki_all/terms?terms.limit=10&terms.fl=text&wt=json&shards=localhost:9999/solr/de_wiki_all_shard1_replica_n1&shards.qt=de_wiki_all_shard1_replica_n1 > > { > "responseHeader":{ > "zkConnected":true, > "status":0, > "QTime":20274}, > "terms":{ > "text":{ > "8":671396, > "application":671396, > "articles":671396, > "charset":671396, > "de":671396, > "f":671396, > "utf":671396, > "wiki":671396, > "xhtml":671396, > "xml":671396}}} The value of 'shards.qt' should be /terms, not the name of a core.  Here's something you might want to try instead for the second query, so you won't need shards.qt at all: http://localhost:9999/solr/de_wiki_all_shard1_replica_n1/terms?terms.limit=10&terms.fl=text&wt=json&distrib=false You might actually want to add shards.qt=/terms to the first query, or even to the definition of the /terms handler in solrconfig.xml so that all distributed queries are sent to the same handler instead of going to /select. > reveals: > (1) querying one shard takes 20 secs vs 5 secs for the whole index That is strange.  With the shards.qt parameter set to a core name, I'm surprised you got anything at all on the second query, but maybe when it couldn't find a handler with that name, it just defaulted to /select like it would if you didn't include the parameter.  I wonder if having an invalid handler contributed to the speed. > (2) the counts for one shards are higher than for the whole index If you're not changing the index between the requests, and it doesn't sound like you are, I have no idea why that might happen. > (3) the f: hard drive is samsung SSD 850 evo 4TB (CrystalDeiskMark shows > ~500MB/s seq and ~300MBs random read/writes), CPU:i7-6400 @3.4GHz. Querying > for 20 secs shows that java process is neither being pushed on the CPU nor > on the SDD side to the limits. What is the bottleneck in this computation? If the amount of memory in the system (NOT talking about heap size here) is not sufficient to effectively cache the index, then Solr must actually hit the disk to satisfy a query.  Even an SSD is not as fast as memory.  You haven't indicated how much disk space is being consumed by the eight index cores or how much total memory the system has.  A little more than 8GB of the system's memory is being taken up by the four Solr processes.  Because you've asked for two replicas, there are two complete copies of the index on the system, and both copies will count in the total amount of resources that are required. If there *is* sufficient memory for effective index caching, then the disk will barely see any usage during queries, because Solr will get most of the data it needs from the OS disk cache (system memory).  This will also reduce the impact on the CPU, because it will not be waiting for I/O. Running a query is not going to read the entire index.  If it did, Solr would not be fast. > (4) the output format is slightly different (compare ',' vs ':' and vector > vs list). I wonder why That I cannot explain.  The first response doesn't look right to me.  It passes RFC 4627 validation, but the software parsing the response would have to be very different for each of the output formats. Thanks, Shawn