Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
From: Suresh Pendap <SPendap@walmartlabs.com>
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Subject: Re: unable to get more throughput with more threads
Thread-Topic: unable to get more throughput with more threads
Thread-Index: AQHSpDMajcsmMUTcnUeFknMBpDB04qGjMYEAgAAZd4D//6gCgA==
Date: Fri, 24 Mar 2017 04:37:28 +0000
Message-ID: <D4F9EFF3.6BCC0%spendap@walmartlabs.com>
References: <D4F9B226.6BB73%spendap@walmartlabs.com>
 <CAB_OoUOihq+p=MSHrUXDsPNLf7jJ2TT066oMSAyq=jovGMvEPw@mail.gmail.com>
 <CAF2DzVW1naajv5DGtdoSERWt50jwOPzAwVtEdjtFLXB6_PAW9w@mail.gmail.com>
In-Reply-To: <CAF2DzVW1naajv5DGtdoSERWt50jwOPzAwVtEdjtFLXB6_PAW9w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [216.207.42.140]
x-ms-office365-filtering-correlation-id: 613c6261-d1b7-4c5f-2831-08d4726f7a5a
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254075);SRVR:BN1P100MB0166;
x-microsoft-exchange-diagnostics: 1;BN1P100MB0166;7:m8XU8yUkcPzDP9L5igr2uas/3dfpczBne5GzIB25vFZtlFMuj2/Y14vifrTM6tlbznMLVLNUUBx9djXE57m9amj+6b1mFGAsuYhs67hhXEjc8QFszXFq/E+0C8imr32enMfH0Bo2Ey3nlv1iFTOC0VekATMDJtUzx3575fO1OLsh48soRXMrRpsa3NVFhCF0uQZslTwsUh+fsbK+FzZydtFl2RACitO8Gg+0zJ+08tr0gaE49BExyWmUMAEHH/7gG2l97K5RgqgK7Hx1EY6F9p3sqyA+kg3yBOpcCHNtMo3IEFxXJFvD/tDSijzBdksIPQGR95NXRGd5n7rexm04FA==;20:QannxPAqlFVEbUBQpfviTOrDjyP6DRiVsXgpY/ukPiStVtg8uGpFYIZEvm5s8JoKzD8IDZ1zbOA0vnCzEE8xsitndzSQuzwhFoYKcdT/FeI99mIMGLzLDVFWSUJDeUOhdd/fOchf3sIJ95JSf89PZP7K3bbj5wS++rzO4a0DC+5WPZWIqg0PjxZajwAJap/Yok/33V4M/eHGcn5s70yjChehcPj3aT51CoI/BJoIPYkjafTnHywFq6YPcC7NrzVkxdMTAV3KlH1eO6MQuG7uhvowhrcYScpMt/Nub1ZlfS2BmG1aNdNxKiWew2ruxqD2QUu/W+HqX3J28vYdQCqe+MuQuMFufxlnnrk/3vjbgtriVgEiSzwj+WSuQY+pL4etsDUri1UoSS2EMh0PUScxSK8MiQSRBKXRhcDe6BTk4HiFCnIW0FBJq8j4VGyvYe115RpP0z56D8nS0hyWhpEKN/46iVaFmUnKzzf/hSIksy1WdwF8y07BC0GMlF4oc5ZG
x-microsoft-antispam-prvs: <BN1P100MB0166972C6029E609529FF9E6A43E0@BN1P100MB0166.NAMP100.PROD.OUTLOOK.COM>
x-exchange-antispam-report-test: UriScan:(158342451672863);
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040375)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6041248)(20161123558025)(20161123560025)(20161123562025)(20161123555025)(20161123564025)(6072148);SRVR:BN1P100MB0166;BCL:0;PCL:0;RULEID:;SRVR:BN1P100MB0166;
x-forefront-prvs: 0256C18696
x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(39450400003)(24454002)(377454003)(110136004)(81166006)(36756003)(54356999)(8676002)(2351001)(3846002)(2900100001)(5640700003)(53546009)(8936002)(6506006)(66066001)(25786009)(38730400002)(305945005)(5660300001)(2950100002)(229853002)(6436002)(6916009)(86362001)(3280700002)(50986999)(80792005)(102836003)(2501003)(189998001)(76176999)(3660700001)(6116002)(6486002)(53936002)(2906002)(122556002)(6512007);DIR:OUT;SFP:1101;SCL:1;SRVR:BN1P100MB0166;H:BN1P100MB0167.NAMP100.PROD.OUTLOOK.COM;FPR:;SPF:None;MLV:sfv;LANG:en;
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3A337E59E4B3A04F982732E4088CA4A8@NAMP100.PROD.OUTLOOK.COM>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2017 04:37:28.4026
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 3cbcc3d3-094d-4006-9849-0d11d61f484d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1P100MB0166
X-OriginatorOrg: walmartlabs.com
X-CFilter-Loop: Reflected
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-03-24_03:,,
 signatures=0
archived-at: Fri, 24 Mar 2017 04:38:44 -0000

Edwin,
The heap was not being used much, only 1GB of heap was being used out of
8GB.  I do have space to allocate more to the heap size.
I was reading in some SOLR performance blogs that it is better not to use
large heap size, instead it is better to provide lot of space to the
Operating system
Disk cache so that maximum documents stay in memory in the buffer cache.

Regards
Suresh


On 3/23/17 7:52 PM, "Zheng Lin Edwin Yeo" <edwinyeozl@gmail.com> wrote:

>I also did find that beyond 10 threads for 8GB heap size , there isn't
>much
>improvement with the performance. But you can increase your heap size a
>little if your system allows it.
>
>By the way, which Solr version are you using?
>
>Regards,
>Edwin
>
>
>On 24 March 2017 at 09:21, Matt Magnusson <magnusson3@gmail.com> wrote:
>
>> Out of curosity, what is your index size? I'm trying to do something
>> similar with maximizing output, I'm currently looking at streaming
>> expressions which I'm seeing some interesting results for, I'm also
>> finding that the direct mass query route seems to hit a wall for
>> performance. I'm also finding that about 10 threads seems to be an
>> optimum number.
>>
>> On Thu, Mar 23, 2017 at 8:10 PM, Suresh Pendap <SPendap@walmartlabs.com>
>> wrote:
>> > Hi,
>> > I am new to SOLR search engine technology and I am trying to get some
>> performance numbers to get maximum throughput from the SOLR cluster of a
>> given size.
>> > I am currently doing only query load testing in which I randomly fire
>>a
>> bunch of queries to the SOLR cluster to generate the query load.  I
>> understand that it is not the ideal workload as the
>> > ingestion and commits happening invalidate the Solr Caches, so it is
>> advisable to perform query load along with some documents being
>>ingested.
>> >
>> > The SOLR cluster was made up of 2 shards and 2 replicas. So there were
>> total 4 replicas serving the queries. The SOLR nodes were running on an
>>LXD
>> container with 12 cores and 88GB RAM.
>> > The heap size allocated was 8g min and 8g max. All the other SOLR
>> configurations were default.
>> >
>> > The client node was running on an 8 core VM.
>> >
>> > I performed the test with 1 thread, 10 client threads and 50 client
>> threads.  I noticed that as I increased the number of threads, the query
>> latency kept increasing drastically which I was not expecting.
>> >
>> > Since my initial test was randomly picking queries from a file, I
>> decided to keep things constant and ran the program which fired the same
>> query again and again. Since it is the same query, all the documents
>>will
>> > be in the Cache and the query response time should be very fast. I was
>> also expecting that with 10 or 50 client threads, the query latency
>>should
>> not be increasing.
>> >
>> > The throughput increased only up to 10 client threads but then it was
>> same for 50 threads, 100 threads and the latency of the query kept
>> increasing as I increased the number of threads.
>> > The query was returning 2 documents only.
>> >
>> > The table below summarizes the numbers that I was saying with a single
>> query.
>> >
>> >
>> >
>> >
>> >
>> > #No of Client Nodes
>> >         #No of Threads  99 pct Latency  95 pct latency  throughput
>> CPU Utilization Server Configuration
>> >
>> > 1       1       9 ms    7 ms    180 reqs/sec    8%
>> >
>> > Heap size: ms=3D8g, mx=3D8g
>> >
>> > default configuration
>> >
>> >
>> > 1       10      400 ms  360 ms  360 reqs/sec    10%
>> >
>> > Heap size: ms=3D8g, mx=3D8g
>> >
>> > default configuration
>> >
>> >
>> >
>> >
>> > I also ran the client program on the SOLR server node in order to rule
>> our the network latency factor. On the server node also the response
>>time
>> was higher for 10 threads, but the amplification was smaller.
>> >
>> > I am getting an impression that probably my query requests are getting
>> queued up and limited due to probably some thread pool size on the
>>server
>> side.  However I saw that the default jetty.xml does
>> > have the thread pool of min size of 10 and  max of 10000.
>> >
>> > Is there any other internal SOLR thread pool configuration which might
>> be limiting the query response time?
>> >
>> > I wanted to check with the community if what I am seeing is abnormal
>> behavior, what could be the issue?  Is there any configuration that I
>>can
>> tweak to get better query response times for more load?
>> >
>> > Regards
>> > Suresh
>> >
>>