From solr-user-return-139954-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Sat Mar 17 21:03:36 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4736A180652 for ; Sat, 17 Mar 2018 21:03:36 +0100 (CET) Received: (qmail 3515 invoked by uid 500); 17 Mar 2018 20:03:34 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 3499 invoked by uid 99); 17 Mar 2018 20:03:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Mar 2018 20:03:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0A97A180156 for ; Sat, 17 Mar 2018 20:03:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.13 X-Spam-Level: ** X-Spam-Status: No, score=2.13 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id JfLK4aC9XsYy for ; Sat, 17 Mar 2018 20:03:31 +0000 (UTC) Received: from mail-pl0-f52.google.com (mail-pl0-f52.google.com [209.85.160.52]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2BE5C5F189 for ; Sat, 17 Mar 2018 20:03:30 +0000 (UTC) Received: by mail-pl0-f52.google.com with SMTP id w12-v6so7857973plp.4 for ; Sat, 17 Mar 2018 13:03:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=qmSDg6kDm1axYxCyMspgf2xvp/uQEfxCJDpmWmVcudg=; b=EUtn2259AZRJtiByrvB3FvdDQEVwwTQjZ9w+g6o530Rthx0thStdVA2bZlMv++fNXd 8v3DQ1P3nMOt/Xvhs4LFKWWbbWrf9eUe313rAEZEAvJZvBOI1fpJS8ZCIQBmBBbCD+jF xO/8g5QymDRFIGOTNWYIB7iEmJMeWomAoRwqzh9hADxRLqLtBlY20DqhjGIRQAMhYHKC TdgDdM6TuX59gmGrTJ4ivUzIYT50nr6PhW9rd0RBZYWTTx5xD1rgyriCwTJwiJO1oC/3 TULYc9M/kNrFVMsWD2q4E/6W1murdUTfmNsNtHAJxhai1TAfmjw2duVokLC54DXfGCHG Ltjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=qmSDg6kDm1axYxCyMspgf2xvp/uQEfxCJDpmWmVcudg=; b=PyS9TeRDSzdIhWSHD/FSPbuqaBkRVDLbBfoD0Ot0rI0J4mVNuxkKSk7xAKdEBg4srF n+Zjb1meE3TP1wIkrSF57tCDzlG2C1UsdOV3TIu0FAkthlMCT+GF6DKo0rJICjJOgoqC Ozxzjb7Ga//cFl5ynbVHfuNMS4tQN5F/YGO5gA61z3XcKyOBhQVuLyOZ86ms+nBwcOoJ 4SFwMn5XANhEPXgIAroiCS3TzTf92G5Gn10oRHQTlVWXJ1/PvEhNV83pXrY/9XhbjG3R n4fNXEwh2NXeLc+9+X6mpBhWMyHKtn5ETt840K5SuAL1erC4L0dP8dIwioRgGs7mJTWN 9VHg== X-Gm-Message-State: AElRT7GdAoehgIS6tD+FqJcZM8xH5E6VPT7UAv9Oqz5t7+LAX5crIWHe ARDlIh4wwvM6jMUBbrJInODJmC3oxXBZ5JBBNGJ/gA== X-Google-Smtp-Source: AG47ELtaVhirP8PaR2ioaGZOHUFKza7DpKkvz0U6LdEtgUAJBFP8PpABRxUEA/AFLT4eNWi3yEy6NJA8YtwQtoos7hY= X-Received: by 2002:a17:902:6103:: with SMTP id t3-v6mr6778493plj.76.1521317003137; Sat, 17 Mar 2018 13:03:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.236.176.2 with HTTP; Sat, 17 Mar 2018 13:03:22 -0700 (PDT) In-Reply-To: References: <4eb975df-51fb-e610-8cd7-d60ec3fdd71b@elyograg.org> <00fd09ab-5a24-9d8d-cda6-81302d7acab1@elyograg.org> <490f6d5d-4ed5-0a75-468e-51ec32023e36@elyograg.org> <9122824E-E3A1-439C-B3C9-86F210395F23@wunderwood.org> <2630A6FE-5B9D-4C84-B818-DA36470086F2@wunderwood.org> From: BlackIce Date: Sat, 17 Mar 2018 21:03:22 +0100 Message-ID: Subject: Re: Some performance questions.... To: solr-user@lucene.apache.org Content-Type: multipart/alternative; boundary="0000000000009015b10567a137b0" --0000000000009015b10567a137b0 Content-Type: text/plain; charset="UTF-8" Looks like I've opened up a very interesting can of worms.... Thank you to all that are posting to this thread, I'm learning a lot... The way I see it now... a Single Solr instance on this machine, seems like the most intelligent choice. And then as upgrade path, adding in-expensive machines. This adds me storage space, cpu power and starts to build up on the parallezation cluster and load balancing. Greetz On Sat, Mar 17, 2018 at 11:23 AM, Deepak Goel wrote: > On 17 Mar 2018 05:19, "Walter Underwood" wrote: > > > On Mar 16, 2018, at 3:26 PM, Deepak Goel wrote: > > > > Can you please post results of your test? > > > > Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource > > > I could, but it probably would not be useful for your documents or your > queries. > > We have 22 million homework problems. Our queries are often hundreds of > words long, > because students copy and paste entire problems. After pre-processing, the > average query > is still 25 words. > > For load benchmarking, I use access logs from production. I typically > gather over a half-million > lines of log. Using production logs means that queries have the same > statistical distribution > as prod, so the cache hit rates are reasonable. > > Before each benchmark, I restart all the Solr instances to clear the > caches. Then the first part > of the query log is used to warm the caches, typically about 4000 queries. > > After that, the measured benchmark run starts. This uses JMeter with > 100-500 threads. Each > thread is configured with a constant throughput timer so a constant load is > offered. Test run > one or two hours. Recently, I ran a test with a rate of 1000 > requests/minute for one hour. > > During the benchmark, I monitor the CPU usage. Our systems are configured > with enough RAM > so that disk is not accessed for search indexes. If the CPU goes over > 75-80%, there is congestion > and queries will slow down. Also, if the run queue (load average) increases > over the number of > CPUs, there will be congestion. > > After the benchmark run, the JMeter log is analyzed to report response time > percentiles for > each Solr request handler. > > > Sorry for being rude. But the ' results ' please, not the ' road to the > results ' > > > wunder > Walter Underwood > wunder@wunderwood.org > http://observer.wunderwood.org/ (my blog) > --0000000000009015b10567a137b0--