From solr-user-return-143834-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Tue Sep 18 19:00:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BC91B180672 for ; Tue, 18 Sep 2018 19:00:45 +0200 (CEST) Received: (qmail 904 invoked by uid 500); 18 Sep 2018 17:00:43 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 873 invoked by uid 99); 18 Sep 2018 17:00:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Sep 2018 17:00:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BD15AC7F2E for ; Tue, 18 Sep 2018 17:00:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.9 X-Spam-Level: * X-Spam-Status: No, score=1.9 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ELUQmaLro7fl for ; Tue, 18 Sep 2018 17:00:42 +0000 (UTC) Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DEE165F4E7 for ; Tue, 18 Sep 2018 17:00:41 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id w17-v6so2740951otk.3 for ; Tue, 18 Sep 2018 10:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=KY8Myjw3GvFprxx86F8GiLMfRiQXrYGwRHS53HwzLc0=; b=cRzdEocxabnyLO4I5/BNhyU8Pyu46JVFsAO3Tzuwx8zXT7VBvt4tnVg8ZBaPJ7gAGY o0TmeICBIK6OhRRqGEmVg8ACAQWbLx90BwD0T5dHNnRr9bFZXl+TW3y8jv1esjwfi3I3 i1CazBTFH+CzOcqj3L0XkZafyvp+1Wqq8kMja0NJEBmOvECBFIPF8nPHhQJKlZUJ97/R 2Jl/hfG0wFCcY98s7dhdVykIgFgNtNWgPMTmNoqiidDAP0xulr0c+T5mi5wunTBUpzjT /txSY+VeffOtd/D4CST/ilyMXFvQVXwR5qQPc4RSY77v+xbBhKs4ppGSHHOJH2wIxw/l U2aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=KY8Myjw3GvFprxx86F8GiLMfRiQXrYGwRHS53HwzLc0=; b=fSSbUp6DxR1WeSUHCDdrMQovOrmmo4mMDme61Jd5HT1FWtf7RUEd1UPCNNvwefHd2q Hsy7H3JJQepjZPWwOqzN444IdgIY5Wr0722XRtp9JdaG2GC+RIMed4mkKHCpWLAfbyvJ MlRMzEhRq3Nziy2EHdV7Chi2Wsk5GPGDbAXGj+SZrKRX7GhuzxYYkYyG/iSI30/9irzG Y5yGWliEF4WZn5gAod8iQCvYcDbT2Y25O2v+hMKXsecqYMtzCBl8pX4gOjRyqFb6Bnu+ 6pOPZZIJzuHTNpIfdhMfW6HgqkeDjcWwA2Hn/rHaUg+tcBtMhdAiO9pPCKuIEKbxS3jd kbjA== X-Gm-Message-State: APzg51BfXD38qYrf865ZdGv0wawdDlBkkdF4P8AQtdMTU8Aj7zm3WTtt JsqYrGNC7Ze3LQVnhLX8LVXagHh5ypZgAzfyXFRCQdgz X-Google-Smtp-Source: ANB0VdYsremecd3wFXM5LgFTmJ88dI7zpCowdVxD/9P3YmxAVrU1qb0rrv7404KYik74J2KNk3TOnhd+nYbLUpHFFxQ= X-Received: by 2002:a9d:2cc6:: with SMTP id e6-v6mr17031446otd.154.1537290040872; Tue, 18 Sep 2018 10:00:40 -0700 (PDT) MIME-Version: 1.0 From: Ganesh Sethuraman Date: Tue, 18 Sep 2018 13:00:29 -0400 Message-ID: Subject: Solr 7.2.1 Collection Backup Performance issue To: solr-user@lucene.apache.org Content-Type: multipart/alternative; boundary="000000000000cd99520576283a2e" --000000000000cd99520576283a2e Content-Type: text/plain; charset="UTF-8" Hi We are using Solr 7.2.1 with SolrCloud with 35 collections with 1 node ZK ensemble (in lower environment, we will have 3 nodes ensemble) in AWS. We are testing to see if we have Async Solr Cloud backup ( https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup) done every time we are create a new collection or update an existing collection. There are 1 replica and 8 shards per collection. Two Solr nodes. For the largest collection (index size of 80GB), we see that BACKUP to the EFS drive takes about ~10 mins. We are doing lot of /get (real time get) option from the application. We are seeing that that the performance significantly (2x) degrades on the read (get) performance when we BACK-UP is going on in parallel. Is there anyway to tune the system so that read does not suffer? Any other best practices? like should we run back up during off peak load? Is there a way to keep track of which collections are already backed up? --000000000000cd99520576283a2e--