From solr-user-return-139798-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Wed Mar 14 02:15:58 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 697E518067B for ; Wed, 14 Mar 2018 02:15:54 +0100 (CET) Received: (qmail 74744 invoked by uid 500); 14 Mar 2018 01:15:52 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 74732 invoked by uid 99); 14 Mar 2018 01:15:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Mar 2018 01:15:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 25BBD1A0155 for ; Wed, 14 Mar 2018 01:15:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id ANUyf_xcqz41 for ; Wed, 14 Mar 2018 01:15:49 +0000 (UTC) Received: from mail-it0-f52.google.com (mail-it0-f52.google.com [209.85.214.52]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id F08185F216 for ; Wed, 14 Mar 2018 01:15:48 +0000 (UTC) Received: by mail-it0-f52.google.com with SMTP id u5-v6so2611469itc.1 for ; Tue, 13 Mar 2018 18:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=DT2H0LUuCyiJ+65F9tqvjEZRyd5CfdWYFWS0h2JQ7BE=; b=fuFkDgDesXT0XDtKHd9QariVozcJDz0h7ud3oL1QI0kCtWFjO7CB9Axp2MaGPGRwB4 Z9F3RhHE11o/Rz6StRlvSWIHE9Wlxz3QTMKjP04nrtog3lPDGkyhhC4ZoMpSKR7kA16Q 5938sWaZ1vGQ3JG4qQqBtJfc5WFvTkngBNlme76PRuqzhsARD032//thK9PP1SqYpuIg 10eGoxZyXvAj3muC8NOEeE63v1mkMNvqFGW79yvuWsXrb1oDaYAVg0wuR0sVZwLPwBAs iLOYdlNjQsvP7lsPlBpC37i2oof+gwDjke+9L47fvYnNrPOb/wKKz6E/qSU4NOtInk4V TqUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=DT2H0LUuCyiJ+65F9tqvjEZRyd5CfdWYFWS0h2JQ7BE=; b=bVjpX9jrQjFU0j+sMLHYaXcVPa7CBACYa9Cymjze83ZIToWkYS05p1yvwkwX0tq4L2 0xn8N6W/Rx8pP6okKWxheGg0kLFuY4uqjCoMPapCJCcd1q2KONovpZ/8zJPMl3Pbttq2 XYm/2qukpAT4RvQE1x9ZIJ087IK9QD35m3SgKbhurQJDTG7oY9e0evq8wivGJXshpv1T jTWWwtt3JApSNiJJonI87g3CtczBvaXe3FD1wuVxNlyODnprgeHE4lKvXsZcCm4f/d6D as1JiaHqh3ybZpfuPYufjIwR1fsxFmbyJVQba9RN15ZSmTIO6aTgR7cEec0Wf0Kj+RjT m1fA== X-Gm-Message-State: AElRT7GVrKew6B9/kVW25Hrjwu5670A3uFdxEdi3vyfFN1UaPP8RFggt IdrC3GPhQHlDG2Q/hbgM9Jlhdc/Vo91xNqhOyL23Jg== X-Google-Smtp-Source: AG47ELvuwHu0RdSc/Pm/RbeCVFtNRTPz+dDKpxpix0h7dmkPT4NOE2eb3cnsbS+1jvohako6edO6XKZJE5MGq8lX4d4= X-Received: by 10.36.6.140 with SMTP id 134mr135589itv.75.1520990147433; Tue, 13 Mar 2018 18:15:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.150.156 with HTTP; Tue, 13 Mar 2018 18:15:46 -0700 (PDT) In-Reply-To: <1b4b5180-f355-816c-7e10-698d04a206c1@leximation.com> References: <1b4b5180-f355-816c-7e10-698d04a206c1@leximation.com> From: Greg Roodt Date: Wed, 14 Mar 2018 12:15:46 +1100 Message-ID: Subject: Re: Scoping SolrCloud setup To: solr-user@lucene.apache.org Content-Type: multipart/alternative; boundary="001a113f7a3471dcbb0567551d2f" --001a113f7a3471dcbb0567551d2f Content-Type: text/plain; charset="UTF-8" A single shard is much simpler conceptually and also cheaper to query. I would say that even your 1.2M collection can be a single shard. I'm running a single shard setup 4X that size. You can still have replicas of this shard for redundancy / availability purposes. I'm not an expert, but I think one of the deciding factors is if your index can fit into RAM (not JVM Heap, but OS cache). What are the sizes of your indexes? On 14 March 2018 at 11:01, Scott Prentice wrote: > We're in the process of moving from 12 single-core collections (non-cloud > Solr) on 3 VMs to a SolrCloud setup. Our collections aren't huge, ranging > in size from 50K to 150K documents with one at 1.2M docs. Our max query > frequency is rather low .. probably no more than 10-20/min. We do update > frequently, maybe 10-100 documents every 10 mins. > > Our prototype setup is using 3 VMs (4 core, 16GB RAM each), and we've got > each collection split into 2 shards with 3 replicas (one per VM). Also, > Zookeeper is running on each VM. I understand that it's best to have each > ZK server on a separate machine, but hoping this will work for now. > > This all seemed like a good place to start, but after reading lots of > articles and posts, I'm thinking that maybe our smaller collections (under > 100K docs) should just be one shard each, and maybe the 1.2M collection > should be more like 6 shards. How do you decide how many shards is right? > > Also, our current live system is separated into dev/stage/prod tiers, not, > all of these tiers are together on each of the cloud VMs. This bothers some > people, thinking that it may make our production environment less stable. I > know that in an ideal world, we'd have them all on separate systems, but > with the replication, it seems like we're going to make the overall system > more stable. Is this a correct understanding? > > I'm just wondering anyone has opinions on whether we're going in a > reasonable direction or not. Are there any articles that discuss these > initial sizing/scoping issues? > > Thanks! > ...scott > > > --001a113f7a3471dcbb0567551d2f--