Return-Path: 
 <solr-user-return-95339-apmail-lucene-solr-user-archive=lucene.apache.org@lucene.apache.org>
X-Original-To: apmail-lucene-solr-user-archive@minotaur.apache.org
Delivered-To: apmail-lucene-solr-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 644D010FB3
	for <apmail-lucene-solr-user-archive@minotaur.apache.org>;
 Mon, 23 Dec 2013 22:44:07 +0000 (UTC)
Received: (qmail 92585 invoked by uid 500); 23 Dec 2013 22:44:03 -0000
Delivered-To: apmail-lucene-solr-user-archive@lucene.apache.org
Received: (qmail 92471 invoked by uid 500); 23 Dec 2013 22:44:03 -0000
Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:solr-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:solr-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:solr-user@lucene.apache.org>
List-Id: <solr-user.lucene.apache.org>
Reply-To: solr-user@lucene.apache.org
Delivered-To: mailing list solr-user@lucene.apache.org
Received: (qmail 92463 invoked by uid 99); 23 Dec 2013 22:44:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Dec 2013 22:44:03 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gpreston@marinsoftware.com
 designates 209.85.212.47 as permitted sender)
Received: from [209.85.212.47] (HELO mail-vb0-f47.google.com) (209.85.212.47)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Dec 2013 22:43:57 +0000
Received: by mail-vb0-f47.google.com with SMTP id q12so2992227vbe.6
        for <solr-user@lucene.apache.org>;
 Mon, 23 Dec 2013 14:43:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=marinsoftware.com; s=google;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        bh=ypCVfdv7fMyNZjn0uXyld4RQTDNp5XL8KsdEs/c8+Ig=;
        b=VGpi0VXLQrl7pIemn8ZGp/KOpcVgFbF1wAk51F51CrfD7eO7vDNhOI1a8F85P5aSkk
         7KFV+oip3J5t4QGSWrK92uC/A6an7aq4lDsyKnS+Kf0aeBq/xO/y/RWb16TnLljKIhIV
         fPjRNyYyr1OBre3jTONMZeuXEyHdKRRuh9Q5s=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=ypCVfdv7fMyNZjn0uXyld4RQTDNp5XL8KsdEs/c8+Ig=;
        b=LUOmU2dZAvzdZwIR2PDx3UUfBEzSJUrBw/PkAL1DaFPXTdIWFH2LBN8YIqsdI39Y61
         nsdf4xLQNgyvma2S75Ti0i3wcRT/TQlfFgQOeVej0Ec1H5e2ZpNLUJvV57GHUHPkxG+n
         k8RR5QY+h8Gt6asrPXz0yg0bjkrP6ypnurISS2ywCDvb7NhF8y0U1FzEOsKEc6BJiKAH
         /G3SaJsTUkO6+5S2DKF56s9T+Q1HqdnGLFfHfOc1vqCXaIq7S1UarUsWpZlwmS7s+ZWI
         DEo5GSpdW0BhbzzEQaVzRtSbu0aZOk8ODvuVDu1rUa+lM1Ns2/QckJXuA8mtS3ywOOT3
         it0w==
X-Gm-Message-State: 
 ALoCoQkGX4T7Bc79Q4ewXFLtidNKRQsNjZ67dx8tfeRoU8PtRGoV7NMTFjDlbDWb+1ThY8nmYyXn
X-Received: by 10.220.57.195 with SMTP id d3mr3799738vch.15.1387838616572;
 Mon, 23 Dec 2013 14:43:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.52.105.230 with HTTP; Mon, 23 Dec 2013 14:43:16 -0800 (PST)
In-Reply-To: <52B8B659.90101@gmail.com>
References: <52B6D189.6040601@gmail.com> <52B74F71.1070801@gmail.com>
 <CAHsXEHWUdb9GGBiy_10_u7qKqr7QskiRdNmL80d0krWPpieseQ@mail.gmail.com>
 <52B75546.6030705@gmail.com> <52B7A465.4030306@elyograg.org>
 <52B810E7.2050409@gmail.com>
 <CAG6_zk01Qea8ef1vBm6xtSNBPQN6P9yQsZfoe9q=v5ePA0rvrA@mail.gmail.com>
 <52B8B659.90101@gmail.com>
From: Greg Preston <gpreston@marinsoftware.com>
Date: Mon, 23 Dec 2013 14:43:16 -0800
Message-ID: 
 <CAG6_zk3O42Kvo6myeZAGFX92cxB_j9PeesV+7YSrnUAkj6Zcxw@mail.gmail.com>
Subject: Re: adding a node to SolrCloud
To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I believe you can just define multiple cores:

<core default="true" instanceDir="shard1/"
name="collectionName_shard1" shard="shard1"/>
<core default="true" instanceDir="shard2/"
name="collectionName_shard2" shard="shard2"/>
...

(this is the old style solr.xml.  I don't know how to do it in the newer style)

Also, make sure you don't define a non-relative <dataDir> in
solrconfig.xml, or you may run into issues with cores trying to use
the same data dir.






-Greg


On Mon, Dec 23, 2013 at 2:16 PM, David Santamauro
<david.santamauro@gmail.com> wrote:
> On 12/23/2013 05:03 PM, Greg Preston wrote:
>>>
>>> Yes, I'm well aware of the performance implications, many of which are
>>> mitigated by 2TB of SSD and 512GB RAM
>>
>>
>> I've got a very similar setup in production.  2TB SSD, 256G RAM (128G
>> heaps), and 1 - 1.5 TB of index per node.  We're in the process of
>> splitting that to multiple JVMs per host.  GC pauses were causing ZK
>> timeouts (you should up that in solr.xml).  And resync's after the
>> timeouts took long enough that a large tlog built up (we have near
>> continuous indexing), and we couldn't replay the tlog fast enough to
>> catch up to current.
>
>
> GC pauses are a huge issue in our current production environment (monolithic
> index) and general performance was meager, hence the move to a distributed
> design. We will have 8 nodes with ~ 200GB per node, one shard each and
> performance for single and most multi-term queries has become sub-second and
> throughput has increased 10-fold. Larger boolean queries can still take 2-3s
> but we can live with that.
>
> At any rate, I still can't figure out what my solr.xml is supposed to look
> like on the node with all 8 redundant shards.
>
> David
>
>
>
>> On Mon, Dec 23, 2013 at 2:31 AM, David Santamauro
>> <david.santamauro@gmail.com> wrote:
>>>
>>> On 12/22/2013 09:48 PM, Shawn Heisey wrote:
>>>>
>>>>
>>>> On 12/22/2013 2:10 PM, David Santamauro wrote:
>>>>>
>>>>>
>>>>> My goal is to have a redundant copy of all 8 currently running, but
>>>>> non-redundant shards. This setup (8 nodes with no replicas) was a test
>>>>> and it has proven quite functional from a performance perspective.
>>>>> Loading, though, takes almost 3 weeks so I'm really not in a position
>>>>> to
>>>>> redesign the distribution, though I can add nodes.
>>>>>
>>>>> I have acquired another resource, a very large machine that I'd like to
>>>>> use to hold the replicas of the currently deployed 8-nodes.
>>>>>
>>>>> I realize I can run 8 jetty/tomcats and accomplish my goal but that is
>>>>> a
>>>>> maintenance headache and is really a last resort. I really would just
>>>>> like to be able to deploy this big machine with 'numShards=8'.
>>>>>
>>>>> Is that possible or do I really need to have 8 other nodes running?
>>>>
>>>>
>>>>
>>>> You don't want to run more than one container or Solr instance per
>>>> machine.  Things can get very confused, and it's too much overhead.
>>>
>>>
>>>>
>>>>
>>>> With existing collections, you can simply run the CoreAdmin CREATE
>>>>
>>>> action on the new node with more resources.
>>>>
>>>> So you'd do something like this, once for each of the 8 existing parts:
>>>>
>>>>
>>>>
>>>> http://newnode:port/solr/admin/cores?action=CREATE&name=collname_shard1_replica2&collection=collname&shard=shard1
>>>>
>>>> It will automatically replicate the shard from its current leader.
>>>
>>>
>>>
>>> Fantastic! Clearly my understanding of "collection", vs "core" vs "shard"
>>> was lacking but now I see the relationship better.
>>>
>>>
>>>>
>>>> One thing to be aware of: With 1.4TB of index data, it might be
>>>> impossible to keep enough of the index in RAM for good performance,
>>>> unless the machine has a terabyte or more of RAM.
>>>
>>>
>>>
>>> Yes, I'm well aware of the performance implications, many of which are
>>> mitigated by 2TB of SSD and 512GB RAM.
>>>
>>> Thanks for the nudge in the right direction. The first node/shard1 is
>>> replicating right now.
>>>
>>> David
>>>
>>>
>>>
>