Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of solr@elyograg.org designates
 166.70.79.219 as permitted sender)
Message-ID: <53569C15.5090501@elyograg.org>
Date: Tue, 22 Apr 2014 10:43:01 -0600
From: Shawn Heisey <solr@elyograg.org>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: maximum number of shards per SolrCloud
References: 
 <CAPX-pyMTijWxUVv31daftKt4MLJHtjvrmEdJPfya_+O+3CgOGw@mail.gmail.com>
 <CAN4YXvfAYkL=5Rg235THqQMGSeURu4xm100D0ntOUGGD0yMX-w@mail.gmail.com>
 <89280C92-3782-450C-B8DF-B35D5ED0DA80@gmail.com>
In-Reply-To: <89280C92-3782-450C-B8DF-B35D5ED0DA80@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 4/22/2014 10:02 AM, yypvsxf19870706 wrote:
>      I am curious of the influences when have more than 2G docs in a core.And we plan to have  5g docs/core.
>
>     Please give me some suggestion about how to plan num of docs in a core ?

One Solr core contains one Lucene index.  It can't be divided further 
than that without a significant redesign.  Quick note: Although 
SolrCloud can handle five billion documents with no problem, you can't 
have five billion documents in a single shard/core.

The only hard limitation in the entire system is that you can't have 
more than approximately 2 billion documents in a single Lucene index.  
This is because a Java integer (which is a signed 32-bit number) is what 
gets used for internal Lucene document identifiers. Deleted documents 
count against that limit.  It is theoretically possible to overcome this 
limitation, but it would be a MAJOR change to Lucene, requiring major 
changes in Solr as well.

The other limitations you can run into with a large SolrCloud are mostly 
a matter of configuration, system resources, and scaling to multiple 
servers.  They are not hard limitations in the software.

I would never put more than about 1 billion documents in a single core.  
For performance reasons, it would be a good idea to never exceed a few 
hundred million.  When a high query rate is required, loading only one 
Solr core per server may be a requirement.

Thanks,
Shawn