Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of adhir@symplicity.com
 designates 66.151.109.78 as permitted sender)
Message-Id: <F593C9B6-9C90-49CE-B125-0A3DF06571A5@symplicity.com>
From: Alok Dhir <adhir@symplicity.com>
To: solr-user@lucene.apache.org
In-Reply-To: <C06E68F9-01C5-44FA-9820-CF98A9E79F7D@symplicity.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: SOLR Performance
Date: Mon, 3 Nov 2008 16:47:27 -0500
References: <C06E68F9-01C5-44FA-9820-CF98A9E79F7D@symplicity.com>

We've moved past this issue by reducing date precision -- thanks to  
all for the help.  Now we're at another problem.

There is relatively constant updating of the index -- new log entries  
are pumped in from several applications continuously.  Obviously, new  
entries do not appear in searches until after a commit occurs.

The problem is, issuing a commit causes searches to come to a  
screeching halt for up to 2 minutes.  We're up to around 80M docs.   
Index size is 27G.  The number of docs will soon be 800M, which  
doesn't bode well for these "pauses" in search performance.

I'd appreciate any suggestions.

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:

> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>
> Fairly simple schema -- no large text fields, standard request  
> handler.  4 small facet fields.
>
> The index is an event log -- a primary search/retrieval requirement  
> is date range queries.
>
> A simple query without a date range subquery is ridiculously fast -  
> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>
> Concrete example, this query just look 18s:
>
> 	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO  
> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>
> The exact same query without the date range took 2ms.
>
> I saw a thread from Apr 2008 which explains the problem being due to  
> too much precision on the DateField type, and the range expansion  
> leading to far too many elements being checked.  Proposed solution  
> appears to be a hack where you index date fields as strings and  
> hacking together date functions to generate proper queries/format  
> results.
>
> Does this remain the recommended solution to this issue?
>
> Thanks
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>