lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan McKinley <ryan...@gmail.com>
Subject Re: fast update handlers
Date Thu, 10 May 2007 17:06:56 GMT

I don't know if this helps, but...

Do *all* your queries need to include the fast updates?  I have a setup 
where there are some cases that need the newest stuff but most cases can 
wait 5 mins (or so)

In that case, I have two solr instances pointing to the same index 
files.  One is used for updates and queries that need everything.  The 
other is a read-only index that serves the majority of queries.

What is nice about this is that you can set different cache sizes and 
auto-warming for the different cases.

ryan


Will Johnson wrote:
> The problem is I want the newly added documents to be made searchable
> every 1-2 seconds so I need the commits.  I was hoping that the caches
> could be stored/tied to the IndexSearcher then a MultiSearcher could
> take advantage of the multiple sub indexes and their respective caches.
> 
> 
> I think the best approach now will be to write a top level federator
> that can merge the large ~static index and the smaller more dynamic
> index.
> 
> - will
> 
> 
> 
> -----Original Message-----
> From: Charlie Jackson [mailto:Charlie.Jackson@cision.com] 
> Sent: Thursday, May 10, 2007 10:53 AM
> To: solr-user@lucene.apache.org
> Subject: RE: fast update handlers
> 
> What about issuing separate commits to the index on a regularly
> scheduled basis? For example, you add documents to the index every 2
> seconds, or however often, but these operations don't commit. Instead,
> you have a cron'd script or something that just issues a commit every 5
> or 10 minutes or whatever interval you'd like. 
> 
> I had to do something similar when I was running a re-index of my entire
> dataset. My program wasn't issuing commits, so I just cron'd a commit
> for every half hour so it didn't overload the server. 
> 
> Thanks,
> Charlie
> 
> 
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Thursday, May 10, 2007 9:07 AM
> To: solr-user@lucene.apache.org
> Subject: Re: fast update handlers
> 
> On 5/10/07, Will Johnson <wjohnson@getconnected.com> wrote:
>> I guess I was more concerned with doing the frequent commits and how
>> that would affect the caches.  Say I have 2M docs in my main index but
> I
>> want to add docs every 2 seconds all while doing queries.  if I do
>> commits every 2 seconds I basically loose any caching advantage and my
>> faceting performance goes down the tube.  If however, I were to add
>> things to a smaller index and then roll it into the larger one every
> ~30
>> minutes then I only take the hit on computing the larger filters
> caches
>> on that interval.  Further, if my smaller index were based on a
>> RAMDirectory instead of a FSDirectory I assume computing the filter
> sets
>> for the smaller index should be fast enough even every 2 seconds.
> 
> There isn't currently any support for incrementally updating filters.
> 
> -Yonik
> 


Mime
View raw message