lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien DUDOGNON <damien.dudog...@ebuzzing.com>
Subject Re: Replication lag after cache optimizations
Date Sun, 02 Sep 2012 08:57:07 GMT
Thanks for your answer Erick.

For the polling interval, we use 1 second for the small index and 1 minute
for the big one. I'll try to increase it up to 5 minutes and see if it
solve the problem.The issue doesn't occur with the default cache settings
(i.e. cache size=512).

Indeed, we strive for near real time searching, so I'll take a look at 4.0
with NRT support, hoping it is sufficiently stable for production
environment.

Kind regards,
Damien

2012/9/2 Erick Erickson <erickerickson@gmail.com>

> You polling interval is much too short. 1 second
> is probably getting you into resource contention
> issues.
>
> A more reasonable interval is on the order of several
> minutes. If you really need near real time
> searching, consider 4.0 which supports NRT
>
> Best
> Erick
>
> On Fri, Aug 31, 2012 at 10:02 AM, Damien Dudognon
> <damien.dudognon@ebuzzing.com> wrote:
> > Hi,
> >
> > We get some troubles with the solr replication after cache
> optimizations. We use a lot the facet features.
> >
> > We have increased the cache size and its initial size. We have also
> change the queryResultCache from LRU to FastLRU and the fieldValueCache was
> activated (see below the detailed configuration). This optimization step
> allows to  divide the average time per request by 50 (from 260ms to 5ms).
> >
> > However, with these modifications we noticed an important replication
> lag. This issue is not troublesome for smaller indexes (about 300.000
> elements - 3Gb), but it becomes critical if the index size is significant
> (30 million elements - 70Gb). In fact, the slaves can't make up for lag and
> they become out-of-date (and consequently unusable).
> >
> > Has anyone ever been faced this kind of problems?
> >
> > Our environment :
> > - Solr 3.4.0
> > - Java 1.6.0_26
> > - Debian 6.0.3
> >
> > Best regards,
> > Damien
> >
> > ------------------------------------------------------------------
> > My previous cache settings (fieldValueCache was disabled):
> > ------------------------------------------------------------------
> > <filterCache class="solr.FastLRUCache" size="512" initialSize="512"
> autowarmCount="0" />
> > <queryResultCache class="solr.LRUCache" size="512" initialSize="512"
> autowarmCount="0" />
> > <documentCache class="solr.FastLRUCache" size="512" initialSize="512"
> autowarmCount="0" />
> > <!--
> > <fieldValueCache class="solr.FastLRUCache" size="512"
> autowarmCount="128" showItems="32" />
> > -->
> >
> > ------------------------------------------------------------------
> > The settings now used:
> > ------------------------------------------------------------------
> > <filterCache class="solr.FastLRUCache" size="16384" initialSize="4096"
> autowarmCount="0" />
> > <queryResultCache class="solr.FastLRUCache" size="16384"
> initialSize="4096" autowarmCount="0" />
> > <documentCache class="solr.FastLRUCache" size="16384" initialSize="4096"
> autowarmCount="0" />
> > <fieldValueCache class="solr.FastLRUCache" size="16384"
> autowarmCount="1024" showItems="32" />
> >
> > ------------------------------------------------------------------
> > The replication config:
> > ------------------------------------------------------------------
> >   <requestHandler name="/replication" class="solr.ReplicationHandler" >
> >     <lst name="master">
> >       <str name="enable">${solr.enable.master:false}</str>
> >       <str name="replicateAfter">commit</str>
> >       <str name="replicateAfter">startup</str>
> >       <str name="confFiles">schema.xml,stopwords.txt</str>
> >     </lst>
> >     <lst name="slave">
> >       <str name="enable">${solr.enable.slave:false}</str>
> >       <str name="masterUrl">http://solrmaster:
> ${jetty.port:8083}/solr/en/replication</str>
> >       <str name="pollInterval">00:00:01</str>
> >     </lst>
> >   </requestHandler>
>



-- 

Damien DUDOGNON

R&D Engineer - PhD Student

   - +33 (0)6 62 79 34 10
   - damien.dudognon@ebuzzing.com
   -
      -
   - linkedin.com/in/damiendudognon<http://www.linkedin.com/in/damiendudognon>
   - www.irit.fr/~Damien.Dudognon


   - 1, avenue Jean Rieux
   - 31500 Toulouse
   - France
   - +33 (0)5 62 48 33 90
   -

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message