lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Rangarajan <arunrangara...@gmail.com>
Subject Re: Date query not returning results only some time
Date Sun, 02 Mar 2014 19:25:39 GMT
Erick,
Thanks a lot for the detailed explanation. That clarified things for me
better.


On Sun, Mar 2, 2014 at 10:04 AM, Erick Erickson <erickerickson@gmail.com>wrote:

> Well, in M/S setups the master shouldn't be searching at all,
> but that's a nit.
>
> That aside, whether the master has opened a new or
> searcher or not is irrelevant to what the slave replicates.
> What _is_ relevant is whether any of the files on disk that
> comprise the index (i.e. the segment files) have been
> changed. Really, if any of them have been closed/merged
> whatever since the last sync. Imagine it like this (this isn't
> quite what happens, but it's a useful model). The slave
> says "here's a list of my segments, is it the same as the
> list of closed segments on the master?" If the answer
> is no, a replication is performed. Actually, this is done
> much more efficiently, but that's the idea.
>
> You seem to be really asking about the whole issue of whether
> searches on the various nodes (master + slaves) is
> consistent. This is one of the problems with M/S setups, they
> can be different by whatever has happened in the polling interval.
>
> The state of the master's searchers just doesn't enter the picture.
>
> Glad the problem is solved no matter what.
>
> Erick
>
> On Sat, Mar 1, 2014 at 10:26 PM, Arun Rangarajan
> <arunrangarajan@gmail.com> wrote:
> >> The slave is polling the master after the interval specified in
> > solrconfig.xml. The slave essentially asks "has anything changed?" If
> so, the
> > changes are brought down to the slave.
> > Yes, I understand this, but if master does not open a new searcher after
> > auto commits (which would indicate that the new index is not quite ready
> > yet) and if master is still using the old index to serve search
> requests, I
> > would expect the slave to do the same as well. Or the slave should at
> least
> > not replicate or not open a new searcher, until the master opened a new
> > searcher. But that is just the way I see it and it may be wrong.
> >
> >> What's your polling interval on the slave anyway? Sounds like it's quite
> > frequent if you notice this immediately after the DIH starts.
> > No, polling interval is set to 1 hour, but the full import was set to run
> > at 1 AM. I believe a delete followed by few docs got replicated after the
> > first few auto commits when the slave probably polled around 1:10 AM and
> > slave index had few docs for an hour before the next polling happened,
> > which is why the date query was returning empty results for exactly that
> > one hour. (The full index takes about 1.5 hours to finish.)
> >
> > Anyway the problem is now solved by specifying "clean=false" in the DIH
> > full import command.
> >
> >
> > On Sat, Mar 1, 2014 at 9:12 AM, Erick Erickson <erickerickson@gmail.com
> >wrote:
> >
> >> bq: the slave anyway replicates the index after auto commits! (Is this
> >> desired behavior?)
> >>
> >> Absolutely it's desired behavior. The slave is polling the master
> >> after the interval
> >> specified in solrconfig.xml. The slave essentially asks "has anything
> >> changed?" If so,
> >> the changes are brought down to the slave. And by definition, commits
> >> change the index,
> >> especially if all docs have been deleted....
> >>
> >> What's your polling interval on the slave anyway? Sounds like it's
> >> quite frequent if you
> >> notice this immediately after the DIH starts.
> >>
> >> Best,
> >> Erick
> >>
> >> On Fri, Feb 28, 2014 at 9:04 PM, Arun Rangarajan
> >> <arunrangarajan@gmail.com> wrote:
> >> > I believe I figured out what the issue is. Even though we do not open
> a
> >> new
> >> > searcher on master during full import, the slave anyway replicates the
> >> > index after auto commits! (Is this desired behavior?) Since
> "clean=true"
> >> > this meant all the docs were deleted on slave and a partial index got
> >> > replicated! The reason only the date query did not return any results
> is
> >> > because recently created docs have higher doc IDs and we index by
> >> ascending
> >> > order of IDs!
> >> >
> >> > I believe I have two options:
> >> > - as Chris suggested I have to use "clean=false" so the existing docs
> are
> >> > not deleted first on the slave. Since we have primary keys, newly
> added
> >> > docs will overwrite old docs as they get added.
> >> > - disable replication after commits. Replicate only after optimize.
> >> >
> >> > Thx all for your help.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Feb 28, 2014 at 8:06 PM, Arun Rangarajan
> >> > <arunrangarajan@gmail.com>wrote:
> >> >
> >> >> Thx, Erick and Chris.
> >> >>
> >> >> This is indeed very strange. Other queries which do not restrict by
> the
> >> >> date field are returning results, so the index is definitely not
> empty.
> >> Has
> >> >> it got something to do with the date query part, with NOW/DAY or
> >> something
> >> >> in here?
> >> >> first_publish_date:[NOW/DAY-33DAYS TO NOW/DAY-3DAYS]
> >> >>
> >> >> For now, I have set up a script to just log the number of docs on the
> >> >> slave every minute. Will monitor and report the findings.
> >> >>
> >> >>
> >> >> On Fri, Feb 28, 2014 at 6:49 PM, Chris Hostetter <
> >> hossman_lucene@fucit.org
> >> >> > wrote:
> >> >>
> >> >>>
> >> >>> : This is odd. The full import, I think, deletes the
> >> >>> : docs in the index when it starts.
> >> >>>
> >> >>> Yeah, if you are doing a full-import everyday, and you don't want
> it to
> >> >>> delete all docs when it starts, you need to specify "clearn=false"
> >> >>>
> >> >>>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler#UploadingStructuredDataStoreDatawiththeDataImportHandler-Parametersforthefull-importCommand
> >> >>>
> >> >>>
> >> >>>
> >> >>> -Hoss
> >> >>> http://www.lucidworks.com/
> >> >>>
> >> >>
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message