Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of greg.pendlebury@gmail.com
 designates 74.125.82.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <alpine.DEB.2.02.1403061657000.7827@frisbee>
References: 
 <CA+A4HB1xRDVtYSMAT55E51X89F+7qn+MQqdo4_7An7tLb2bSKw@mail.gmail.com>
	<alpine.DEB.2.02.1403061657000.7827@frisbee>
Date: Fri, 7 Mar 2014 12:19:21 +1100
Message-ID: 
 <CA+A4HB1zBG1qOcbGPjFMgvjZ1pr9ORyQU9A675eS-xcq1dOWMg@mail.gmail.com>
Subject: Re: Solr 4.7.0 - cursorMark question
From: Greg Pendlebury <greg.pendlebury@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=089e014940782f659804f3fa0ba3

--089e014940782f659804f3fa0ba3
Content-Type: text/plain; charset=ISO-8859-1

Thank-you, that all sounds great. My assumption about documents being
missed was something like this:

A,B,C,D

where they are sorted by timestamp first and ID second. Say the first
'page' of results is 'A,B', and before the second page is requested both
documents B + C receive update events and the new order (by timestamp) is:

A,D,B,C

In that situation D would always be missed, whether the cursorMark 'C or
greater' or 'greater than B' (I'm not sure which it is in practice), simply
because the cursorMark is the unique ID and the unique ID is not your first
sort mechanism.

However, I'm not really concerned about that anyway since it is not a use
case we consider important, and in an information science sense of things I
think it is a non-trivial problem to solve without brute force caching of
all result sets. I'm just happy that we don't have to get our users to
replace existing sort options; we just need to add a unique ID field at the
end and change the parameters we send into the cluster.

Thanks,
Greg


On 7 March 2014 11:05, Chris Hostetter <hossman_lucene@fucit.org> wrote:

>
> : At the end of the linked doco there is an example that doesn't make sense
> : to me, because it mentions "sort=timestamp asc" and is then followed by
> : pseudo code that sorts by id only. I understand that cursorMark requires
>
> Ok ... 2 things contributing to the confusion.
>
> 1) the para that refers to "sort=timestamp asc" should be fixed to include
> "id" as well.
>
> 2) psuedo-code you're refering to that uses "sort => 'id asc'" isn't ment
> to give an example of specifically tailing by timestamp -- it's an
> extension on the earlier example (of fetching all docs sorting on id) to
> show "tailing" new docs with new (increasing) ids ... i'll try to fix the
> wording to better elborate
>
> : that "sort clauses must include the uniqueKey field", but is it really
> just
> : 'include', or is it the only field that sort can be performed on?
> :
> : ie. can sort be specified as 'sort=timestamp asc, id asc'?
>
> That will absolutely work ... i'll update the doc to include more examples
> with multi-clause sort criteria.
>
> : I am assuming that if the index is changed between requests than we can
> : still 'miss' or duplicate documents by not sorting on the id as the only
> : sort parameter, but I can live with that scenario. cursorMark is still
>
> If you are using a timestamp param, you should never "miss" a document
> (assuming every doc gets a timestamp) but yes: you can absolutely get the
> same doc twice if it's updated after the first time you fetch it -- that's
> one of the advantages of sorting on a timestamp field like that.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

--089e014940782f659804f3fa0ba3--