lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-3816) Need a more granular nrt system that is close to a realtime system.
Date Sun, 11 Nov 2012 01:07:13 GMT

    [ https://issues.apache.org/jira/browse/SOLR-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494815#comment-13494815
] 

Yonik Seeley commented on SOLR-3816:
------------------------------------

So I took a quick look at the patch, and it's as I feared from earlier discussions - the reader
is changed out from under the searcher at random times.  This approach simply won't work.
 Lucene and Solr have searchers that work on a non-changing reader.

You'll get incorrect documents back, incorrect facets back, pretty much any number of random
looking bugs because internal docids will be changing underneath you.  One only has to look
at this snippet of the patch as an example of an attempt to be defensive about these changing
docids:

{code}
+      /* realtime NRT changes */
+      // We may be getting docs that are beyond maxDoc, ignore for this request. 
+      if (doc < maxDoc) {
+        bits.fastSet(doc);
+      }
{code}

So how is it that tests can pass?  Well, the vast majority of our tests (like querying, faceting,
etc), index documents and then test the requests.  They do not currently test requests while
concurrently indexing (and such tests would be much more difficult to write of course... one
would need to know exactly what documents made it into the index in order to know what the
correct results should be.

I'm sorry folks, but this really looks like a dead end.

                
> Need a more granular nrt system that is close to a realtime system.
> -------------------------------------------------------------------
>
>                 Key: SOLR-3816
>                 URL: https://issues.apache.org/jira/browse/SOLR-3816
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, replication (java), search, SearchComponents - other,
SolrCloud, update
>    Affects Versions: 4.0
>            Reporter: Nagendra Nagarajayya
>              Labels: nrt, realtime, replication, search, solrcloud, update
>         Attachments: alltests_passed_with_realtime_turnedoff.log, SOLR-3816_4.0_branch.patch,
SOLR-3816-4.x.trunk.patch, solr-3816-realtime_nrt.patch
>
>
> Need a more granular NRT system that is close to a realtime system. A realtime system
should be able to reflect changes to the index as and when docs are added/updated to the index.
soft-commit offers NRT and is more realtime friendly than hard commit but is limited by the
dependency on the SolrIndexSearcher being closed and reopened and offers a coarse granular
NRT. Closing and reopening of the SolrIndexSearcher may impact performance also.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message