lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lucene/Solr QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition
Date Fri, 14 Jun 2019 11:46:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16863964#comment-16863964
] 

Lucene/Solr QA commented on SOLR-13490:
---------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 5 new or modified test files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 34s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 10s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 10s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green}  3m 44s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green}  3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green}
 3m 10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m  9s{color} | {color:red}
core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 37s{color} | {color:green}
solrj in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 25s{color} | {color:green}
test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 44s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | solr.cloud.AliasIntegrationTest |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13490 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12971740/SOLR-13490.patch
|
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  validatesourcepatterns
 |
| uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36
UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 81e8b38 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
| unit | https://builds.apache.org/job/PreCommit-SOLR-Build/435/artifact/out/patch-unit-solr_core.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-SOLR-Build/435/testReport/ |
| modules | C: solr/core solr/solrj solr/test-framework U: solr |
| Console output | https://builds.apache.org/job/PreCommit-SOLR-Build/435/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk)
Watcher race condition
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-13490
>                 URL: https://issues.apache.org/jira/browse/SOLR-13490
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Hoss Man
>            Assignee: Hoss Man
>            Priority: Major
>         Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch
>
>
> I was investigating some failures in {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}}
which lead me to the hunch that {{waitForState}} wasn't ensuring that the predicates registered
would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way the {{CollectionStateWatcher}}
/ {{CollectionStatePredicate}} APIs pass in *both* the {{DocCollection}}, and the "current"
{{liveNodes}} - but are only _triggered_ by the {{StateWatcher}} on the {{state.json}} (which
is used to rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / {{CollectionStatePredicate}}
are called, they get the "fresh" {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only updates {{ZkStateReader.liveNodes}}
and triggers any {{LiveNodesListener}} - it does *NOT* invoke any {{CollectionStateWatcher}}
that may have replicas hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means there is
a race condition where the following can happen...
>  * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
>  * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if "replicaX"
of collectionA is on a "down" node
>  * client2 causes shutdown of node N1 which is hosting replicaX
>  * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
>  ** DocCollection for collectionA is rebuilt
>  ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new DocCollection
>  *** watcherZ sees that replicaX is on N1, but thinks N1 is live
>  *** watcherZ says "everything ok, not the event i was waiting for" and doesn't take
any action
>  * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
>  ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified fired with
the updated liveNodes (unless/until another {{state.json}} change is made for collectionA.
> ----
> While this is definitely be problematic in _tests_ that deal with node lifecyle and use
things like {{SolrCloudTestCase.waitForState(..., SolrCloudTestCase.clusterShape(...))}} to
check for the expected shards/replicas, a cursory search of how/where {{ZkStateReader.waitForState(...)}}
and {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core suggests that
could also lead to bad behavior in situations like reacting to shard leader loss, waiting
for all leaders of SYSTEM_COLL to come online for upgrade, running PrepRecoveryOp, etc...
(anywhere that liveNodes is used by the watcher/predicate)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message