From dev-return-359445-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Mon Jun 17 22:39:05 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id C0A0718062B for ; Tue, 18 Jun 2019 00:39:04 +0200 (CEST) Received: (qmail 2221 invoked by uid 500); 17 Jun 2019 22:39:02 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 2211 invoked by uid 99); 17 Jun 2019 22:39:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jun 2019 22:39:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B4A53E2CFA for ; Mon, 17 Jun 2019 22:39:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id DCDE624637 for ; Mon, 17 Jun 2019 22:39:00 +0000 (UTC) Date: Mon, 17 Jun 2019 22:39:00 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: dev@lucene.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866041#comment-16866041 ] ASF subversion and git services commented on SOLR-13490: -------------------------------------------------------- Commit 592d10d7ce82c9d0f38ea66f3c2f1a4113649728 in lucene-solr's branch refs/heads/branch_8x from Chris M. Hostetter [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=592d10d ] SOLR-13490: fix TestWaitForStateWithJettyShutdowns to use correct (randomized) JettyConfig (cherry picked from commit 7eb8703df64b4fdda8113ddcbcd0b4d2413ecc38) > waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition > ----------------------------------------------------------------------------------------------------------- > > Key: SOLR-13490 > URL: https://issues.apache.org/jira/browse/SOLR-13490 > Project: Solr > Issue Type: Bug > Reporter: Hoss Man > Assignee: Hoss Man > Priority: Major > Fix For: master (9.0), 8.2 > > Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch > > > I was investigating some failures in {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the hunch that {{waitForState}} wasn't ensuring that the predicates registered would always be called if/when a node was shutdown. > Digging into it a bit more, I found that the root cause seems to be the way the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} are called, they get the "fresh" {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}} > Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas hosted on any of changed nodes. > Since there is no garunteed order that Watchers will be triggered, this means there is a race condition where the following can happen... > * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}} > * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if "replicaX" of collectionA is on a "down" node > * client2 causes shutdown of node N1 which is hosting replicaX > * client1's zkStateReader gets a WatchedEvent for state.json of collectionA > ** DocCollection for collectionA is rebuilt > ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new DocCollection > *** watcherZ sees that replicaX is on N1, but thinks N1 is live > *** watcherZ says "everything ok, not the event i was waiting for" and doesn't take any action > * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE > ** zkStateReader.liveNodes is rebuilt > ...at no point in this sequence (or after this) will watcherZ be notified fired with the updated liveNodes (unless/until another {{state.json}} change is made for collectionA. > ---- > While this is definitely be problematic in _tests_ that deal with node lifecyle and use things like {{SolrCloudTestCase.waitForState(..., SolrCloudTestCase.clusterShape(...))}} to check for the expected shards/replicas, a cursory search of how/where {{ZkStateReader.waitForState(...)}} and {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core suggests that could also lead to bad behavior in situations like reacting to shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by the watcher/predicate) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org