Mailing-List: contact dev-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@zookeeper.apache.org
Date: Sun, 13 Aug 2017 05:20:01 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: dev@zookeeper.apache.org
Message-ID: <JIRA.13093873.1502388449000.20066.1502601601668@Atlassian.JIRA>
In-Reply-To: <JIRA.13093873.1502388449000@Atlassian.JIRA>
References: <JIRA.13093873.1502388449000@Atlassian.JIRA> <JIRA.13093873.1502388449955@jira-lw-us.apache.org>
Subject: [jira] [Commented] (ZOOKEEPER-2872) Interrupted snapshot sync
 causes data loss
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 13 Aug 2017 05:20:10 -0000


    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124803#comment-16124803 ] 

ASF GitHub Bot commented on ZOOKEEPER-2872:
-------------------------------------------

Github user hanm commented on a diff in the pull request:

    https://github.com/apache/zookeeper/pull/333#discussion_r132832594
  
    --- Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java ---
    @@ -364,6 +364,7 @@ protected void syncWithLeader(long newLeaderZxid) throws Exception{
             readPacket(qp);
             LinkedList<Long> packetsCommitted = new LinkedList<Long>();
             LinkedList<PacketInFlight> packetsNotCommitted = new LinkedList<PacketInFlight>();
    +        boolean syncSnapshot = false;
    --- End diff --
    
    We can level this variable definition up so it's clustered with `snapshotNeed` boolean.
    
    Another possibility is to get ride of this variable and use existing `snapshotNeeded` - but that will do fysnc snapshot for TRUNC sync, which the existing patch will not. 


> Interrupted snapshot sync causes data loss
> ------------------------------------------
>
>                 Key: ZOOKEEPER-2872
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2872
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.4.10, 3.5.3, 3.6.0
>            Reporter: Brian Nixon
>
> There is a way for observers to permanently lose data from their local data tree while remaining members of good standing with the ensemble and continuing to serve client traffic when the following chain of events occurs.
> 1. The observer dies in epoch N from machine failure.
> 2. The observer comes back up in epoch N+1 and requests a snapshot sync to catch up.
> 3. The machine powers off before the snapshot is synced to disc and after some txn's have been logged (depending on the OS, this can happen!).
> 4. The observer comes back a second time and replays its most recent snapshot (epoch <= N) as well as the txn logs (epoch N+1). 
> 5. A diff sync is requested from the leader and the observer broadcasts availability.
> In this scenario, any commits from epoch N that the observer did not receive before it died the first time will never be exposed to the observer and no part of the ensemble will complain. 
> This situation is not unique to observers and can happen to any learner. As a simple fix, fsync-ing the snapshots received from the leader will avoid the case of missing snapshots causing data loss.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)