From user-return-10535-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org Thu Oct 20 17:22:20 2016 Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4DC1A19850 for ; Thu, 20 Oct 2016 17:22:20 +0000 (UTC) Received: (qmail 62698 invoked by uid 500); 20 Oct 2016 17:22:19 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 62643 invoked by uid 500); 20 Oct 2016 17:22:19 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 62631 invoked by uid 99); 20 Oct 2016 17:22:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2016 17:22:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C9CD9C17EB for ; Thu, 20 Oct 2016 17:22:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.498 X-Spam-Level: ** X-Spam-Status: No, score=2.498 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id sFEionCEA7hB for ; Thu, 20 Oct 2016 17:22:16 +0000 (UTC) Received: from mail-vk0-f43.google.com (mail-vk0-f43.google.com [209.85.213.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B03AC5F2C3 for ; Thu, 20 Oct 2016 17:22:16 +0000 (UTC) Received: by mail-vk0-f43.google.com with SMTP id 2so82670751vkb.3 for ; Thu, 20 Oct 2016 10:22:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=zRtO1PzjoyqJu9lgUC3tCZcQ0uaTIHrix9Vc4o47ZsE=; b=nBTMgdKqMVb1XUcyhH7I6AzJ2dZxJhl6h5QEj6jvmjNeY4DMDiLYrhNA/OBG4gzknf tssaDVrt2NbZtnhS5w1vndQMYV2dnr393GLEMuTqdzrZD5HZhHy61Z37XoEKAuecAg3d xTBfdFbXDW1IbNpHZRMNDuQmDZsOMRr/HItts8BjpQiOQzcfr5ZmY8EOyYPIQSonb9hI ZazH8NqlJdmq9FAdjJzhYNsnq8owFvpO+vBrK9BYHM/QeICwjzRiucLq72RxcSBjdd0J tqF1sFz5f1Xp62xtDXo2cuyG4o/sCoQDPzQxo31NaCrmI8w8+XDt6Pts7ekGKpIGQx5d dJOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=zRtO1PzjoyqJu9lgUC3tCZcQ0uaTIHrix9Vc4o47ZsE=; b=kxs846KlrIr+Q8jsUpWu0Wx6TRW+W1u73p8JcWJh+nhHsD1aCkCvGlNvkrPgqvl3IY tA6ldnm3mjb0ixfnK3lxQ4WD1BRhdXGLp++9sxZRvRpHjnCKx8NnGNdGEfPHuDUGIUrc 9h+HAKyogYUBQ0QtXQlTYh4Ov8YKOt8YQEwgu4nLI7Ly/2OjiAdJfb1YYm1ZkdeHmxT5 ftxt+lFnSHXAoQAmjN90jJl99CuqNEmvvFYY+VRPCVb33ndRThj3RRf9404HfgOK9QuX uR6TDRit8ACSRnAt4af1kvu2bDrphhd8l4oyZ3m9Tto3/tv4xfgEuIWYAy/kIndIFNr/ CauQ== X-Gm-Message-State: AA6/9Rm9lFXuje5E9nOKtWguRyVKGm/ufAj7kMAuOpLTgjxo4Y/Z2ZTJZzBR3symf+bTN8iK5GuqSR04VORFAWhj X-Received: by 10.31.152.195 with SMTP id a186mr2564555vke.79.1476983834544; Thu, 20 Oct 2016 10:17:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.69.226 with HTTP; Thu, 20 Oct 2016 10:16:44 -0700 (PDT) In-Reply-To: References: From: Michael Han Date: Thu, 20 Oct 2016 10:16:44 -0700 Message-ID: Subject: Re: Adding and removing replicas? To: UserZooKeeper , steve@scalyr.com Cc: Rakesh Radhakrishnan Content-Type: multipart/alternative; boundary=001a1141bf42cc394d053f4f1803 --001a1141bf42cc394d053f4f1803 Content-Type: text/plain; charset=UTF-8 Hi Steve, I don't think there are official ZooKeeper documentation regarding best practices for adding / removing replicas prior to 3.5.0 or how to do a rolling restart. The community is working towards 3.5.3 release which will remove the alpha tag. So 3.5.3 might be good for you if you want a stable release with dynamic reconfig feature. It's unclear to me about the timeframe of the release though. On Thu, Oct 20, 2016 at 9:26 AM, Steve Newman wrote: > Thanks for the pointer. This looks like a nice improvement. > > If I'm reading http://zookeeper.apache.org/releases.html correctly, this > feature is only available in alpha release. In the near term, I need a > procedure I can follow for a stable release. Is there any documentation > regarding best practices for adding / removing replicas prior to 3.5.0? The > 3.5.2 documentation you linked to is somewhat alarming regarding prior > releases: > > "Prior to the 3.5.0 release, the membership and all other configuration > parameters of Zookeeper were static - loaded during boot and immutable at > runtime. Operators resorted to ''rolling restarts'' - a manually intensive > and error-prone method of changing the configuration *that has caused data > loss and inconsistency in production*." > > Thanks, > Steve > > On Thu, Oct 20, 2016 at 8:03 AM, Rakesh Radhakrishnan > wrote: > > > Hi Steve, > > > > I'd suggest you to look at ZooKeeper-3.5.2 latest version and use dynamic > > reconfig feature. This will help to resize(add/remove zk server) your > > cluster without restarting entire cluster. > > > > Please refer the following links to understand more about the dynamic > > reconfig feature:- > > https://zookeeper.apache.org/doc/r3.5.2-alpha/zookeeperReconfig.html > > http://www.slideshare.net/Hadoop_Summit/dynamic- > > reconfiguration-of-zookeeper > > > > Regards, > > Rakesh > > > > On Thu, Oct 20, 2016 at 3:19 AM, Steve Newman wrote: > > > >> Apologies for a basic question, but I've been researching and haven't > been > >> able to find the answer online. > >> > >> What is the best way to add or remove replicas from a running ZooKeeper > >> cluster, with minimal downtime? To add a replica, the naive answer would > >> seem to be: > >> > >> 1. Prepare the new replica(s), i.e. install ZooKeeper and set up the > >> configuration files. > >> 2. Edit the configuration for all replicas (new and existing) to list > the > >> new replicas. > >> 3. Restart all replicas. (Simultaneously? Or gradually, one at a time?) > >> > >> Is this the best way to do it? Step 3 seems scary in a production > cluster. > >> Also, will the new replicas smoothly pick up the existing data, or is it > >> better to seed them with a snapshot somehow? > >> > >> Similarly, the naive answer for removing a replica would seem to be: > >> > >> 1. Halt the ZooKeeper process. > >> 2. Edit the configuration for all other replicas to remove the replica > >> that's going away. > >> 3. Restart all remaining replicas (one at a time?). > >> > >> Again, is this the best approach? > >> > >> Thanks, > >> Steve > >> > > > > > -- Cheers Michael. --001a1141bf42cc394d053f4f1803--