From dev-return-74519-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Fri Oct 12 21:49:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1F57F18067A for ; Fri, 12 Oct 2018 21:49:03 +0200 (CEST) Received: (qmail 82011 invoked by uid 500); 12 Oct 2018 19:49:03 -0000 Mailing-List: contact dev-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list dev@zookeeper.apache.org Received: (qmail 82000 invoked by uid 99); 12 Oct 2018 19:49:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Oct 2018 19:49:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8C9521809CC for ; Fri, 12 Oct 2018 19:49:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id NGrdpU_Mpbzc for ; Fri, 12 Oct 2018 19:49:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 6090F5F3AC for ; Fri, 12 Oct 2018 19:49:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id C4754E25D5 for ; Fri, 12 Oct 2018 19:49:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4EDB324DC4 for ; Fri, 12 Oct 2018 19:49:00 +0000 (UTC) Date: Fri, 12 Oct 2018 19:49:00 +0000 (UTC) From: "Brian Nixon (JIRA)" To: dev@zookeeper.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ZOOKEEPER-3140) Allow Followers to host Observers MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ZOOKEEPER-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648379#comment-16648379 ] Brian Nixon commented on ZOOKEEPER-3140: ---------------------------------------- A note on future work - it would be cool to see the serialized format of the QuorumVerifier (used for the dynamic config files and the like) updated to a more extensible form so we can track more topology and port information through it. This would give us a lot more flexibility in setting and propagating the observer master port, in particular in letting each server publish its own port instead of requiring a single static port for the whole ensemble. Would also be useful for purposes such as ZOOKEEPER-3166. I thought there was an existent Jira on this requested change but didn't see one in a cursory search. I will create one specifically eventually, if I get the time. :) > Allow Followers to host Observers > --------------------------------- > > Key: ZOOKEEPER-3140 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3140 > Project: ZooKeeper > Issue Type: New Feature > Components: server > Affects Versions: 3.6.0 > Reporter: Brian Nixon > Assignee: Brian Nixon > Priority: Minor > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > Observers function simple as non-voting members of the ensemble, sharing the Learner interface with Followers and holding only a slightly difference internal pipeline. Both maintain connections along the quorum port with the Leader by which they learn of all new proposals on the ensemble. > > There are benefits to allowing Observers to connect to the Followers to plug into the commit stream in addition to connecting to the Leader. It shifts the burden of supporting Observers off the Leader and allow it to focus on coordinating the commit of writes. This means better performance when the Leader is under high load, particularly high network load such as can happen after a leader election when many Learners need to sync. It also reduces the total network connections maintained on the Leader when there are a high number of observers. One the other end, Observer availability is improved since it will take shorter time for a high number of Observers to finish syncing and start serving client traffic. > > The current implementation only supports scaling the number of Observers into the hundreds before performance begins to degrade. By opening up Followers to also host Observers, over a thousand observers can be hosted on a typical ensemble without major negative impact under both normal operation and during post-leader election sync. -- This message was sent by Atlassian JIRA (v7.6.3#76005)