Return-Path: Delivered-To: apmail-hadoop-zookeeper-commits-archive@minotaur.apache.org Received: (qmail 87637 invoked from network); 11 Aug 2009 21:26:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 11 Aug 2009 21:26:50 -0000 Received: (qmail 27054 invoked by uid 500); 11 Aug 2009 21:26:57 -0000 Delivered-To: apmail-hadoop-zookeeper-commits-archive@hadoop.apache.org Received: (qmail 27026 invoked by uid 500); 11 Aug 2009 21:26:57 -0000 Mailing-List: contact zookeeper-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: zookeeper-dev@ Delivered-To: mailing list zookeeper-commits@hadoop.apache.org Received: (qmail 27016 invoked by uid 99); 11 Aug 2009 21:26:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 21:26:57 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED,OBSCURED_EMAIL X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 21:26:43 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id EEF9E23888A2; Tue, 11 Aug 2009 21:26:21 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r803305 - in /hadoop/zookeeper/branches/branch-3.2: ./ docs/ src/docs/src/documentation/content/xdocs/ src/java/main/org/apache/zookeeper/server/quorum/ src/java/main/org/apache/zookeeper/server/quorum/flexible/ src/java/test/org/apache/zoo... Date: Tue, 11 Aug 2009 21:26:19 -0000 To: zookeeper-commits@hadoop.apache.org From: mahadev@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20090811212621.EEF9E23888A2@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: mahadev Date: Tue Aug 11 21:26:12 2009 New Revision: 803305 URL: http://svn.apache.org/viewvc?rev=803305&view=rev Log: ZOOKEEPER-498. Unending Leader Elections : WAN configuration (flavio via mahadev) Added: hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.html hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.pdf hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperHierarchicalQuorums.xml Modified: hadoop/zookeeper/branches/branch-3.2/CHANGES.txt hadoop/zookeeper/branches/branch-3.2/docs/index.html hadoop/zookeeper/branches/branch-3.2/docs/index.pdf hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.html hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.pdf hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/index.xml hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarchical.java hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/FLEZeroWeightTest.java hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java Modified: hadoop/zookeeper/branches/branch-3.2/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/CHANGES.txt?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/CHANGES.txt (original) +++ hadoop/zookeeper/branches/branch-3.2/CHANGES.txt Tue Aug 11 21:26:12 2009 @@ -58,6 +58,9 @@ ZOOKEEPER-477. zkCleanup.sh is flaky (fernando via mahadev) + ZOOKEEPER-498. Unending Leader Elections : WAN configuration (flavio via + mahadev) + IMPROVEMENTS: NEW FEATURES: Modified: hadoop/zookeeper/branches/branch-3.2/docs/index.html URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/docs/index.html?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/docs/index.html (original) +++ hadoop/zookeeper/branches/branch-3.2/docs/index.html Tue Aug 11 21:26:12 2009 @@ -284,6 +284,10 @@
  • JMX - how to enable JMX in ZooKeeper
    • +
  • +Hierarchical quorums +
    • + Modified: hadoop/zookeeper/branches/branch-3.2/docs/index.pdf URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/docs/index.pdf?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== Binary files - no diff available. Modified: hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.html URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.html?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.html (original) +++ hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.html Tue Aug 11 21:26:12 2009 @@ -1150,6 +1150,9 @@ The left-hand side of the assignment is a colon-separated list of server identifiers. Note that groups must be disjoint and the union of all groups must be the ZooKeeper ensemble.

    +

    You will find an example here + +

    @@ -1165,11 +1168,14 @@ the weight of server is 1. If the configuration defines groups, but not weights, then a value of 1 will be assigned to all servers.

    +

    You will find an example here + +

    - +

    Authentication & Authorization Options

    The options in this section allow control over authentication/authorization performed by the service.

    @@ -1203,7 +1209,7 @@ - +

    Unsafe Options

    The following options can be useful, but be careful when you use them. The risk of each is explained along with the explanation of what @@ -1248,7 +1254,7 @@ - +

    ZooKeeper Commands: The Four Letter Words

    ZooKeeper responds to a small set of commands. Each command is composed of four letters. You issue the commands to ZooKeeper via telnet @@ -1312,7 +1318,7 @@ 

    $ echo ruok | nc 127.0.0.1 5111
 imok
    - +

    Data File Management

    ZooKeeper stores its data in a data directory and its transaction log in a transaction log directory. By default these two directories are @@ -1320,7 +1326,7 @@ transaction log files in a separate directory than the data files. Throughput increases and latency decreases when transaction logs reside on a dedicated log devices.

    - +

    The Data Directory

    This directory has two files in it:

      @@ -1366,14 +1372,14 @@ idempotent nature of its updates. By replaying the transaction log against fuzzy snapshots ZooKeeper gets the state of the system at the end of the log.

      - +

      The Log Directory

      The Log Directory contains the ZooKeeper transaction logs. Before any update takes place, ZooKeeper ensures that the transaction that represents the update is written to non-volatile storage. A new log file is started each time a snapshot is begun. The log file's suffix is the first zxid written to that log.

      - +

      File Management

      The format of snapshot and log files does not change between standalone ZooKeeper servers and different configurations of @@ -1393,7 +1399,7 @@ this document for more details on setting a retention policy and maintenance of ZooKeeper storage.

      - +

      Things to Avoid

      Here are some common problems you can avoid by configuring ZooKeeper correctly:

      @@ -1447,7 +1453,7 @@ - +

      Best Practices

      For best results, take note of the following list of good Zookeeper practices:

      Modified: hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.pdf URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/docs/zookeeperAdmin.pdf?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== Binary files - no diff available. Added: hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.html URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.html?rev=803305&view=auto ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.html (added) +++ hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.html Tue Aug 11 21:26:12 2009 @@ -0,0 +1,272 @@ + + + + + + + +Introduction to hierarchical quorums + + + + + + + + + +
      + +
      +Apache > Hadoop > ZooKeeper +
      + +
      + + + + + + + +
      +
      +  + +
      +
      + + + + +
      +
      +
      +
      + +
      + + +
      + +
      + +   +
      + + + + + +
      +
      +PDF -icon
      + PDF       +
      +

      Introduction to hierarchical quorums

      + + + + + +

      + This document gives an example of how to use hierarchical quorums. The basic idea is + very simple. First, we split servers into groups, and add a line for each group listing + the servers that form this group. Next we have to assign a weight to each server. +

      + + +

      + The following example shows how to configure a system with three groups of three servers + each, and we assign a weight of 1 to each server: +

      + + +
      +    group.1=1:2:3
+    group.2=4:5:6
+    group.3=7:8:9
+   
+    weight.1=1
+    weight.2=1
+    weight.3=1
+    weight.4=1
+    weight.5=1
+    weight.6=1
+    weight.7=1
+    weight.8=1
+    weight.9=1
+
      + + +

      + When running the system, we are able to form a quorum once we have a majority of votes from + a majority of non-zero-weight groups. Groups that have zero weight are discarded and not + considered when forming quorums. Looking at the example, we are able to form a quorum once + we have votes from at least two servers from each of two different groups. +

      + +

      + +

      +
      + +
       
      +
      + + + Added: hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.pdf URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.pdf?rev=803305&view=auto ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.pdf (added) +++ hadoop/zookeeper/branches/branch-3.2/docs/zookeeperHierarchicalQuorums.pdf Tue Aug 11 21:26:12 2009 @@ -0,0 +1,94 @@ +%PDF-1.3 +%ª«¬­ +4 0 obj +<< /Type /Info +/Producer (FOP 0.20.5) >> +endobj +5 0 obj +<< /Length 1156 /Filter [ /ASCII85Decode /FlateDecode ] + >> +stream +Gatm;d>jt['Re;/pi0?gEG]2>7%/mKeQK;[.^b8cLUI^%R,F:gX::AarVDr_iL2Kr[]8m0Y&j+*3PI]/q/Fu7ob\jl0gUS07u"3V8'l6G&TJnMZ=,r1'D)NEc[0[63(;Z32E"n[Z-eNA^$AFCL5rO[BRaHHN?>Q;Ei%#8())ZS]!*^&SbcC_jgdJHX'eW/U-#0C]4A[c^*\ENCuL#KWiI8G-YH6ihh(bC`-C-fj3)k_F^e>qimHaZl?&Z?1.].Gl_#G[[1EccH\H.bu46E'<04j`Rdq7-m6R\0NlIZ;P\4CbD]C9UM%O3hNU;jMJVb="9\i,P^G^@''BBtEl&Deu&6FW4@p-o7CfGE:6T]/C_2:U,H\IoY[d=8l!qe<1i*d40GD^J_N(_J<4,hVTGY*AhmO2BI"[$*pC'YTA)'[[SM1+Qc^)S.4+>lK,;8!b;+t>9XiP[6Ke!lh)>b4jPn_1Y^9])]Y;VfT_2Y5"OcnaC$J7hF$0[4$g4*;iW!(n;t\qoqIK^V2DVb 'nHCK;eW%_D`pYB`F+glaWiFi$[@o&P*QVO>YOj7Upsr%CG!=t*8MrF>G&&?+VbB)E>Aj9oNcF3o.#XU&N:MClcS&l__snNB4,`>90ga;7RP#mrnFbK)"7\%Ul7UODO!B^sDp*P')hGh@%J\+cc##`tYF:c-qVSQ%#.F~> +endstream +endobj +6 0 obj +<< /Type /Page +/Parent 1 0 R +/MediaBox [ 0 0 612 792 ] +/Resources 3 0 R +/Contents 5 0 R +>> +endobj +7 0 obj +<< /Type /Font +/Subtype /Type1 +/Name /F3 +/BaseFont /Helvetica-Bold +/Encoding /WinAnsiEncoding >> +endobj +8 0 obj +<< /Type /Font +/Subtype /Type1 +/Name /F5 +/BaseFont /Times-Roman +/Encoding /WinAnsiEncoding >> +endobj +9 0 obj +<< /Type /Font +/Subtype /Type1 +/Name /F1 +/BaseFont /Helvetica +/Encoding /WinAnsiEncoding >> +endobj +10 0 obj +<< /Type /Font +/Subtype /Type1 +/Name /F9 +/BaseFont /Courier +/Encoding /WinAnsiEncoding >> +endobj +11 0 obj +<< /Type /Font +/Subtype /Type1 +/Name /F2 +/BaseFont /Helvetica-Oblique +/Encoding /WinAnsiEncoding >> +endobj +1 0 obj +<< /Type /Pages +/Count 1 +/Kids [6 0 R ] >> +endobj +2 0 obj +<< /Type /Catalog +/Pages 1 0 R + >> +endobj +3 0 obj +<< +/Font << /F3 7 0 R /F5 8 0 R /F1 9 0 R /F9 10 0 R /F2 11 0 R >> +/ProcSet [ /PDF /ImageC /Text ] >> +endobj +xref +0 12 +0000000000 65535 f +0000001975 00000 n +0000002033 00000 n +0000002083 00000 n +0000000015 00000 n +0000000071 00000 n +0000001319 00000 n +0000001425 00000 n +0000001537 00000 n +0000001646 00000 n +0000001753 00000 n +0000001859 00000 n +trailer +<< +/Size 12 +/Root 2 0 R +/Info 4 0 R +>> +startxref +2203 +%%EOF Modified: hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/index.xml URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/index.xml?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/index.xml (original) +++ hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/index.xml Tue Aug 11 21:26:12 2009 @@ -63,6 +63,7 @@
    • Administrator's Guide - a guide for system administrators and anyone else who might deploy ZooKeeper
    • Quota Guide - a guide for system administrators on Quotas in ZooKeeper.
    • JMX - how to enable JMX in ZooKeeper
      • +
    • Hierarchical quorums
    Modified: hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml (original) +++ hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperAdmin.xml Tue Aug 11 21:26:12 2009 @@ -800,6 +800,9 @@ The left-hand side of the assignment is a colon-separated list of server identifiers. Note that groups must be disjoint and the union of all groups must be the ZooKeeper ensemble. + + You will find an example here + @@ -816,6 +819,9 @@ the weight of server is 1. If the configuration defines groups, but not weights, then a value of 1 will be assigned to all servers. + + You will find an example here + Added: hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperHierarchicalQuorums.xml URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperHierarchicalQuorums.xml?rev=803305&view=auto ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperHierarchicalQuorums.xml (added) +++ hadoop/zookeeper/branches/branch-3.2/src/docs/src/documentation/content/xdocs/zookeeperHierarchicalQuorums.xml Tue Aug 11 21:26:12 2009 @@ -0,0 +1,75 @@ + + + + +
    + Introduction to hierarchical quorums + + + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. You may + obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0. + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an "AS IS" + BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing permissions + and limitations under the License. + + + + This document contains information about hierarchical quorums. + + + + + This document gives an example of how to use hierarchical quorums. The basic idea is + very simple. First, we split servers into groups, and add a line for each group listing + the servers that form this group. Next we have to assign a weight to each server. + + + + The following example shows how to configure a system with three groups of three servers + each, and we assign a weight of 1 to each server: + + + + group.1=1:2:3 + group.2=4:5:6 + group.3=7:8:9 + + weight.1=1 + weight.2=1 + weight.3=1 + weight.4=1 + weight.5=1 + weight.6=1 + weight.7=1 + weight.8=1 + weight.9=1 + + + + When running the system, we are able to form a quorum once we have a majority of votes from + a majority of non-zero-weight groups. Groups that have zero weight are discarded and not + considered when forming quorums. Looking at the example, we are able to form a quorum once + we have votes from at least two servers from each of two different groups. + +
    \ No newline at end of file Modified: hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java (original) +++ hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/FastLeaderElection.java Tue Aug 11 21:26:12 2009 @@ -172,20 +172,22 @@ */ class WorkerReceiver implements Runnable { - + volatile boolean stop; QuorumCnxManager manager; WorkerReceiver(QuorumCnxManager manager) { + this.stop = false; this.manager = manager; } public void run() { Message response; - while (true) { + while (!stop) { // Sleeps on receive try{ - response = manager.recvQueue.take(); + response = manager.recvQueue.poll(3000, TimeUnit.MILLISECONDS); + if(response == null) continue; // Receive new message LOG.debug("Receive new message."); @@ -266,6 +268,7 @@ e.toString()); } } + LOG.info("WorkerReceiver is down"); } } @@ -276,23 +279,26 @@ */ class WorkerSender implements Runnable { - + volatile boolean stop; QuorumCnxManager manager; WorkerSender(QuorumCnxManager manager){ + this.stop = false; this.manager = manager; } public void run() { - while (true) { + while (!stop) { try { - ToSend m = sendqueue.take(); + ToSend m = sendqueue.poll(3000, TimeUnit.MILLISECONDS); + if(m == null) continue; + process(m); } catch (InterruptedException e) { break; } - } + LOG.info("WorkerSender is down"); } /** @@ -326,6 +332,10 @@ return (sendqueue.isEmpty() || recvqueue.isEmpty()); } + + WorkerSender ws; + WorkerReceiver wr; + /** * Constructor of class Messenger. * @@ -333,20 +343,33 @@ */ Messenger(QuorumCnxManager manager) { - Thread t = new Thread(new WorkerSender(manager), + this.ws = new WorkerSender(manager); + + Thread t = new Thread(this.ws, "WorkerSender Thread"); t.setDaemon(true); t.start(); - t = new Thread(new WorkerReceiver(manager), + this.wr = new WorkerReceiver(manager); + + t = new Thread(this.wr, "WorkerReceiver Thread"); t.setDaemon(true); t.start(); } + + /** + * Stops instances of WorkerSender and WorkerReceiver + */ + void halt(){ + this.ws.stop = true; + this.wr.stop = true; + } } - QuorumPeer self; + QuorumPeer self; + Messenger messenger; volatile long logicalclock; /* Election instance */ long proposedLeader; long proposedZxid; @@ -369,7 +392,8 @@ * @param manager Connection manager */ public FastLeaderElection(QuorumPeer self, QuorumCnxManager manager){ - this.manager = manager; + this.stop = false; + this.manager = manager; starter(self, manager); } @@ -390,7 +414,7 @@ sendqueue = new LinkedBlockingQueue(); recvqueue = new LinkedBlockingQueue(); - new Messenger(manager); + this.messenger = new Messenger(manager); } private void leaveInstance() { @@ -401,9 +425,14 @@ return manager; } + volatile boolean stop; public void shutdown(){ + stop = true; LOG.debug("Shutting down connection manager"); manager.halt(); + LOG.debug("Shutting down messenger"); + messenger.halt(); + LOG.debug("FLE is down"); } @@ -434,6 +463,10 @@ */ private boolean totalOrderPredicate(long newId, long newZxid, long curId, long curZxid) { LOG.debug("id: " + newId + ", proposed id: " + curId + ", zxid: " + newZxid + ", proposed zxid: " + curZxid); + if(self.getQuorumVerifier().getWeight(newId) == 0){ + return false; + } + if ((newZxid > curZxid) || ((newZxid == curZxid) && (newId > curId))) return true; @@ -550,7 +583,8 @@ * Loop in which we exchange notifications until we find a leader */ - while (self.getPeerState() == ServerState.LOOKING) { + while ((self.getPeerState() == ServerState.LOOKING) && + (!stop)){ /* * Remove next notification from queue, times out after 2 times * the termination time @@ -606,14 +640,12 @@ } LOG.info("Adding vote"); - /* - * Skip zero-weight servers - */ - if(self.getQuorumVerifier().getWeight(n.sid) != 0) - recvset.put(n.sid, new Vote(n.leader, n.zxid, n.epoch)); + + recvset.put(n.sid, new Vote(n.leader, n.zxid, n.epoch)); //If have received from all nodes, then terminate - if (self.quorumPeers.size() == recvset.size()) { + if ((self.quorumPeers.size() == recvset.size()) && + (self.getQuorumVerifier().getWeight(proposedLeader) != 0)){ self.setPeerState((proposedLeader == self.getId()) ? ServerState.LEADING: ServerState.FOLLOWING); leaveInstance(); Modified: hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java (original) +++ hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java Tue Aug 11 21:26:12 2009 @@ -338,7 +338,7 @@ e); } } else { - LOG.error("There is a connection for server " + sid); + LOG.debug("There is a connection for server " + sid); } } Modified: hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarchical.java URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarchical.java?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarchical.java (original) +++ hadoop/zookeeper/branches/branch-3.2/src/java/main/org/apache/zookeeper/server/quorum/flexible/QuorumHierarchical.java Tue Aug 11 21:26:12 2009 @@ -121,8 +121,8 @@ this.serverGroup = serverGroup; this.groupWeight = new HashMap(); - computeGroupWeight(); this.numGroups = numGroups; + computeGroupWeight(); } @@ -219,7 +219,11 @@ * Do not consider groups with weight zero */ for(long weight: groupWeight.values()){ - if(weight == 0) numGroups--; + LOG.debug("Group weight: " + weight); + if(weight == ((long) 0)){ + numGroups--; + LOG.debug("One zero-weight group: " + 1 + ", " + numGroups); + } } } @@ -233,7 +237,7 @@ * Adds up weights per group */ if(set.size() == 0) return false; - else LOG.info("Set size: " + set.size()); + else LOG.debug("Set size: " + set.size()); for(long sid : set){ Long gid = serverGroup.get(sid); @@ -250,17 +254,18 @@ */ int majGroupCounter = 0; for(long gid : expansion.keySet()) { - LOG.info("gid: " + expansion.get(gid)); + LOG.debug("Group info: " + expansion.get(gid) + ", " + gid + ", " + groupWeight.get(gid)); if(expansion.get(gid) > (groupWeight.get(gid) / 2) ) majGroupCounter++; } + LOG.debug("Majority group counter: " + majGroupCounter + ", " + numGroups); if((majGroupCounter > (numGroups / 2))){ - LOG.info("Positive set size: " + set.size()); + LOG.debug("Positive set size: " + set.size()); return true; } else { - LOG.info("Negative set size: " + set.size()); + LOG.debug("Negative set size: " + set.size()); return false; } } Modified: hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/FLEZeroWeightTest.java URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/FLEZeroWeightTest.java?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/FLEZeroWeightTest.java (original) +++ hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/FLEZeroWeightTest.java Tue Aug 11 21:26:12 2009 @@ -105,7 +105,7 @@ class LEThread extends Thread { int i; QuorumPeer peer; - //int peerRound = 1; + boolean fail; LEThread(QuorumPeer peer, int i) { this.i = i; @@ -116,6 +116,7 @@ public void run() { try { Vote v = null; + fail = false; while(true){ //while(true) { @@ -137,7 +138,7 @@ votes[i] = v; if((peer.getPeerState() == ServerState.LEADING) && - (peer.getId() > 2)) fail("Elected zero-weight server"); + (peer.getId() > 2)) fail = true; if((peer.getPeerState() == ServerState.FOLLOWING) || (peer.getPeerState() == ServerState.LEADING)) break; @@ -150,10 +151,10 @@ } @Test - public void testHierarchicalQuorum() throws Exception { + public void testZeroWeightQuorum() throws Exception { FastLeaderElection le[] = new FastLeaderElection[count]; - LOG.info("TestHierarchicalQuorum: " + getName()+ ", " + count); + LOG.info("TestZeroWeightQuorum: " + getName()+ ", " + count); for(int i = 0; i < count; i++) { peers.put(Long.valueOf(i), new QuorumServer(i, @@ -177,6 +178,9 @@ threads.get(i).join(15000); if (threads.get(i).isAlive()) { fail("Threads didn't join"); + } else { + if(threads.get(i).fail) + fail("Elected zero-weight server"); } } } Modified: hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java URL: http://svn.apache.org/viewvc/hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java?rev=803305&r1=803304&r2=803305&view=diff ============================================================================== --- hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java (original) +++ hadoop/zookeeper/branches/branch-3.2/src/java/test/org/apache/zookeeper/test/HierarchicalQuorumTest.java Tue Aug 11 21:26:12 2009 @@ -1,5 +1,4 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one +/* Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file @@ -17,168 +16,259 @@ */ package org.apache.zookeeper.test; + import java.io.ByteArrayInputStream; import java.io.File; +import java.io.IOException; import java.net.InetSocketAddress; -import java.util.ArrayList; import java.util.HashMap; +import java.util.LinkedHashSet; import java.util.Properties; -import java.util.Random; - -import junit.framework.TestCase; +import java.util.Set; import org.apache.log4j.Logger; +import org.apache.zookeeper.ZooKeeper; import org.apache.zookeeper.server.quorum.FastLeaderElection; import org.apache.zookeeper.server.quorum.QuorumPeer; -import org.apache.zookeeper.server.quorum.Vote; import org.apache.zookeeper.server.quorum.QuorumPeer.QuorumServer; -import org.apache.zookeeper.server.quorum.QuorumPeer.ServerState; import org.apache.zookeeper.server.quorum.flexible.QuorumHierarchical; +import org.junit.After; import org.junit.Before; import org.junit.Test; -public class HierarchicalQuorumTest extends TestCase { - private static final Logger LOG = Logger.getLogger(HierarchicalQuorumTest.class); - - Properties qp; +/** + * Comprehensive test of hierarchical quorums, assuming servers with zero weight. + * This test uses ClientTest to verify that the ensemble works after a leader is + * elected. + * + * This implementation is based on QuorumBase, the main difference being that it + * uses hierarchical quorums and FLE. + */ - int count; - int baseport; - int baseLEport; - HashMap peers; - ArrayList threads; - File tmpdir[]; - int port[]; - Object finalObj; - - volatile Vote votes[]; - volatile boolean leaderDies; - volatile long leader = -1; - Random rand = new Random(); +public class HierarchicalQuorumTest extends ClientBase { + private static final Logger LOG = Logger.getLogger(QuorumBase.class); + File s1dir, s2dir, s3dir, s4dir, s5dir; + QuorumPeer s1, s2, s3, s4, s5; + private int port1; + private int port2; + private int port3; + private int port4; + private int port5; + + private int leport1; + private int leport2; + private int leport3; + private int leport4; + private int leport5; - @Before + Properties qp; + private final ClientTest ct = new ClientTest(); + @Override protected void setUp() throws Exception { - count = 9; - baseport= 33003; - baseLEport = 43003; - - peers = new HashMap(count); - threads = new ArrayList(count); - votes = new Vote[count]; - tmpdir = new File[count]; - port = new int[count]; - finalObj = new Object(); - - String config = "group.1=0:1:2\n" + - "group.2=3:4:5\n" + - "group.3=6:7:8\n\n" + - "weight.0=1\n" + + LOG.info("STARTING " + getName()); + int baseport = 12333; + int baseLEport = 14333; + + setupTestEnv(); + + JMXEnv.setUp(); + + //setUpAll(); + + port1 = baseport; + port2 = baseport + 1; + port3 = baseport + 2; + port4 = baseport + 3; + port5 = baseport + 4; + leport1 = baseLEport; + leport2 = baseLEport + 1; + leport3 = baseLEport + 2; + leport4 = baseLEport + 3; + leport5 = baseLEport + 4; + + hostPort = "127.0.0.1:" + port1 + + ",127.0.0.1:" + port2 + + ",127.0.0.1:" + port3 + + ",127.0.0.1:" + port4 + + ",127.0.0.1:" + port5; + LOG.info("Ports are: " + hostPort); + + s1dir = ClientBase.createTmpDir(); + s2dir = ClientBase.createTmpDir(); + s3dir = ClientBase.createTmpDir(); + s4dir = ClientBase.createTmpDir(); + s5dir = ClientBase.createTmpDir(); + + String config = "group.1=1:2:3\n" + + "group.2=4:5\n" + "weight.1=1\n" + "weight.2=1\n" + "weight.3=1\n" + - "weight.4=1\n" + - "weight.5=1\n" + - "weight.6=1\n" + - "weight.7=1\n" + - "weight.8=1"; + "weight.4=0\n" + + "weight.5=0\n"; ByteArrayInputStream is = new ByteArrayInputStream(config.getBytes()); this.qp = new Properties(); + qp.load(is); + startServers(); + + ct.hostPort = hostPort; + //ct.setUpAll(); + + LOG.info("Setup finished"); + } + + + void startServers() throws Exception { + int tickTime = 2000; + int initLimit = 3; + int syncLimit = 3; + HashMap peers = new HashMap(); + peers.put(Long.valueOf(1), new QuorumServer(1, + new InetSocketAddress("127.0.0.1", port1 + 1000), + new InetSocketAddress("127.0.0.1", leport1 + 1000))); + peers.put(Long.valueOf(2), new QuorumServer(2, + new InetSocketAddress("127.0.0.1", port2 + 1000), + new InetSocketAddress("127.0.0.1", leport2 + 1000))); + peers.put(Long.valueOf(3), new QuorumServer(3, + new InetSocketAddress("127.0.0.1", port3 + 1000), + new InetSocketAddress("127.0.0.1", leport3 + 1000))); + peers.put(Long.valueOf(4), new QuorumServer(4, + new InetSocketAddress("127.0.0.1", port4 + 1000), + new InetSocketAddress("127.0.0.1", leport4 + 1000))); + peers.put(Long.valueOf(5), new QuorumServer(5, + new InetSocketAddress("127.0.0.1", port5 + 1000), + new InetSocketAddress("127.0.0.1", leport5 + 1000))); + + LOG.info("creating QuorumPeer 1 port " + port1); + QuorumHierarchical hq1 = new QuorumHierarchical(qp); + s1 = new QuorumPeer(peers, s1dir, s1dir, port1, 3, 1, tickTime, initLimit, syncLimit, hq1); + assertEquals(port1, s1.getClientPort()); + + LOG.info("creating QuorumPeer 2 port " + port2); + QuorumHierarchical hq2 = new QuorumHierarchical(qp); + s2 = new QuorumPeer(peers, s2dir, s2dir, port2, 3, 2, tickTime, initLimit, syncLimit, hq2); + assertEquals(port2, s2.getClientPort()); + + LOG.info("creating QuorumPeer 3 port " + port3); + QuorumHierarchical hq3 = new QuorumHierarchical(qp); + s3 = new QuorumPeer(peers, s3dir, s3dir, port3, 3, 3, tickTime, initLimit, syncLimit, hq3); + assertEquals(port3, s3.getClientPort()); + + LOG.info("creating QuorumPeer 4 port " + port4); + QuorumHierarchical hq4 = new QuorumHierarchical(qp); + s4 = new QuorumPeer(peers, s4dir, s4dir, port4, 3, 4, tickTime, initLimit, syncLimit, hq4); + assertEquals(port4, s4.getClientPort()); + + LOG.info("creating QuorumPeer 5 port " + port5); + QuorumHierarchical hq5 = new QuorumHierarchical(qp); + s5 = new QuorumPeer(peers, s5dir, s5dir, port5, 3, 5, tickTime, initLimit, syncLimit, hq5); + assertEquals(port5, s5.getClientPort()); + LOG.info("start QuorumPeer 1"); + s1.start(); + LOG.info("start QuorumPeer 2"); + s2.start(); + LOG.info("start QuorumPeer 3"); + s3.start(); + LOG.info("start QuorumPeer 4"); + s4.start(); + LOG.info("start QuorumPeer 5"); + s5.start(); + LOG.info("started QuorumPeer 5"); + + LOG.info ("Closing ports " + hostPort); + for (String hp : hostPort.split(",")) { + assertTrue("waiting for server up", + ClientBase.waitForServerUp(hp, + CONNECTION_TIMEOUT)); + LOG.info(hp + " is accepting client connections"); + } - LOG.info("SetUp " + getName()); + // interesting to see what's there... + JMXEnv.dump(); + // make sure we have these 5 servers listed + Set ensureNames = new LinkedHashSet(); + for (int i = 1; i <= 5; i++) { + ensureNames.add("InMemoryDataTree"); + } + for (int i = 1; i <= 5; i++) { + ensureNames.add("name0=ReplicatedServer_id" + i + + ",name1=replica." + i + ",name2="); + } + for (int i = 1; i <= 5; i++) { + for (int j = 1; j <= 5; j++) { + ensureNames.add("name0=ReplicatedServer_id" + i + + ",name1=replica." + j); + } + } + for (int i = 1; i <= 5; i++) { + ensureNames.add("name0=ReplicatedServer_id" + i); + } + JMXEnv.ensureAll(ensureNames.toArray(new String[ensureNames.size()])); } + @After + @Override protected void tearDown() throws Exception { - for(int i = 0; i < threads.size(); i++) { - threads.get(i).peer.shutdown(); - ((FastLeaderElection) threads.get(i).peer.getElectionAlg()).shutdown(); + LOG.info("TearDown started"); + //ct.tearDownAll(); + + LOG.info("Shutting down server 1"); + shutdown(s1); + LOG.info("Shutting down server 2"); + shutdown(s2); + LOG.info("Shutting down server 3"); + shutdown(s3); + LOG.info("Shutting down server 4"); + shutdown(s4); + LOG.info("Shutting down server 5"); + shutdown(s5); + + for (String hp : hostPort.split(",")) { + assertTrue("waiting for server down", + ClientBase.waitForServerDown(hp, + ClientBase.CONNECTION_TIMEOUT)); + LOG.info(hp + " is no longer accepting client connections"); } + + JMXEnv.tearDown(); + LOG.info("FINISHED " + getName()); } - class LEThread extends Thread { - int i; - QuorumPeer peer; - //int peerRound = 1; - - LEThread(QuorumPeer peer, int i) { - this.i = i; - this.peer = peer; - LOG.info("Constructor: " + getName()); - } - - public void run() { - try { - Vote v = null; - while(true){ - - //while(true) { - peer.setPeerState(ServerState.LOOKING); - LOG.info("Going to call leader election."); - v = peer.getElectionAlg().lookForLeader(); - if(v == null){ - LOG.info("Thread " + i + " got a null vote"); - return; - } - - /* - * A real zookeeper would take care of setting the current vote. Here - * we do it manually. - */ - peer.setCurrentVote(v); - - LOG.info("Finished election: " + i + ", " + v.id); - votes[i] = v; - - if((peer.getPeerState() == ServerState.FOLLOWING) || - (peer.getPeerState() == ServerState.LEADING)) break; - } - LOG.debug("Thread " + i + " votes " + v); - } catch (InterruptedException e) { - e.printStackTrace(); + protected void shutdown(QuorumPeer qp) { + try { + ((FastLeaderElection) qp.getElectionAlg()).shutdown(); + LOG.info("Done with leader election"); + qp.shutdown(); + LOG.info("Done with quorum peer"); + qp.join(30000); + if (qp.isAlive()) { + fail("QP failed to shutdown in 30 seconds"); } + } catch (InterruptedException e) { + LOG.debug("QP interrupted", e); } } - @Test - public void testHierarchicalQuorum() throws Exception { - runTest(0); + protected ZooKeeper createClient() + throws IOException, InterruptedException + { + return createClient(hostPort); } - - @Test - public void testHierarchicalQuorumPartial() throws Exception { - runTest(3); + + protected ZooKeeper createClient(String hp) + throws IOException, InterruptedException + { + CountdownWatcher watcher = new CountdownWatcher(); + return createClient(watcher, hp); } - - private void runTest(int delta) throws Exception { - FastLeaderElection le[] = new FastLeaderElection[count]; - LOG.info("TestHierarchicalQuorum: " + getName()+ ", " + count); - for(int i = 0; i < count; i++) { - peers.put(Long.valueOf(i), new QuorumServer(i, new InetSocketAddress(baseport+i), - new InetSocketAddress(baseLEport+i))); - tmpdir[i] = ClientBase.createTmpDir(); - port[i] = baseport+i; - } - - for(int i = 0; i < (le.length - delta); i++) { - QuorumHierarchical hq = new QuorumHierarchical(qp); - QuorumPeer peer = new QuorumPeer(peers, tmpdir[i], tmpdir[i], port[i], 3, i, 2, 2, 2, hq); - peer.startLeaderElection(); - LEThread thread = new LEThread(peer, i); - thread.start(); - threads.add(thread); - } - LOG.info("Started threads " + getName()); - - for(int i = 0; i < threads.size(); i++) { - threads.get(i).join(15000); - if (threads.get(i).isAlive()) { - fail("Threads didn't join"); - } - } + @Test + public void testHierarchicalQuorum() throws Throwable { + ct.testHammerBasic(); } } \ No newline at end of file