Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3A49D200C56 for ; Thu, 30 Mar 2017 08:26:39 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 38F19160B9A; Thu, 30 Mar 2017 06:26:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0EA22160BA3 for ; Thu, 30 Mar 2017 08:26:36 +0200 (CEST) Received: (qmail 86349 invoked by uid 500); 30 Mar 2017 06:26:35 -0000 Mailing-List: contact commits-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list commits@zookeeper.apache.org Received: (qmail 81491 invoked by uid 99); 30 Mar 2017 06:26:27 -0000 Received: from Unknown (HELO svn01-us-west.apache.org) (209.188.14.144) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Mar 2017 06:26:27 +0000 Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id CA6423A4854 for ; Thu, 30 Mar 2017 06:26:25 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1009357 [26/43] - in /websites/staging/zookeeper/trunk/content: ./ doc/r3.4.10/ doc/r3.4.10/api/ doc/r3.4.10/api/org/ doc/r3.4.10/api/org/apache/ doc/r3.4.10/api/org/apache/zookeeper/ doc/r3.4.10/api/org/apache/zookeeper/class-use/ doc/r3.... Date: Thu, 30 Mar 2017 06:26:23 -0000 To: commits@zookeeper.apache.org From: buildbot@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20170330062625.CA6423A4854@svn01-us-west.apache.org> archived-at: Thu, 30 Mar 2017 06:26:39 -0000 Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/SnapshotFormatter.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/SnapshotFormatter.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/SnapshotFormatter.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,283 @@ + + + + + + +SnapshotFormatter (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.zookeeper.server +
+Class SnapshotFormatter

+
+java.lang.Object
+  extended by org.apache.zookeeper.server.SnapshotFormatter
+
+
+
+
public class SnapshotFormatter
extends Object
+ + +

+Dump a snapshot file to stdout. +

+ +

+

+ +

+ + + + + + + + + + + +
+Constructor Summary
SnapshotFormatter() + +
+           
+  + + + + + + + + + + + + + + + +
+Method Summary
+static voidmain(String[] args) + +
+          USAGE: SnapshotFormatter snapshot_file
+ voidrun(String snapshotFileName) + +
+           
+  + + + + + + + +
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Constructor Detail
+ +

+SnapshotFormatter

+
+public SnapshotFormatter()
+
+
+ + + + + + + + +
+Method Detail
+ +

+main

+
+public static void main(String[] args)
+                 throws Exception
+
+
USAGE: SnapshotFormatter snapshot_file +

+

+ +
Throws: +
Exception
+
+
+
+ +

+run

+
+public void run(String snapshotFileName)
+         throws IOException
+
+
+ +
Throws: +
IOException
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/ZooKeeperServerMain.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/ZooKeeperServerMain.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/ZooKeeperServerMain.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,280 @@ + + + + + + +ZooKeeperServerMain (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.zookeeper.server +
+Class ZooKeeperServerMain

+
+java.lang.Object
+  extended by org.apache.zookeeper.server.ZooKeeperServerMain
+
+
+
+
public class ZooKeeperServerMain
extends Object
+ + +

+This class starts and runs a standalone ZooKeeperServer. +

+ +

+

+ +

+ + + + + + + + + + + +
+Constructor Summary
ZooKeeperServerMain() + +
+           
+  + + + + + + + + + + + + + + + +
+Method Summary
+static voidmain(String[] args) + +
+           
+ voidrunFromConfig(org.apache.zookeeper.server.ServerConfig config) + +
+          Run from a ServerConfig.
+  + + + + + + + +
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Constructor Detail
+ +

+ZooKeeperServerMain

+
+public ZooKeeperServerMain()
+
+
+ + + + + + + + +
+Method Detail
+ +

+main

+
+public static void main(String[] args)
+
+
+
+
+
+
+ +

+runFromConfig

+
+public void runFromConfig(org.apache.zookeeper.server.ServerConfig config)
+                   throws IOException
+
+
Run from a ServerConfig. +

+

+
Parameters:
config - ServerConfig to use. +
Throws: +
IOException
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/LogFormatter.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/LogFormatter.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/LogFormatter.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.LogFormatter (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.LogFormatter

+
+No usage of org.apache.zookeeper.server.LogFormatter +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/PurgeTxnLog.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/PurgeTxnLog.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/PurgeTxnLog.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.PurgeTxnLog (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.PurgeTxnLog

+
+No usage of org.apache.zookeeper.server.PurgeTxnLog +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/SnapshotFormatter.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/SnapshotFormatter.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/SnapshotFormatter.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.SnapshotFormatter (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.SnapshotFormatter

+
+No usage of org.apache.zookeeper.server.SnapshotFormatter +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/ZooKeeperServerMain.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/ZooKeeperServerMain.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/class-use/ZooKeeperServerMain.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.ZooKeeperServerMain (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.ZooKeeperServerMain

+
+No usage of org.apache.zookeeper.server.ZooKeeperServerMain +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-frame.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-frame.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-frame.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,38 @@ + + + + + + +org.apache.zookeeper.server (ZooKeeper 3.4.10 API) + + + + + + + + + + + +org.apache.zookeeper.server + + + + +
+Classes  + +
+LogFormatter +
+PurgeTxnLog +
+SnapshotFormatter +
+ZooKeeperServerMain
+ + + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-summary.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-summary.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-summary.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,264 @@ + + + + + + +org.apache.zookeeper.server (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview   Package  Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV PACKAGE  + NEXT PACKAGE + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+

+Package org.apache.zookeeper.server +

+ZooKeeper server theory of operation +

+See: +
+          Description +

+ + + + + + + + + + + + + + + + + + + + + +
+Class Summary
LogFormatter 
PurgeTxnLogthis class is used to clean up the + snapshot and data log dir's.
SnapshotFormatterDump a snapshot file to stdout.
ZooKeeperServerMainThis class starts and runs a standalone ZooKeeperServer.
+  + +

+

+Package org.apache.zookeeper.server Description +

+ +

+

ZooKeeper server theory of operation

+ZooKeeperServer is designed to work in standalone mode and also +be extensible so that it can be used to implement the quorum based +version of ZooKeeper. +

+ZooKeeper maintains a order when processing requests: +

    +
  • All requests will be processed in order. +
  • All responses will return in order. +
  • All watches will be sent in the order that the update takes place. +
+

+We will explain the three aspects of ZooKeeperServer: request processing, data +structure maintenance, and session tracking. + +

Request processing

+ +Requests are received by the ServerCnxn. Demarshalling of a request is +done by ClientRequestHandler. After a request has been demarshalled, +ClientRequestHandler invokes the relevant method in ZooKeeper and marshals +the result. +

+If the request is just a query, it will be processed by ZooKeeper and returned. +Otherwise, the request will be validated and a transaction will be generated +and logged. This the request will then wait until the request has been logged +before continuing processing. +

+Requests are logged as a group. Transactions are queued up and the SyncThread +will process them at predefined intervals. (Currently 20ms) The SyncThread +interacts with ZooKeeperServer the txnQueue. Transactions are added to the +txnQueue of SyncThread via queueItem. When the transaction has been synced to +disk, its callback will be invoked which will cause the request processing to +be completed. + +

Data structure maintenance

+ +ZooKeeper data is stored in-memory. Each znode is stored in a DataNode object. +This object is accessed through a hash table that maps paths to DataNodes. +DataNodes also organize themselves into a tree. This tree is only used for +serializing nodes. +

+We guarantee that changes to nodes are stored to non-volatile media before +responding to a client. We do this quickly by writing changes as a sequence +of transactions in a log file. Even though we flush transactions as a group, +we need to avoid seeks as much as possible. Also, since the server can fail +at any point, we need to be careful of partial records. +

+We address the above problems by +

    +
  • Pre-allocating 1M chunks of file space. This allows us to append to the +file without causing seeks to update file size. It also means that we need +to check for the end of the log by looking for a zero length transaction +rather than simply end of file. +
  • Writing a signature at the end of each transaction. When processing +transactions, we only use transactions that have a valid signature at the end. +
+

+As the server runs, the log file will grow quite large. To avoid long startup +times we periodically take a snapshot of the tree of DataNodes. We cannot +take the snapshot synchronously as the data takes a while to write out, so +instead we asynchronously write out the tree. This means that we end up +with a "corrupt" snapshot of the data tree. More formally if we define T +to be the real snapshot of the tree at the time we begin taking the snapshot +and l as the sequence of transactions that are applied to the tree between +the time the snapshot begins and the time the snapshot completes, we write +to disk T+l' where l' is a subset of the transactions in l. While we do not +have a way of figuring out which transactions make up l', it doesn't really +matter. T+l'+l = T+l since the transactions we log are idempotent (applying +the transaction multiple times has the same result as applying the transaction +once). So when we restore the snapshot we also play all transactions in the log +that occur after the snapshot was begun. We can easily figure out where to +start the replay because we start a new logfile when we start a snapshot. Both +the snapshot file and log file have a numeric suffix that represent the +transaction id that created the respective files. + +

Session tracking

+Rather than tracking sessions exactly, we track them in batches. That are +processed at fixed intervals. This is easier to implement than exact +session tracking and it is more efficient in terms of performance. It also +provides a small grace period for session renewal. +

+ +

+

+
+
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview   Package  Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV PACKAGE  + NEXT PACKAGE + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-tree.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-tree.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-tree.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,153 @@ + + + + + + +org.apache.zookeeper.server Class Hierarchy (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class  Use   Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Hierarchy For Package org.apache.zookeeper.server +

+
+
+
Package Hierarchies:
All Packages
+
+

+Class Hierarchy +

+ +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class  Use   Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-use.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-use.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.10/api/org/apache/zookeeper/server/package-use.html Thu Mar 30 06:26:15 2017 @@ -0,0 +1,144 @@ + + + + + + +Uses of Package org.apache.zookeeper.server (ZooKeeper 3.4.10 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Package
org.apache.zookeeper.server

+
+No usage of org.apache.zookeeper.server +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2017 The Apache Software Foundation + +