Return-Path: X-Original-To: apmail-zookeeper-commits-archive@www.apache.org Delivered-To: apmail-zookeeper-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BF9F375E1 for ; Wed, 23 Nov 2011 08:47:49 +0000 (UTC) Received: (qmail 8066 invoked by uid 500); 23 Nov 2011 08:47:49 -0000 Delivered-To: apmail-zookeeper-commits-archive@zookeeper.apache.org Received: (qmail 8026 invoked by uid 500); 23 Nov 2011 08:47:49 -0000 Mailing-List: contact commits-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ Delivered-To: mailing list commits@zookeeper.apache.org Received: (qmail 7753 invoked by uid 99); 23 Nov 2011 08:47:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Nov 2011 08:47:47 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Nov 2011 08:47:42 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 0A97F2388C7E for ; Wed, 23 Nov 2011 08:46:57 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r799148 [25/41] - in /websites/staging/zookeeper/trunk/content: ./ bookkeeper/docs/trunk/ doc/r3.4.0/ doc/r3.4.0/api/ doc/r3.4.0/api/org/ doc/r3.4.0/api/org/apache/ doc/r3.4.0/api/org/apache/zookeeper/ doc/r3.4.0/api/org/apache/zookeeper/cl... Date: Wed, 23 Nov 2011 08:46:36 -0000 To: commits@zookeeper.apache.org From: buildbot@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111123084657.0A97F2388C7E@eris.apache.org> Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/package-use.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/package-use.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/package-use.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,275 @@ + + + + + + +Uses of Package org.apache.zookeeper (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Package
org.apache.zookeeper

+
+ + + + + + + + + +
+Packages that use org.apache.zookeeper
org.apache.zookeeper  
+  +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Classes in org.apache.zookeeper used by org.apache.zookeeper
AsyncCallback + +
+           
AsyncCallback.ACLCallback + +
+           
AsyncCallback.Children2Callback + +
+           
AsyncCallback.ChildrenCallback + +
+           
AsyncCallback.DataCallback + +
+           
AsyncCallback.StatCallback + +
+           
AsyncCallback.StringCallback + +
+           
AsyncCallback.VoidCallback + +
+           
CreateMode + +
+          CreateMode value determines how the znode is created on ZooKeeper.
KeeperException + +
+           
KeeperException.Code + +
+          Codes which represent the various KeeperException + types.
KeeperException.CodeDeprecated + +
+          Deprecated. 
WatchedEvent + +
+          A WatchedEvent represents a change on the ZooKeeper that a Watcher + is able to respond to.
Watcher + +
+          This interface specifies the public interface an event handler class must + implement.
Watcher.Event.EventType + +
+          Enumeration of types of events that may occur on the ZooKeeper
Watcher.Event.KeeperState + +
+          Enumeration of states the ZooKeeper may be at the event
ZooKeeper + +
+          This is the main class of ZooKeeper client library.
ZooKeeper.States + +
+           
+  +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/LogFormatter.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/LogFormatter.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/LogFormatter.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,255 @@ + + + + + + +LogFormatter (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.zookeeper.server +
+Class LogFormatter

+
+java.lang.Object
+  extended by org.apache.zookeeper.server.LogFormatter
+
+
+
+
public class LogFormatter
extends Object
+ + +

+

+ +

+ + + + + + + + + + + +
+Constructor Summary
LogFormatter() + +
+           
+  + + + + + + + + + + + +
+Method Summary
+static voidmain(String[] args) + +
+           
+  + + + + + + + +
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Constructor Detail
+ +

+LogFormatter

+
+public LogFormatter()
+
+
+ + + + + + + + +
+Method Detail
+ +

+main

+
+public static void main(String[] args)
+                 throws Exception
+
+
+
Parameters:
args - +
Throws: +
Exception
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/PurgeTxnLog.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/PurgeTxnLog.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/PurgeTxnLog.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,296 @@ + + + + + + +PurgeTxnLog (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.zookeeper.server +
+Class PurgeTxnLog

+
+java.lang.Object
+  extended by org.apache.zookeeper.server.PurgeTxnLog
+
+
+
+
public class PurgeTxnLog
extends Object
+ + +

+this class is used to clean up the + snapshot and data log dir's. This is usually + run as a cronjob on the zookeeper server machine. + Invocation of this class will clean up the datalogdir + files and snapdir files keeping the last "-n" snapshot files + and the corresponding logs. +

+ +

+

+ +

+ + + + + + + + + + + +
+Constructor Summary
PurgeTxnLog() + +
+           
+  + + + + + + + + + + + + + + + +
+Method Summary
+static voidmain(String[] args) + +
+           
+static voidpurge(File dataDir, + File snapDir, + int num) + +
+          purges the snapshot and logs keeping the last num snapshots + and the corresponding logs.
+  + + + + + + + +
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Constructor Detail
+ +

+PurgeTxnLog

+
+public PurgeTxnLog()
+
+
+ + + + + + + + +
+Method Detail
+ +

+purge

+
+public static void purge(File dataDir,
+                         File snapDir,
+                         int num)
+                  throws IOException
+
+
purges the snapshot and logs keeping the last num snapshots + and the corresponding logs. +

+

+
Parameters:
dataDir - the dir that has the logs
snapDir - the dir that has the snapshots
num - the number of snapshots to keep +
Throws: +
IOException
+
+
+
+ +

+main

+
+public static void main(String[] args)
+                 throws IOException
+
+
+
Parameters:
args - PurgeTxnLog dataLogDir + dataLogDir -- txn log directory + -n num (number of snapshots to keep) +
Throws: +
IOException
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/ZooKeeperServerMain.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/ZooKeeperServerMain.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/ZooKeeperServerMain.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,280 @@ + + + + + + +ZooKeeperServerMain (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+ +

+ +org.apache.zookeeper.server +
+Class ZooKeeperServerMain

+
+java.lang.Object
+  extended by org.apache.zookeeper.server.ZooKeeperServerMain
+
+
+
+
public class ZooKeeperServerMain
extends Object
+ + +

+This class starts and runs a standalone ZooKeeperServer. +

+ +

+

+ +

+ + + + + + + + + + + +
+Constructor Summary
ZooKeeperServerMain() + +
+           
+  + + + + + + + + + + + + + + + +
+Method Summary
+static voidmain(String[] args) + +
+           
+ voidrunFromConfig(org.apache.zookeeper.server.ServerConfig config) + +
+          Run from a ServerConfig.
+  + + + + + + + +
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
+  +

+ + + + + + + + +
+Constructor Detail
+ +

+ZooKeeperServerMain

+
+public ZooKeeperServerMain()
+
+
+ + + + + + + + +
+Method Detail
+ +

+main

+
+public static void main(String[] args)
+
+
+
+
+
+
+ +

+runFromConfig

+
+public void runFromConfig(org.apache.zookeeper.server.ServerConfig config)
+                   throws IOException
+
+
Run from a ServerConfig. +

+

+
Parameters:
config - ServerConfig to use. +
Throws: +
IOException
+
+
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV CLASS  + NEXT CLASS + FRAMES   + NO FRAMES   +  + + + +
+ SUMMARY: NESTED | FIELD | CONSTR | METHOD +DETAIL: FIELD | CONSTR | METHOD
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/LogFormatter.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/LogFormatter.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/LogFormatter.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.LogFormatter (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.LogFormatter

+
+No usage of org.apache.zookeeper.server.LogFormatter +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/PurgeTxnLog.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/PurgeTxnLog.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/PurgeTxnLog.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.PurgeTxnLog (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.PurgeTxnLog

+
+No usage of org.apache.zookeeper.server.PurgeTxnLog +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/ZooKeeperServerMain.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/ZooKeeperServerMain.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/class-use/ZooKeeperServerMain.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,144 @@ + + + + + + +Uses of Class org.apache.zookeeper.server.ZooKeeperServerMain (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+
+

+Uses of Class
org.apache.zookeeper.server.ZooKeeperServerMain

+
+No usage of org.apache.zookeeper.server.ZooKeeperServerMain +

+

+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview  Package  Class   Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV  + NEXT + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2011 The Apache Software Foundation + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/package-frame.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/package-frame.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/package-frame.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,36 @@ + + + + + + +org.apache.zookeeper.server (ZooKeeper 3.4.0 API) + + + + + + + + + + + +org.apache.zookeeper.server + + + + +
+Classes  + +
+LogFormatter +
+PurgeTxnLog +
+ZooKeeperServerMain
+ + + + Added: websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/package-summary.html ============================================================================== --- websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/package-summary.html (added) +++ websites/staging/zookeeper/trunk/content/doc/r3.4.0/api/org/apache/zookeeper/server/package-summary.html Wed Nov 23 08:46:18 2011 @@ -0,0 +1,260 @@ + + + + + + +org.apache.zookeeper.server (ZooKeeper 3.4.0 API) + + + + + + + + + + + + +
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview   Package  Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV PACKAGE  + NEXT PACKAGE + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+

+Package org.apache.zookeeper.server +

+ZooKeeper server theory of operation +

+See: +
+          Description +

+ + + + + + + + + + + + + + + + + +
+Class Summary
LogFormatter 
PurgeTxnLogthis class is used to clean up the + snapshot and data log dir's.
ZooKeeperServerMainThis class starts and runs a standalone ZooKeeperServer.
+  + +

+

+Package org.apache.zookeeper.server Description +

+ +

+

ZooKeeper server theory of operation

+ZooKeeperServer is designed to work in standalone mode and also +be extensible so that it can be used to implement the quorum based +version of ZooKeeper. +

+ZooKeeper maintains a order when processing requests: +

    +
  • All requests will be processed in order. +
  • All responses will return in order. +
  • All watches will be sent in the order that the update takes place. +
+

+We will explain the three aspects of ZooKeeperServer: request processing, data +structure maintenance, and session tracking. + +

Request processing

+ +Requests are received by the ServerCnxn. Demarshalling of a request is +done by ClientRequestHandler. After a request has been demarshalled, +ClientRequestHandler invokes the relevant method in ZooKeeper and marshals +the result. +

+If the request is just a query, it will be processed by ZooKeeper and returned. +Otherwise, the request will be validated and a transaction will be generated +and logged. This the request will then wait until the request has been logged +before continuing processing. +

+Requests are logged as a group. Transactions are queued up and the SyncThread +will process them at predefined intervals. (Currently 20ms) The SyncThread +interacts with ZooKeeperServer the txnQueue. Transactions are added to the +txnQueue of SyncThread via queueItem. When the transaction has been synced to +disk, its callback will be invoked which will cause the request processing to +be completed. + +

Data structure maintenance

+ +ZooKeeper data is stored in-memory. Each znode is stored in a DataNode object. +This object is accessed through a hash table that maps paths to DataNodes. +DataNodes also organize themselves into a tree. This tree is only used for +serializing nodes. +

+We guarantee that changes to nodes are stored to non-volatile media before +responding to a client. We do this quickly by writing changes as a sequence +of transactions in a log file. Even though we flush transactions as a group, +we need to avoid seeks as much as possible. Also, since the server can fail +at any point, we need to be careful of partial records. +

+We address the above problems by +

    +
  • Pre-allocating 1M chunks of file space. This allows us to append to the +file without causing seeks to update file size. It also means that we need +to check for the end of the log by looking for a zero length transaction +rather than simply end of file. +
  • Writing a signature at the end of each transaction. When processing +transactions, we only use transactions that have a valid signature at the end. +
+

+As the server runs, the log file will grow quite large. To avoid long startup +times we periodically take a snapshot of the tree of DataNodes. We cannot +take the snapshot synchronously as the data takes a while to write out, so +instead we asynchronously write out the tree. This means that we end up +with a "corrupt" snapshot of the data tree. More formally if we define T +to be the real snapshot of the tree at the time we begin taking the snapshot +and l as the sequence of transactions that are applied to the tree between +the time the snapshot begins and the time the snapshot completes, we write +to disk T+l' where l' is a subset of the transactions in l. While we do not +have a way of figuring out which transactions make up l', it doesn't really +matter. T+l'+l = T+l since the transactions we log are idempotent (applying +the transaction multiple times has the same result as applying the transaction +once). So when we restore the snapshot we also play all transactions in the log +that occur after the snapshot was begun. We can easily figure out where to +start the replay because we start a new logfile when we start a snapshot. Both +the snapshot file and log file have a numeric suffix that represent the +transaction id that created the respective files. + +

Session tracking

+Rather than tracking sessions exactly, we track them in batches. That are +processed at fixed intervals. This is easier to implement than exact +session tracking and it is more efficient in terms of performance. It also +provides a small grace period for session renewal. +

+ +

+

+
+
+ + + + + + + + + + + + + + + +
+ + + + + + + + + + + + +
Overview   Package  Class  Use  Tree  Deprecated  Index  Help 
+		 + +
+ PREV PACKAGE  + NEXT PACKAGE + FRAMES   + NO FRAMES   +  + + + +
+ + + +
+Copyright © 2011 The Apache Software Foundation + +