Return-Path: X-Original-To: apmail-helix-commits-archive@minotaur.apache.org Delivered-To: apmail-helix-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C99FF106B5 for ; Sat, 23 Nov 2013 07:13:26 +0000 (UTC) Received: (qmail 97253 invoked by uid 500); 23 Nov 2013 07:13:19 -0000 Delivered-To: apmail-helix-commits-archive@helix.apache.org Received: (qmail 97153 invoked by uid 500); 23 Nov 2013 07:13:16 -0000 Mailing-List: contact commits-help@helix.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@helix.incubator.apache.org Delivered-To: mailing list commits@helix.incubator.apache.org Received: (qmail 96976 invoked by uid 99); 23 Nov 2013 07:13:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Nov 2013 07:13:08 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Nov 2013 07:13:02 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 872DB2388C8A; Sat, 23 Nov 2013 07:11:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1544749 [18/23] - in /incubator/helix/site-content: ./ apidocs/reference/org/apache/helix/manager/zk/ helix-admin-webapp/ helix-agent/ helix-core/ helix-examples/ images/ involved/ recipes/ recipes/distributed-lock-manager/ recipes/rabbitm... Date: Sat, 23 Nov 2013 07:11:30 -0000 To: commits@helix.incubator.apache.org From: kanak@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20131123071151.872DB2388C8A@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Added: incubator/helix/site-content/tutorial_admin.html URL: http://svn.apache.org/viewvc/incubator/helix/site-content/tutorial_admin.html?rev=1544749&view=auto ============================================================================== --- incubator/helix/site-content/tutorial_admin.html (added) +++ incubator/helix/site-content/tutorial_admin.html Sat Nov 23 07:11:13 2013 @@ -0,0 +1,1010 @@ + + + + + + + + + Apache Helix - Tutorial - Admin Operations + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + + + + + +
  • + + + + +
+ + +
+ +
+
+
+ +
+ + + + + + +
+ +

+

Helix Tutorial: Admin Operations

+

Helix provides a set of admin api for cluster management operations. They are supported via:

+ +
    + +
  • Java API
    • + +
  • Commandline interface
    • + +
  • REST interface via helix-admin-webapp
    • +
+
+
+

Java API

+

See interface org.apache.helix.HelixAdmin

+
+

Command-line interface

+

The command-line tool comes with helix-core package:

+

Get the command-line tool:

+ +
+
  - git clone https://git-wip-us.apache.org/repos/asf/incubator-helix.git
+  - cd incubator-helix
+  - ./build
+  - cd helix-core/target/helix-core-pkg/bin
+  - chmod +x *.sh
+
+

Get help:

+ +
+
  - ./helix-admin.sh --help
+
+

All other commands have this form:

+ +
+
  ./helix-admin.sh --zkSvr <ZookeeperServerAddress> <command> <parameters>
+
+

Admin commands and brief description:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Command syntax Description
--activateCluster <clusterName controllerCluster true/false> Enable/disable a cluster in distributed controller mode
--addCluster <clusterName> Add a new cluster
--addIdealState <clusterName resourceName fileName.json> Add an ideal state to a cluster
--addInstanceTag <clusterName instanceName tag> Add a tag to an instance
--addNode <clusterName instanceId> Add an instance to a cluster
--addResource <clusterName resourceName partitionNumber stateModelName> Add a new resource to a cluster
--addResourceProperty <clusterName resourceName propertyName propertyValue> Add a resource property
--addStateModelDef <clusterName fileName.json> Add a State model definition to a cluster
--dropCluster <clusterName> Delete a cluster
--dropNode <clusterName instanceId> Remove a node from a cluster
--dropResource <clusterName resourceName> Remove an existing resource from a cluster
--enableCluster <clusterName true/false> Enable/disable a cluster
--enableInstance <clusterName instanceId true/false> Enable/disable an instance
--enablePartition <true/false clusterName nodeId resourceName partitionName> Enable/disable a partition
--getConfig <configScope configScopeArgs configKeys> Get user configs
--getConstraints <clusterName constraintType> Get constraints
--help print help information
--instanceGroupTag <instanceTag> Specify instance group tag, used with rebalance command
--listClusterInfo <clusterName> Show information of a cluster
--listClusters List all clusters
--listInstanceInfo <clusterName instanceId> Show information of an instance
--listInstances <clusterName> List all instances in a cluster
--listPartitionInfo <clusterName resourceName partitionName> Show information of a partition
--listResourceInfo <clusterName resourceName> Show information of a resource
--listResources <clusterName> List all resources in a cluster
--listStateModel <clusterName stateModelName> Show information of a state model
--listStateModels <clusterName> List all state models in a cluster
--maxPartitionsPerNode <maxPartitionsPerNode> Specify the max partitions per instance, used with addResourceGroup command
--rebalance <clusterName resourceName replicas> Rebalance a resource
--removeConfig <configScope configScopeArgs configKeys> Remove user configs
--removeConstraint <clusterName constraintType constraintId> Remove a constraint
--removeInstanceTag <clusterName instanceId tag> Remove a tag from an instance
--removeResourceProperty <clusterName resourceName propertyName> Remove a resource property
--resetInstance <clusterName instanceId> Reset all erroneous partitions on an instance
--resetPartition <clusterName instanceId resourceName partitionName> Reset an erroneous partition
--resetResource <clusterName resourceName> Reset all erroneous partitions of a resource
--setConfig <configScope configScopeArgs configKeyValueMap> Set user configs
--setConstraint <clusterName constraintType constraintId constraintKeyValueMap> Set a constraint
--swapInstance <clusterName oldInstance newInstance> Swap an old instance with a new instance
--zkSvr <ZookeeperServerAddress> Provide zookeeper address
+
+

REST interface

+

The REST interface comes wit helix-admin-webapp package:

+ +
+
  - git clone https://git-wip-us.apache.org/repos/asf/incubator-helix.git
+  - cd incubator-helix 
+  - ./build
+  - cd helix-admin-webapp/target/helix-admin-webapp-pkg/bin
+  - chmod +x *.sh
+  - ./run-rest-admin.sh --zkSvr <zookeeperAddress> --port <port> // make sure zookeeper is running
+
+
+

URL and support methods

+ +
    + +
  • +

    /clusters

    + +
      + +
    • List all clusters
      • +
    + +
    +
      curl http://localhost:8100/clusters
+
    + +
      + +
    • Add a cluster
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"addCluster","clusterName":"MyCluster"}' -H "Content-Type: application/json" http://localhost:8100/clusters
+
    • + +
  • +

    /clusters/{clusterName}

    + +
      + +
    • List cluster information
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster
+
    + +
      + +
    • Enable/disable a cluster in distributed controller mode
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"activateCluster","grandCluster":"MyControllerCluster","enabled":"true"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster
+
    + +
      + +
    • Remove a cluster
      • +
    + +
    +
      curl -X DELETE http://localhost:8100/clusters/MyCluster
+
    • + +
  • +

    /clusters/{clusterName}/resourceGroups

    + +
      + +
    • List all resources in a cluster
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/resourceGroups
+
    + +
      + +
    • Add a resource to cluster
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"addResource","resourceGroupName":"MyDB","partitions":"8","stateModelDefRef":"MasterSlave" }' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/resourceGroups
+
    • + +
  • +

    /clusters/{clusterName}/resourceGroups/{resourceName}

    + +
      + +
    • List resource information
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB
+
    + +
      + +
    • Drop a resource
      • +
    + +
    +
      curl -X DELETE http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB
+
    + +
      + +
    • Reset all erroneous partitions of a resource
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"resetResource"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB
+
    • + +
  • +

    /clusters/{clusterName}/resourceGroups/{resourceName}/idealState

    + +
      + +
    • Rebalance a resource
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"rebalance","replicas":"3"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB/idealState
+
    + +
      + +
    • Add an ideal state
      • +
    + +
    +
    echo jsonParameters={
+"command":"addIdealState"
+   }&newIdealState={
+  "id" : "MyDB",
+  "simpleFields" : {
+    "IDEAL_STATE_MODE" : "AUTO",
+    "NUM_PARTITIONS" : "8",
+    "REBALANCE_MODE" : "SEMI_AUTO",
+    "REPLICAS" : "0",
+    "STATE_MODEL_DEF_REF" : "MasterSlave",
+    "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
+  },
+  "listFields" : {
+  },
+  "mapFields" : {
+    "MyDB_0" : {
+      "localhost_1001" : "MASTER",
+      "localhost_1002" : "SLAVE"
+    }
+  }
+}
+> newIdealState.json
+curl -d @'./newIdealState.json' -H 'Content-Type: application/json' http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB/idealState
+
    + +
      + +
    • Add resource property
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"addResourceProperty","REBALANCE_TIMER_PERIOD":"500"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB/idealState
+
    • + +
  • +

    /clusters/{clusterName}/resourceGroups/{resourceName}/externalView

    + +
      + +
    • Show resource external view
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/resourceGroups/MyDB/externalView
+
    • + +
  • +

    /clusters/{clusterName}/instances

    + +
      + +
    • List all instances
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/instances
+
    + +
      + +
    • Add an instance
      • +
    + +
    +
    curl -d 'jsonParameters={"command":"addInstance","instanceNames":"localhost_1001"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/instances
+
    + +
      + +
    • Swap an instance
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"swapInstance","oldInstance":"localhost_1001", "newInstance":"localhost_1002"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/instances
+
    • + +
  • +

    /clusters/{clusterName}/instances/{instanceName}

    + +
      + +
    • Show instance information
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/instances/localhost_1001
+
    + +
      + +
    • Enable/disable an instance
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"enableInstance","enabled":"false"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/instances/localhost_1001
+
    + +
      + +
    • Drop an instance
      • +
    + +
    +
      curl -X DELETE http://localhost:8100/clusters/MyCluster/instances/localhost_1001
+
    + +
      + +
    • Disable/enable partitions on an instance
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"enablePartition","resource": "MyDB","partition":"MyDB_0",  "enabled" : "false"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/instances/localhost_1001
+
    + +
      + +
    • Reset an erroneous partition on an instance
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"resetPartition","resource": "MyDB","partition":"MyDB_0"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/instances/localhost_1001
+
    + +
      + +
    • Reset all erroneous partitions on an instance
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"resetInstance"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/instances/localhost_1001
+
    • + +
  • +

    /clusters/{clusterName}/configs

    + +
      + +
    • Get user cluster level config
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/configs/cluster
+
    + +
      + +
    • Set user cluster level config
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"setConfig","configs":"key1=value1,key2=value2"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/configs/cluster
+
    + +
      + +
    • Remove user cluster level config
      • +
    + +
    +
    curl -d 'jsonParameters={"command":"removeConfig","configs":"key1,key2"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/configs/cluster
+
    + +
      + +
    • Get/set/remove user participant level config
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"setConfig","configs":"key1=value1,key2=value2"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/configs/participant/localhost_1001
+
    + +
      + +
    • Get/set/remove resource level config
      • +
    + +
    +
    curl -d 'jsonParameters={"command":"setConfig","configs":"key1=value1,key2=value2"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/configs/resource/MyDB
+
    • + +
  • +

    /clusters/{clusterName}/controller

    + +
      + +
    • Show controller information
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/Controller
+
    + +
      + +
    • Enable/disable cluster
      • +
    + +
    +
      curl -d 'jsonParameters={"command":"enableCluster","enabled":"false"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/Controller
+
    • + +
  • +

    /zkPath/{path}

    + +
      + +
    • Get information for zookeeper path
      • +
    + +
    +
      curl http://localhost:8100/zkPath/MyCluster
+
    • + +
  • +

    /clusters/{clusterName}/StateModelDefs

    + +
      + +
    • Show all state model definitions
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/StateModelDefs
+
    + +
      + +
    • Add a state mdoel definition
      • +
    + +
    +
      echo jsonParameters={
+    "command":"addStateModelDef"
+   }&newStateModelDef={
+      "id" : "OnlineOffline",
+      "simpleFields" : {
+        "INITIAL_STATE" : "OFFLINE"
+      },
+      "listFields" : {
+        "STATE_PRIORITY_LIST" : [ "ONLINE", "OFFLINE", "DROPPED" ],
+        "STATE_TRANSITION_PRIORITYLIST" : [ "OFFLINE-ONLINE", "ONLINE-OFFLINE", "OFFLINE-DROPPED" ]
+      },
+      "mapFields" : {
+        "DROPPED.meta" : {
+          "count" : "-1"
+        },
+        "OFFLINE.meta" : {
+          "count" : "-1"
+        },
+        "OFFLINE.next" : {
+          "DROPPED" : "DROPPED",
+          "ONLINE" : "ONLINE"
+        },
+        "ONLINE.meta" : {
+          "count" : "R"
+        },
+        "ONLINE.next" : {
+          "DROPPED" : "OFFLINE",
+          "OFFLINE" : "OFFLINE"
+        }
+      }
+    }
+    > newStateModelDef.json
+    curl -d @'./untitled.txt' -H 'Content-Type: application/json' http://localhost:8100/clusters/MyCluster/StateModelDefs
+
    • + +
  • +

    /clusters/{clusterName}/StateModelDefs/{stateModelDefName}

    + +
      + +
    • Show a state model definition
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/StateModelDefs/OnlineOffline
+
    • + +
  • +

    /clusters/{clusterName}/constraints/{constraintType}

    + +
      + +
    • Show all contraints
      • +
    + +
    +
      curl http://localhost:8100/clusters/MyCluster/constraints/MESSAGE_CONSTRAINT
+
    + +
      + +
    • Set a contraint
      • +
    + +
    +
       curl -d 'jsonParameters={"constraintAttributes":"RESOURCE=MyDB,CONSTRAINT_VALUE=1"}' -H "Content-Type: application/json" http://localhost:8100/clusters/MyCluster/constraints/MESSAGE_CONSTRAINT/MyConstraint
+
    + +
      + +
    • Remove a constraint
      • +
    + +
    +
      curl -X DELETE http://localhost:8100/clusters/MyCluster/constraints/MESSAGE_CONSTRAINT/MyConstraint
+
    • +
+
+
+ +
+ +
+
+
Copyright © 2013 + The Apache Software Foundation. + All Rights Reserved. + +
+ + +
+
Apache Helix, Apache, the Apache feather logo, and the Apache Helix project logos are trademarks of The Apache Software Foundation. + All other marks mentioned may be trademarks or registered trademarks of their respective owners.
+ Privacy Policy +
+ +

+ + Built by Maven + +

+ +
+
+ + \ No newline at end of file Added: incubator/helix/site-content/tutorial_controller.html URL: http://svn.apache.org/viewvc/incubator/helix/site-content/tutorial_controller.html?rev=1544749&view=auto ============================================================================== --- incubator/helix/site-content/tutorial_controller.html (added) +++ incubator/helix/site-content/tutorial_controller.html Sat Nov 23 07:11:13 2013 @@ -0,0 +1,304 @@ + + + + + + + + + Apache Helix - Tutorial - Controller + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + + + + + +
  • + + + + +
+ + +
+ +
+
+
+ +
+ + + + + + +
+ +

+

Helix Tutorial: Controller

+

Next, let's implement the controller. This is the brain of the cluster. Helix makes sure there is exactly one active controller running the cluster.

+
+
+

Start the Helix agent

+

It requires the following parameters:

+ +
    + +
  • clusterName: A logical name to represent the group of nodes
    • + +
  • instanceName: A logical name of the process creating the manager instance. Generally this is host:port.
    • + +
  • instanceType: Type of the process. This can be one of the following types, in this case use CONTROLLER: + +
      + +
    • CONTROLLER: Process that controls the cluster, any number of controllers can be started but only one will be active at any given time.
      • + +
    • PARTICIPANT: Process that performs the actual task in the distributed system.
      • + +
    • SPECTATOR: Process that observes the changes in the cluster.
      • + +
    • ADMIN: To carry out system admin actions.
      • +
    • + +
  • zkConnectString: Connection string to Zookeeper. This is of the form host1:port1,host2:port2,host3:port3.
    • +
+ +
+
      manager = HelixManagerFactory.getZKHelixManager(clusterName,
+                                                      instanceName,
+                                                      instanceType,
+                                                      zkConnectString);
+
+
+

Controller Code

+

The Controller needs to know about all changes in the cluster. Helix takes care of this with the default implementation. If you need additional functionality, see GenericHelixController on how to configure the pipeline.

+ +
+
      manager = HelixManagerFactory.getZKHelixManager(clusterName,
+                                                          instanceName,
+                                                          InstanceType.CONTROLLER,
+                                                          zkConnectString);
+     manager.connect();
+     GenericHelixController controller = new GenericHelixController();
+     manager.addConfigChangeListener(controller);
+     manager.addLiveInstanceChangeListener(controller);
+     manager.addIdealStateChangeListener(controller);
+     manager.addExternalViewChangeListener(controller);
+     manager.addControllerListener(controller);
+
+

The snippet above shows how the controller is started. You can also start the controller using command line interface.

+ +
+
cd helix/helix-core/target/helix-core-pkg/bin
+./run-helix-controller.sh --zkSvr <Zookeeper ServerAddress (Required)>  --cluster <Cluster name (Required)>
+
+
+

Controller deployment modes

+

Helix provides multiple options to deploy the controller.

+
+

STANDALONE

+

The Controller can be started as a separate process to manage a cluster. This is the recommended approach. However, since one controller can be a single point of failure, multiple controller processes are required for reliability. Even if multiple controllers are running, only one will be actively managing the cluster at any time and is decided by a leader-election process. If the leader fails, another leader will take over managing the cluster.

+

Even though we recommend this method of deployment, it has the drawback of having to manage an additional service for each cluster. See Controller As a Service option.

+
+

EMBEDDED

+

If setting up a separate controller process is not viable, then it is possible to embed the controller as a library in each of the participants.

+
+

CONTROLLER AS A SERVICE

+

One of the cool features we added in Helix is to use a set of controllers to manage a large number of clusters.

+

For example if you have X clusters to be managed, instead of deploying X*3 (3 controllers for fault tolerance) controllers for each cluster, one can deploy just 3 controllers. Each controller can manage X/3 clusters. If any controller fails, the remaining two will manage X/2 clusters.

+
+
+ +
+ +
+
+
Copyright © 2013 + The Apache Software Foundation. + All Rights Reserved. + +
+ + +
+
Apache Helix, Apache, the Apache feather logo, and the Apache Helix project logos are trademarks of The Apache Software Foundation. + All other marks mentioned may be trademarks or registered trademarks of their respective owners.
+ Privacy Policy +
+ +

+ + Built by Maven + +

+ +
+
+ + \ No newline at end of file Added: incubator/helix/site-content/tutorial_health.html URL: http://svn.apache.org/viewvc/incubator/helix/site-content/tutorial_health.html?rev=1544749&view=auto ============================================================================== --- incubator/helix/site-content/tutorial_health.html (added) +++ incubator/helix/site-content/tutorial_health.html Sat Nov 23 07:11:13 2013 @@ -0,0 +1,253 @@ + + + + + + + + + Apache Helix - Tutorial - Customizing Heath Checks + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + + + + + +
  • + + + + +
+ + +
+ +
+
+
+ +
+ + + + + + +
+ +

+

Helix Tutorial: Customizing Health Checks

+

In this chapter, we'll learn how to customize the health check, based on metrics of your distributed system.

+
+
+

Health Checks

+

Note: this in currently in development mode, not yet ready for production.

+

Helix provides the ability for each node in the system to report health metrics on a periodic basis.

+

Helix supports multiple ways to aggregate these metrics:

+ +
    + +
  • SUM
    • + +
  • AVG
    • + +
  • EXPONENTIAL DECAY
    • + +
  • WINDOW
    • +
+

Helix persists the aggregated value only.

+

Applications can define a threshold on the aggregate values according to the SLAs, and when the SLA is violated Helix will fire an alert. Currently Helix only fires an alert, but in a future release we plan to use these metrics to either mark the node dead or load balance the partitions. This feature will be valuable for distributed systems that support multi-tenancy and have a large variation in work load patterns. In addition, this can be used to detect skewed partitions (hotspots) and rebalance the cluster.

+
+
+ +
+ +
+
+
Copyright © 2013 + The Apache Software Foundation. + All Rights Reserved. + +
+ + +
+
Apache Helix, Apache, the Apache feather logo, and the Apache Helix project logos are trademarks of The Apache Software Foundation. + All other marks mentioned may be trademarks or registered trademarks of their respective owners.
+ Privacy Policy +
+ +

+ + Built by Maven + +

+ +
+
+ + \ No newline at end of file Added: incubator/helix/site-content/tutorial_messaging.html URL: http://svn.apache.org/viewvc/incubator/helix/site-content/tutorial_messaging.html?rev=1544749&view=auto ============================================================================== --- incubator/helix/site-content/tutorial_messaging.html (added) +++ incubator/helix/site-content/tutorial_messaging.html Sat Nov 23 07:11:13 2013 @@ -0,0 +1,273 @@ + + + + + + + + + Apache Helix - Tutorial - Messaging + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + + + + + +
  • + + + + +
+ + +
+ +
+
+
+ +
+ + + + + + +
+ +

+

Helix Tutorial: Messaging

+

In this chapter, we'll learn about messaging, a convenient feature in Helix for sending messages between nodes of a cluster. This is an interesting feature which is quite useful in practice. It is common that nodes in a distributed system require a mechanism to interact with each other.

+
+
+

Example: Bootstrapping a Replica

+

Consider a search system where the index replica starts up and it does not have an index. A typical solution is to get the index from a common location, or to copy the index from another replica.

+

Helix provides a messaging API for intra-cluster communication between nodes in the system. Helix provides a mechanism to specify the message recipient in terms of resource, partition, and state rather than specifying hostnames. Helix ensures that the message is delivered to all of the required recipients. In this particular use case, the instance can specify the recipient criteria as all replicas of the desired partition to bootstrap. Since Helix is aware of the global state of the system, it can send the message to appropriate nodes. Once the nodes respond, Helix provides the bootstrapping replica with all the responses.

+

This is a very generic API and can also be used to schedule various periodic tasks in the cluster, such as data backups, log cleanup, etc. System Admins can also perform ad-hoc tasks, such as on-demand backups or a system command (such as rm -rf ;) across all nodes of the cluster

+ +
+
      ClusterMessagingService messagingService = manager.getMessagingService();
+
+      // Construct the Message
+      Message requestBackupUriRequest = new Message(
+          MessageType.USER_DEFINE_MSG, UUID.randomUUID().toString());
+      requestBackupUriRequest
+          .setMsgSubType(BootstrapProcess.REQUEST_BOOTSTRAP_URL);
+      requestBackupUriRequest.setMsgState(MessageState.NEW);
+
+      // Set the Recipient criteria: all nodes that satisfy the criteria will receive the message
+      Criteria recipientCriteria = new Criteria();
+      recipientCriteria.setInstanceName("%");
+      recipientCriteria.setRecipientInstanceType(InstanceType.PARTICIPANT);
+      recipientCriteria.setResource("MyDB");
+      recipientCriteria.setPartition("");
+
+      // Should be processed only by process(es) that are active at the time of sending the message
+      //   This means if the recipient is restarted after message is sent, it will not be processe.
+      recipientCriteria.setSessionSpecific(true);
+
+      // wait for 30 seconds
+      int timeout = 30000;
+
+      // the handler that will be invoked when any recipient responds to the message.
+      BootstrapReplyHandler responseHandler = new BootstrapReplyHandler();
+
+      // this will return only after all recipients respond or after timeout
+      int sentMessageCount = messagingService.sendAndWait(recipientCriteria,
+          requestBackupUriRequest, responseHandler, timeout);
+
+

See HelixManager.DefaultMessagingService in Javadocs for more info.

+
+
+ +
+ +
+
+
Copyright © 2013 + The Apache Software Foundation. + All Rights Reserved. + +
+ + +
+
Apache Helix, Apache, the Apache feather logo, and the Apache Helix project logos are trademarks of The Apache Software Foundation. + All other marks mentioned may be trademarks or registered trademarks of their respective owners.
+ Privacy Policy +
+ +

+ + Built by Maven + +

+ +
+
+ + \ No newline at end of file Added: incubator/helix/site-content/tutorial_participant.html URL: http://svn.apache.org/viewvc/incubator/helix/site-content/tutorial_participant.html?rev=1544749&view=auto ============================================================================== --- incubator/helix/site-content/tutorial_participant.html (added) +++ incubator/helix/site-content/tutorial_participant.html Sat Nov 23 07:11:13 2013 @@ -0,0 +1,325 @@ + + + + + + + + + Apache Helix - Tutorial - Participant + + + + + + + + + + + + + + + + + + +
+
+
+ + + + + + + + + + +
  • + + + + +
+ + +
+ +
+
+
+ +
+ + + + + + +
+ +

+

Helix Tutorial: Participant

+

In this chapter, we'll learn how to implement a Participant, which is a primary functional component of a distributed system.

+
+
+

Start the Helix agent

+

The Helix agent is a common component that connects each system component with the controller.

+

It requires the following parameters:

+ +
    + +
  • clusterName: A logical name to represent the group of nodes
    • + +
  • instanceName: A logical name of the process creating the manager instance. Generally this is host:port.
    • + +
  • instanceType: Type of the process. This can be one of the following types, in this case, use PARTICIPANT + +
      + +
    • CONTROLLER: Process that controls the cluster, any number of controllers can be started but only one will be active at any given time.
      • + +
    • PARTICIPANT: Process that performs the actual task in the distributed system.
      • + +
    • SPECTATOR: Process that observes the changes in the cluster.
      • + +
    • ADMIN: To carry out system admin actions.
      • +
    • + +
  • zkConnectString: Connection string to Zookeeper. This is of the form host1:port1,host2:port2,host3:port3.
    • +
+

After the Helix manager instance is created, only thing that needs to be registered is the state model factory. The methods of the State Model will be called when controller sends transitions to the Participant. In this example, we’ll use the OnlineOffline factory. Other options include:

+ +
    + +
  • MasterSlaveStateModelFactory
    • + +
  • LeaderStandbyStateModelFactory
    • + +
  • BootstrapHandler
    • + +
  • An application defined state model factory
    • +
+ +
+
      manager = HelixManagerFactory.getZKHelixManager(clusterName,
+                                                          instanceName,
+                                                          InstanceType.PARTICIPANT,
+                                                          zkConnectString);
+     StateMachineEngine stateMach = manager.getStateMachineEngine();
+
+     //create a stateModelFactory that returns a statemodel object for each partition. 
+     stateModelFactory = new OnlineOfflineStateModelFactory();     
+     stateMach.registerStateModelFactory(stateModelType, stateModelFactory);
+     manager.connect();
+
+

Helix doesn't know what it means to change from OFFLINE-->ONLINE or ONLINE-->OFFLINE. The following code snippet shows where you insert your system logic for these two state transitions.

+ +
+
public class OnlineOfflineStateModelFactory extends
+        StateModelFactory<StateModel> {
+    @Override
+    public StateModel createNewStateModel(String stateUnitKey) {
+        OnlineOfflineStateModel stateModel = new OnlineOfflineStateModel();
+        return stateModel;
+    }
+    @StateModelInfo(states = "{'OFFLINE','ONLINE'}", initialState = "OFFINE")
+    public static class OnlineOfflineStateModel extends StateModel {
+
+        @Transition(from = "OFFLINE", to = "ONLINE")
+        public void onBecomeOnlineFromOffline(Message message,
+                NotificationContext context) {
+
+            System.out.println("OnlineOfflineStateModel.onBecomeOnlineFromOffline()");
+
+            ////////////////////////////////////////////////////////////////////////////////////////////////
+            // Application logic to handle transition                                                     //
+            // For example, you might start a service, run initialization, etc                            //
+            ////////////////////////////////////////////////////////////////////////////////////////////////
+        }
+
+        @Transition(from = "ONLINE", to = "OFFLINE")
+        public void onBecomeOfflineFromOnline(Message message,
+                NotificationContext context) {
+
+            System.out.println("OnlineOfflineStateModel.onBecomeOfflineFromOnline()");
+
+            ////////////////////////////////////////////////////////////////////////////////////////////////
+            // Application logic to handle transition                                                     //
+            // For example, you might shutdown a service, log this event, or change monitoring settings   //
+            ////////////////////////////////////////////////////////////////////////////////////////////////
+        }
+    }
+}
+
+
+
+ +
+ + + + \ No newline at end of file