brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aledsage <...@git.apache.org>
Subject [GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Date Wed, 22 Jun 2016 19:49:30 GMT
Github user aledsage commented on a diff in the pull request:

    https://github.com/apache/brooklyn-docs/pull/78#discussion_r68122294
  
    --- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
    @@ -0,0 +1,142 @@
    +---
    +title: High Availability (Supplemental)
    +layout: website-normal
    +---
    +
    +This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
    +and provides an example of how to configure a pair of Apache Brooklyn servers to run
in master-standby mode with a shared NFS datastore
    +
    +### Prerequisites
    +- Two VMs (or physical machines) have been provisioned
    +- NFS or another suitable file system has been configured and is available to both VMs*
    +- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both
machines can write to the folder
    +
    +\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS
mount. The recommended option is to use an object
    +store as described in the [Object Store Persistence](./persistence/#object-store-persistence)
documentation. For clarity, a shared NFS folder
    +is assumed in this example
    +
    +### Launching
    +To start, download and install the latest Apache Brooklyn release on both VMs following
the 'OSX / Linux' section
    +of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
    +
    +On the first VM, which will be the master node, run the following to start Brooklyn in
high availability mode:
    +
    +{% highlight bash %}
    +$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
    +{% endhighlight %}
    +
    +Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn
in standby mode:
    +
    +{% highlight bash %}
    +$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
    +{% endhighlight %}
    +
    +### Testing
    +You can now confirm that Brooklyn is running in high availibility mode on the master
by logging into the web console at `http://<ip-address>:8081`.
    +Similarly you can log into the web console on the standby VM where you will see a warning
that the server is not the high availability master.
    +To test a failover, you can simply terminate the process on the first VM and log into
the web console on the second VM. Upon launch, Brooklyn will
    +output its PID to the file `pid.txt`; you can terminate the process by running the following
command from the same directory from which you 
    +launched Brooklyn:
    +
    +{% highlight bash %}
    +$ kill -9 $(cat pid.txt)
    +{% endhighlight %}
    +
    +It is also possiblity to check the high availability state of a running Brooklyn server
using the following curl command:
    +
    +{% highlight bash %}
    +$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
    +{% endhighlight %}
    +
    +This will return one of the following states:
    +
    +{% highlight bash %}
    +
    +"INITIALIZING"
    +"STANDBY"
    +"HOT_STANDBY"
    +"HOT_BACKUP"
    +"MASTER"
    +"FAILED"
    +"TERMINATED"
    +
    +{% endhighlight %}
    +
    +Note: The quotation characters will be included in the reply
    +
    +To obtain information about all of the nodes in the cluster, run the following command
against any of the nodes in the cluster:
    +
    +{% highlight bash %}
    +$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
    +{% endhighlight %}
    +
    +This will return a JSON document describing the Brooklyn nodes in the cluster. An example
of two HA Brooklyn nodes is as follows (whitespace formatting has been
    +added for clarity):
    +
    +{% highlight yaml %}
    +
    +{
    +  ownId: "XkJeXUXE",
    +  masterId: "yAVz0fzo",
    +  nodes: {
    +    yAVz0fzo: {
    +      nodeId: "yAVz0fzo",
    +      nodeUri: "http://<server1-ip-address>:8081/",
    +      status: "MASTER",
    +      localTimestamp: 1466414301065,
    +      remoteTimestamp: 1466414301000
    +    },
    +    XkJeXUXE: {
    +      nodeId: "XkJeXUXE",
    +      nodeUri: "http://<server2-ip-address>:8081/",
    +      status: "STANDBY",
    +      localTimestamp: 1466414301066,
    +      remoteTimestamp: 1466414301000
    +    }
    +  },
    +  links: { }
    +}
    +
    +{% endhighlight %}
    +
    +The examples above show how to use `curl` to manually check the status of Brooklyn via
its REST API. The same REST API calls can also be used by
    +automated third party monitoring tools such as Monit 
    +
    +### Failover
    +When running as a HA standby node, each standby Brooklyn server (in this case there is
only one standby) will check the shared persisted state
    +every 1 second to determine the state of the HA master. If no heartbeat has been recorded
for thirty seconds, then an election will be performed
    +and one of the standby nodes will be promoted to master. At this point all requests should
be directed to the new master node
    +
    +In the event that tasks - such as the provisioning of a new entity - are running when
a failover occurs, the new master will display the current
    +state of the entity, but will not resume its provisioning or re-run any partially completed
tasks. In this case it will usually be necesarry
    +to remove the node and reprovision it
    +
    +### Client Configuration
    +It is the responsibility of the client to connect to the master Brooklyn server. This
can be accomplished in a variety of ways:
    +
    +* **Reverse Proxy**
    +
    +  To allow the client application to automatically fail over in the event of a master
server becoming unavailable, or the promotion of a new master,
    +  a reverse proxy can be configured to route traffic depending on the response returned
by `http://<ip-address>:8081/v1/server/ha/state` (see above).
    +  If a server returns `"MASTER"`, then traffic should be routed to that server, otherwise
it should not be. The client software should be configured
    +  to connect to the reverse proxy server and no action is required by the client in the
event of a failover
    +* **Elastic IP with manual failover**
    --- End diff --
    
    Rename to "Re-allocating an Elastic IP on Failover"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message