brooklyn-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From henev...@apache.org
Subject [4/5] incubator-brooklyn git commit: Docs: adds to ops guide
Date Mon, 02 Mar 2015 16:15:06 GMT
Docs: adds to ops guide

- Renames cli.md to “Launching Brooklyn”
- Adds pages/sections for:
  - requirements (server/OS/etc)
  - cloud credentials / setup
  - state backup
  - log file backup
  - more persistence configuration options
  - more jclouds/location config options


Project: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/commit/1422f46a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/tree/1422f46a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/diff/1422f46a

Branch: refs/heads/master
Commit: 1422f46a005cf7b2ad0d4784446a76096f64721b
Parents: 9debe45
Author: Aled Sage <aled.sage@gmail.com>
Authored: Tue Feb 24 12:05:48 2015 +0000
Committer: Aled Sage <aled.sage@gmail.com>
Committed: Tue Feb 24 12:05:48 2015 +0000

----------------------------------------------------------------------
 docs/guide/ops/cli.md                         | 143 ---------------
 docs/guide/ops/index.md                       |   3 +-
 docs/guide/ops/install-on-server.md           |  12 +-
 docs/guide/ops/launch.md                      | 198 +++++++++++++++++++++
 docs/guide/ops/locations/cloud-credentials.md |  85 +++++++++
 docs/guide/ops/locations/index.md             |  62 +++++--
 docs/guide/ops/logging.md                     |  28 ++-
 docs/guide/ops/persistence/index.md           | 105 ++++++++++-
 docs/guide/ops/requirements.md                |  59 ++++++
 9 files changed, 530 insertions(+), 165 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/cli.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/cli.md b/docs/guide/ops/cli.md
deleted file mode 100644
index 3e44c62..0000000
--- a/docs/guide/ops/cli.md
+++ /dev/null
@@ -1,143 +0,0 @@
----
-title: CLI
-layout: website-normal
----
-
-To launch Brooklyn, from the directory where Brooklyn is unpacked, run:
-
-{% highlight bash %}
-% bin/brooklyn launch
-{% endhighlight %}
-
-With no configuration, this will launch the Brooklyn web console and REST API on [`http://localhost:8081/`](http://localhost:8081/).
-No password is set, but the server is listening only on the loopback network interface for
security.
-Once [security is configured](brooklyn_properties.html), Brooklyn will listen on all network
interfaces by default.
-
-You may wish to [add Brooklyn to your path](#path-setup);
-assuming you've done this, to get information the supported CLI options 
-at any time, just run `brooklyn help`:
-
-{% highlight bash %}
-% bin/brooklyn help
-
-usage: brooklyn [(-q | --quiet)] [(-v | --verbose)] <command> [<args>]
-
-The most commonly used brooklyn commands are:
-    help     Display help information about brooklyn
-    info     Display information about brooklyn
-    launch   Starts a brooklyn application. Note that a BROOKLYN_CLASSPATH environment variable
needs to be set up beforehand to point to the user application classpath.
-
-See 'brooklyn help <command>' for more information on a specific command.
-{% endhighlight %}
-
-
-
-## Configuration
-
-Brooklyn can read configuration from a variety of places:
-        
-* the file `~/.brooklyn/brooklyn.properties` (unless `--noGlobalBrooklynProperties` is specified)
-* another file, if the `--localBrooklynProperties <local brooklyn.properties file>`
-* ``-D`` defines on the brooklyn (java) command-line
-* shell environment variables
-
-These properties are described in more detail [here](brooklyn_properties.html).
-
-
-
-## Path Setup
-
-In order to have easy access to the cli it is useful to configure the PATH environment variable
to also point to the cli's bin directory:
-
-{% highlight bash %}
-BROOKLYN_HOME=/path/to/brooklyn/
-export PATH=$PATH:$BROOKLYN_HOME/usage/dist/target/brooklyn-dist/bin/
-{% endhighlight %}
-
-
-## Running from a Source Build
-
-Here is an example of the commands you might run to get the Brooklyn code, 
-compile it and launch an application:
-
-{% highlight bash %}
-git clone https://github.com/apache/incubator-brooklyn.git
-cd brooklyn
-mvn clean install -DskipTests
-BROOKLYN_HOME=$(pwd)
-export PATH=${PATH}:${BROOKLYN_HOME}/usage/dist/target/brooklyn-dist/bin/
-export BROOKLYN_CLASSPATH=${BROOKLYN_HOME}/examples/simple-web-cluster/target/classes
-brooklyn launch --app brooklyn.demo.SingleWebServerExample --location localhost
-{% endhighlight %}
-
-
-## Extending the Classpath
-
-You can add things to the Brooklyn classpath in a number of ways:
-
-* Add ``.jar`` files to Brooklyn's ``./lib/dropins/`` directory. These are added at the end
of the classpath.
-* Add ``.jar`` files to Brooklyn's ``./lib/patch/`` directory. These are added at the front
of the classpath.
-* Add resources to Brooklyn's ``./conf/`` directory. This directory is at the very front
of the classpath.
-* Use the ``BROOKLYN_CLASSPATH`` environment variable. If set, this is prepended to the Brooklyn
classpath.
-
-
-## Cloud Explorer
-
-The `brooklyn` command line tool includes support for querying (and managing) cloud
-compute resources and blob-store resources. 
-
-For example, `brooklyn cloud-compute list-instances --location aws-ec2:eu-west-1`
-will use the AWS credentials from `brooklyn.properties` and list the VM instances
-running in the given EC2 region.
-
-Use `brooklyn help` and `brooklyn help cloud-compute` to find out more information.
-
-This functionality is not intended as a generic cloud management CLI, but instead 
-solves specific Brooklyn use-cases. The main use-case is discovering the valid 
-configuration options on a given cloud, such as for `imageId` and `hardwareId`.
-
-
-### Cloud Compute
-
-The command `brooklyn cloud-compute` has the following options:
-
-* `list-images`: lists VM images within the given cloud, which can be chosen when
-  provisioning new VMs.
-  This is useful for finding the possible values for the `imageId` configuration.
-
-* `get-image <imageId1> <imageId2> ...`: retrieves metadata about the specific
images.
-
-* `list-hardware-profiles`: lists the ids and the details of the hardware profiles
-  available when provisioning. 
-  This is useful for finding the possible values for the `hardwareId` configuration.
-
-* `default-template`: retrieves metadata about the image and hardware profile that will
-  be used by Brooklyn for that location, if no additional configuration options
-  are supplied.
-
-* `list-instances`: lists the VM instances within the given cloud.
-
-* `terminate-instances <instanceId1> <instanceId2> ...`: Terminates the instances
with
-  the given ids.
-
-
-## Blob Store
-
-The command `brooklyn cloud-blobstore` is used to access a given object store, such as S3
-or Swift. It has the following options:
-
-* `list-containers`: lists the containers (i.e. buckets in S3 terminology) within the 
-  given object store.
-
-* `list-container <containerName>`: lists all the blobs (i.e. objects) contained within

-  the given container.
-
-* `blob --container <containerName> --blob <blobName>`: retrieves the given blob
-  (i.e. object), including metadata and its contents.
-
-
-## Other Topics
-
-The CLI arguments for [persistence and HA](persistence/)
-are described separately,
-as is [detailed configuration](brooklyn_properties.html).

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/index.md b/docs/guide/ops/index.md
index 6cfcb49..dae3071 100644
--- a/docs/guide/ops/index.md
+++ b/docs/guide/ops/index.md
@@ -2,8 +2,9 @@
 title: Operations
 layout: website-normal
 children:
+- requirements.md
 - install-on-server.md
-- cli.md
+- launch.md
 - brooklyn_properties.md
 - locations/
 - persistence/

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/install-on-server.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/install-on-server.md b/docs/guide/ops/install-on-server.md
index 6904504..d4711db 100644
--- a/docs/guide/ops/install-on-server.md
+++ b/docs/guide/ops/install-on-server.md
@@ -1,6 +1,6 @@
 ---
 layout: website-normal
-title: Installing on a Server
+title: Installing Brooklyn
 ---
 
 {% include fields.md %}
@@ -10,7 +10,9 @@ Here we present two *alternatives* to install Brooklyn:
 - [Running the *installation script*](#script)
 - [Manual installation](#manual)
 
+
 ## <a id="script"></a> Running the installation script
+
 There is a simple bash script available to help with the installation process. 
 
 #### Script prerequisites
@@ -29,6 +31,7 @@ $ chmod +x ./brooklyn-install.sh
 $ ./brooklyn-install.sh -s -r <your-server-ip>
 {% endhighlight %}
 
+
 ## <a id="manual"></a> Manual installation
 
 1. [Set up the prerequisites](#prerequisites)
@@ -37,6 +40,7 @@ $ ./brooklyn-install.sh -s -r <your-server-ip>
 1. [Configuring catalog.xml](#configuring-catalog)
 1. [Test the installation](#confirm)
 
+
 ### <a id="prerequisites"></a>Set up the prerequisites
 
 Before installing Apache Brooklyn, it is recommented to configure the host as follows. 
@@ -79,7 +83,9 @@ $ BROOKLYN_DIR="$(pwd)"
 $ export PATH=$PATH:$BROOKLYN_DIR/bin/
 {% endhighlight %}
 
+
 ### <a id="configuring-properties"></a>Configuring brooklyn.properties
+
 Brooklyn deploys applications to Locations. *Locations* can be clouds, machines with fixed
IPs or localhost (for testing).
 
 By default Brooklyn loads configuration parameters (including credentials for any cloud accounts)
from 
@@ -98,7 +104,9 @@ $ chmod 600 ~/.brooklyn/brooklyn.properties
 
 You may need to edit `~/.brooklyn/brooklyn.properties` to ensure that brooklyn can access
cloud locations for application deployment.
 
+
 ### <a id="configuring-catalog"></a>Configuring catalog.xml
+
 By default Brooklyn loads the catalog of available application components and services from

 `~/.brooklyn/catalog.xml`. 
 
@@ -110,7 +118,9 @@ The `catalog.xml` is the application blueprint catalog. The above example
file c
 
 You may need to edit `~/.brooklyn/catalog.xml` to update links to any resources for download.
 
+
 ### <a id="confirm"></a>Confirm installation
+
 We can do a quick test drive by launching Brooklyn:
 
 {% highlight bash %}

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/launch.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/launch.md b/docs/guide/ops/launch.md
new file mode 100644
index 0000000..45d67bd
--- /dev/null
+++ b/docs/guide/ops/launch.md
@@ -0,0 +1,198 @@
+---
+title: Launching Brooklyn
+layout: website-normal
+---
+
+## Launch command
+
+To launch Brooklyn, from the directory where Brooklyn is unpacked, run:
+
+{% highlight bash %}
+% nohup bin/brooklyn launch &
+{% endhighlight %}
+
+With no configuration, this will launch the Brooklyn web console and REST API on [`http://localhost:8081/`](http://localhost:8081/).
+No password is set, but the server is listening only on the loopback network interface for
security.
+Once [security is configured](brooklyn_properties.html), Brooklyn will listen on all network
interfaces by default.
+
+You may wish to [add Brooklyn to your path](#path-setup);
+assuming you've done this, to get information the supported CLI options 
+at any time, just run `brooklyn help`:
+
+{% highlight bash %}
+% bin/brooklyn help
+
+usage: brooklyn [(-q | --quiet)] [(-v | --verbose)] <command> [<args>]
+
+The most commonly used brooklyn commands are:
+    help     Display help information about brooklyn
+    info     Display information about brooklyn
+    launch   Starts a brooklyn application. Note that a BROOKLYN_CLASSPATH environment variable
needs to be set up beforehand to point to the user application classpath.
+
+See 'brooklyn help <command>' for more information on a specific command.
+{% endhighlight %}
+
+It is important that Brooklyn is launched with either `nohup ... &` or `... & disown`,
to ensure 
+it keeps running after the shell terminates.
+
+
+### Other CLI Arguments
+
+The CLI arguments for [persistence and HA](persistence/) are described separately.
+
+
+### Path Setup
+
+In order to have easy access to the cli it is useful to configure the PATH environment 
+variable to also point to the cli's bin directory:
+
+{% highlight bash %}
+BROOKLYN_HOME=/path/to/brooklyn/
+export PATH=$PATH:$BROOKLYN_HOME/usage/dist/target/brooklyn-dist/bin/
+{% endhighlight %}
+
+
+### Memory Usage
+
+The amount of memory required by the Brooklyn process depends on the usage 
+- for example the number of entities/VMs under management.
+
+For a standard Brooklyn deployment, the defaults are to start with 256m, and to grow to 1g
of memory.
+These numbers can be overridden by setting the environment variable `JAVA_OPTS` before launching
+the `brooklyn script`:
+
+    JAVA_OPTS=-Xms1g -Xmx1g -XX:MaxPermSize=256m
+
+Brooklyn stores a task history in-memory using [soft references](http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html).
+This means that, once the task history is large, Brooklyn will continually use the maximum
allocated 
+memory. It will only expunge tasks from memory when this space is required for other objects
within the
+Brooklyn process.
+
+
+## Configuration
+
+### Configuration Files
+
+Brooklyn reads configuration from a variety of places. It aggregates the configuration.
+The list below shows increasing precedence (i.e. the later ones will override values
+from earlier ones, if exactly the same property is specified multiple times).
+
+1. `classpath://brooklyn/location-metadata.properties` is shipped as part of Brooklyn, containing

+   generic metadata such as jurisdiction and geographic information about Cloud providers.
       
+1. The file `~/.brooklyn/location-metadata.properties` (unless `--noGlobalBrooklynProperties`
is specified).
+   This is intended to contain custom metadata about additional locations.
+1. The file `~/.brooklyn/brooklyn.properties` (unless `--noGlobalBrooklynProperties` is specified).
+1. Another properties file, if the `--localBrooklynProperties <local brooklyn.properties
file>` is specified.
+1. Shell environment variables
+1. System properties, supplied with ``-D`` on the brooklyn (Java) command-line.
+
+These properties are described in more detail [here](brooklyn_properties.html).
+
+
+### Extending the Classpath
+
+The default Brooklyn directory structure includes:
+
+* `./conf/`: for configuration resources.
+* `./lib/patch/`: for Jar files containing patches.
+* `./lib/brooklyn/`: for the brooklyn libraries.
+* `./lib/dropins/`: for additional Jars.
+
+Resources added to `conf/` will be available on the classpath.
+
+A patch can be applied by adding a Jar to the `lib/patch/` directory, and restarting Brooklyn.
+All jars in this directory will be at the head of the classpath.
+
+Additional Jars should be added to `lib/dropins/`, prior to starting Brooklyn. These jars
will 
+be at the end of the classpath.
+
+The initial classpath, as set in the `brooklyn` script, is:
+
+    conf:lib/patch/*:lib/brooklyn/*:lib/dropins/*
+
+Additional entries can be added at the head of the classpath by setting the environment variable

+`BROOKLYN_CLASSPATH` before running the `brooklyn` script. 
+
+
+### Replacing the web-console
+
+*Work in progress.*
+
+The Brooklyn web-console is loaded from the classpath as the resource `classpath://brooklyn.war`.
+
+To replace this, an alternative WAR with that name can be added at the head of the classpath.
+However, this approach is likely to change in a future release - consider this feature as
"beta".
+
+
+## Cloud Explorer
+
+The `brooklyn` command line tool includes support for querying (and managing) cloud
+compute resources and blob-store resources. 
+
+For example, `brooklyn cloud-compute list-instances --location aws-ec2:eu-west-1`
+will use the AWS credentials from `brooklyn.properties` and list the VM instances
+running in the given EC2 region.
+
+Use `brooklyn help` and `brooklyn help cloud-compute` to find out more information.
+
+This functionality is not intended as a generic cloud management CLI, but instead 
+solves specific Brooklyn use-cases. The main use-case is discovering the valid 
+configuration options on a given cloud, such as for `imageId` and `hardwareId`.
+
+
+### Cloud Compute
+
+The command `brooklyn cloud-compute` has the following options:
+
+* `list-images`: lists VM images within the given cloud, which can be chosen when
+  provisioning new VMs.
+  This is useful for finding the possible values for the `imageId` configuration.
+
+* `get-image <imageId1> <imageId2> ...`: retrieves metadata about the specific
images.
+
+* `list-hardware-profiles`: lists the ids and the details of the hardware profiles
+  available when provisioning. 
+  This is useful for finding the possible values for the `hardwareId` configuration.
+
+* `default-template`: retrieves metadata about the image and hardware profile that will
+  be used by Brooklyn for that location, if no additional configuration options
+  are supplied.
+
+* `list-instances`: lists the VM instances within the given cloud.
+
+* `terminate-instances <instanceId1> <instanceId2> ...`: Terminates the instances
with
+  the given ids.
+
+
+### Blob Store
+
+The command `brooklyn cloud-blobstore` is used to access a given object store, such as S3
+or Swift. It has the following options:
+
+* `list-containers`: lists the containers (i.e. buckets in S3 terminology) within the 
+  given object store.
+
+* `list-container <containerName>`: lists all the blobs (i.e. objects) contained within

+  the given container.
+
+* `blob --container <containerName> --blob <blobName>`: retrieves the given blob
+  (i.e. object), including metadata and its contents.
+
+  
+## Running from a Source Build
+
+Here is an example of the commands you might run to get the Brooklyn code, 
+compile it and launch an application:
+
+{% highlight bash %}
+git clone https://github.com/apache/incubator-brooklyn.git
+cd brooklyn
+mvn clean install -DskipTests
+BROOKLYN_HOME=$(pwd)
+export PATH=${PATH}:${BROOKLYN_HOME}/usage/dist/target/brooklyn-dist/bin/
+export BROOKLYN_CLASSPATH=${BROOKLYN_HOME}/examples/simple-web-cluster/target/classes
+nohup brooklyn launch --app brooklyn.demo.SingleWebServerExample --location localhost &
+{% endhighlight %}
+
+
+  
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/locations/cloud-credentials.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/locations/cloud-credentials.md b/docs/guide/ops/locations/cloud-credentials.md
new file mode 100644
index 0000000..40cc108
--- /dev/null
+++ b/docs/guide/ops/locations/cloud-credentials.md
@@ -0,0 +1,85 @@
+---
+title: Cloud Setup
+layout: website-normal
+---
+
+To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the "identity"
and 
+"credential" in Brooklyn terminology. 
+
+For private clouds (and for some clouds being targeted using a standard API), the "endpoint"
+must also be specified, which is the cloud's URL.
+
+The [jclouds guides](https://jclouds.apache.org/guides) includes documentation on configuring

+different clouds.
+
+
+## AWS
+
+### Credentials
+
+AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential

+respectively.
+
+These keys are the way for any programmatic mechanism to access the AWS API.
+
+To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws)

+and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html).
+
+An example of the expected format is shown below:
+
+    brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST
+    brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm
+
+
+### Tidying up after jclouds
+
+Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see
+https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group
+being deleted until there are no VMs using it. However, there is eventual consistency for
+recording which VMs still reference those security groups. After deleting the VM, it can
sometimes
+take several minutes before the security group can be deleted. jclouds retries for 3 seconds,
but 
+does not block for longer.
+
+There is utility written by Cloudsoft for deleting these unused resources:
+http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds.
+
+
+## Google Compute Engine
+
+### Credentials
+
+Google Compute Engine (GCE) uses a service account e-mail address for the identity, and a
private key 
+as the credential.
+
+To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google).
+
+An example of the expected format is shown below (note the credential is one long line, 
+with `\n` to represent the new line characters):
+
+    brooklyn.location.jclouds.google-compute-engine.identity=123456789012@developer.gserviceaccount.com
+    brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END
RSA PRIVATE KEY-----
+
+
+### Quotas
+
+GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas).
+
+It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en).
+ 
+
+### Networks
+
+GCE accounts often have a limit to the number of networks that can be created. One work around

+is to manually create a network with the required open ports, and to refer to that named
network
+in Brooklyn's location configuration.
+
+To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1).
+
+For example, for dev/demo purposes an "everything" network could be created that opens all
ports.
+
+| Name                        | everything                  |
+| Description                 | opens all tcp ports         |
+| Source IP Ranges            | 0.0.0.0/0                   |
+| Allowed protocols and ports | tcp:0-65535 and udp:0-65535 |
+
+

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/locations/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/locations/index.md b/docs/guide/ops/locations/index.md
index 235a552..18c9810 100644
--- a/docs/guide/ops/locations/index.md
+++ b/docs/guide/ops/locations/index.md
@@ -6,6 +6,7 @@ children:
 - { section: Inheritance and Named Locations, title: Named Locations }
 - { section: Localhost }
 - { section: BYON }
+- cloud-credentials.md
 - more-locations.md
 - ssh-keys.md
 ---
@@ -72,14 +73,22 @@ The credentials for the initial OS log on are typically discovered from
the clou
 but in some environments this is not possible.
 The keys `loginUser` and either `loginUser.password` or `loginUser.privateKeyFile` can be
used to force
 Brooklyn to use specific credentials for the initial login to a cloud-provisioned machine.
-(This is particularly useful when using a custom image templates where the cloud-side account
management logic is not enabled.)
+
+(This custom login is particularly useful when using a custom image templates where the cloud-side
account 
+management logic is not enabled. For example, a vCD template can have guest customization
that will change
+the root password. This setting tells AMP to only use the given password, rather than the
initial 
+randomly generated password that vCD returns. Without this property, there is a race for
such templates:
+does Brooklyn manage to create the admin user before the guest customization changes the
login and reboots,
+or is the password reset first (the latter means Brooklyn can never ssh to the VM). With
this property, 
+Brooklyn will always wait for guest customization to complete before it is able to ssh at
all. In such
+cases, it is also recommended to use `useJcloudsSshInit=false`.)
 
 Following a successful logon, Brooklyn performs the following steps to configure the machine:
 
-1. to create a new user with the same name as the user `brooklyn` is running as locally
-  (this can be overridden with `user`, below)
+1. creates a new user with the same name as the user `brooklyn` is running as locally
+  (this can be overridden with `user`, below).
 
-1. to install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine,
+1. install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine,
    to make it easy for the operator to `ssh` in
    (override with `privateKeyFile`; or if there is no `id_{r,d}sa{,.pub}` an ad hoc keypair
will be generated;
    if there is a passphrase on the key, this must be supplied)  
@@ -128,6 +137,24 @@ For more keys and more detail on the keys below, see
   before making the `Location` available to entities,
   optionally also using `setup.script.vars` (set as `key1:value1,key2:value2`)
 
+- Use `openIptables=true` to automatically configure `iptables`, to open the TCP ports required
by
+  the software process. One can alternatively use `stopIptables=true` to entirely stop the
+  iptables service.
+
+- Use `installDevUrandom=true` to fall back to using `/dev/urandom` rather than `/dev/random`.
This setting
+  is useful for cloud VMs where there is not enough random entropy, which can cause `/dev/random`
to be
+  extremely slow (causing `ssh` to be extremely slow to respond).
+
+- Use `useJcloudsSshInit=false` to disable the use of the native jclouds support for initial
commands executed 
+  on the VM (e.g. for creating new users, setting root passwords, etc.). Instead, Brooklyn's
ssh support will
+  be used. Timeouts and retries are more configurable within Brooklyn itself. Therefore this
option is particularly 
+  recommended when the VM startup is unusual (for example, if guest customizations will cause
reboots and/or will 
+  change login credentials).
+
+- Use `brooklyn.ssh.config.noDeleteAfterExec=true` can be used during dev/test. This prevents
the scripts executed 
+  on the VMs from being deleted on completion. This can help with debugging some issues.
However, the contents of the 
+  scripts and the stdout/stderr of their execution is also available in the AMP debug log.
+
 
 ###### VM Creation
     
@@ -150,12 +177,13 @@ For more keys and more detail on the keys below, see
 - User metadata can be attached, using the syntax ``userMetadata=key=value,key2="value 2"``
 
 - You can specify the number of attempts Brooklyn should make to create
-  machines with ``machineCreateAttempts`` (jclouds only). This is useful for
+  machines with ``machineCreateAttempts`` (jclouds only). This is extremely useful for
   working around the rare occasions in which cloud providers give machines that
   are dead on arrival.
 
-- If you want to investigate failures, set `destroyOnFailure` to `false`
+- If you want to investigate failures, set `destroyOnFailure=false`
   to keep failed VM's around. (You'll have to manually clean them up.)
+  The default is false: if a VM fails to start, or is never ssh'able, then the VM will be
terminated.
 
 
 ### Inheritance and Named Locations
@@ -191,20 +219,21 @@ brooklyn.location.named.AWS\ Virginia\ Large\ Centos.user=root
 brooklyn.location.named.AWS\ Virginia\ Large\ Centos.minRam=4096
 {% endhighlight %}
 
-If you are confused about inheritance order, the following test may be useful:
+The precedence for configuration defined at different levels is that the most value
+defined in the most specific context will apply.
 
+For example, in the example below the config key is repeatedly overridden. If you deploy
+`location: named:my-aws`, Brooklyn will get `VAL5` or `KEY`:
+  
 {% highlight bash %}
-brooklyn.location.jclouds.KEY=VAL1
-brooklyn.location.jclouds.aws-ec2.KEY=VAL2
-brooklyn.location.named.my-aws=jclouds:aws-ec2
-brooklyn.location.named.my-aws.KEY=VAL3
+brooklyn.location.KEY=VAL1
+brooklyn.location.jclouds.KEY=VAL2
+brooklyn.location.jclouds.aws-ec2.KEY=VAL3
+brooklyn.location.jclouds.aws-ec2@us-west-1.KEY=VAL4
+brooklyn.location.named.my-aws=jclouds:aws-ec2:us-west-1
+brooklyn.location.named.my-aws.KEY=VAL5
 {% endhighlight %}
 
-In the above example, if you deploy to `location: named:my-aws`,
-Brooklyn will get `VAL3` for `KEY`;
-this overrides `VAL2` which applies by default to all `aws-ec2` locations,
-which in turn overrides `VAL1`, which applies to all jclouds locations. 
-
 
 ### Localhost
 
@@ -272,3 +301,4 @@ brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyPassphrase=s3cr3tpassph
 
 * [More Locations](more-locations.html)
 * [SSH Keys](ssh-keys.html)
+* [Cloud Credentials](cloud-credentials.md)

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/logging.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/logging.md b/docs/guide/ops/logging.md
index fc1da3c..81d55e9 100644
--- a/docs/guide/ops/logging.md
+++ b/docs/guide/ops/logging.md
@@ -3,7 +3,6 @@ title: Logging
 layout: website-normal
 ---
 
-
 Brooklyn uses the SLF4J logging facade, which allows use of many popular frameworks including
`logback`, 
 `java.util.logging` and `log4j`.
 
@@ -17,6 +16,7 @@ some other problem has occured which the user is expected to attend to
 
 Loggers follow the ``package.ClassName`` naming standard.  
 
+
 ## Standard Configuration
 
 A `logback.xml` file is included in the `conf/` directly of the Brooklyn distro;
@@ -37,6 +37,32 @@ and create a file `logback-debug.xml` based on the
 from that project.
 
 
+## Log File Backup
+
+*This sub-section is a work in progress; feedback from the community is extremely welcome.*
+
+The default rolling log files can be backed up periodically, e.g. using a CRON job.
+
+Note however that the rolling log file naming scheme will rename the historic zipped log
files 
+such that `brooklyn.debug-1.log.zip` is the most recent zipped log file. When the current
+`brooklyn.debug.log` is to be zipped, the previous zip file will be renamed 
+`brooklyn.debug-2.log.zip`. This renaming of files can make RSYNC or backups tricky.
+
+An option is to covert/move the file to a name that includes the last-modified timestamp.

+For example (on mac):
+
+    LOG_FILE=brooklyn.debug-1.log.zip
+    TIMESTAMP=`stat -f '%Um' $LOG_FILE`
+    mv $LOG_FILE /path/to/archive/brooklyn.debug-$TIMESTAMP.log.zip
+
+
+## Logging aggregators
+
+Integration with systems like Logstash and Splunk is possible using standard logback configuration.
+Logback can be configured to [write to the syslog](http://logback.qos.ch/manual/appenders.html#SyslogAppender),

+which can then [feed its logs to Logstash](http://www.logstash.net/docs/1.4.2/inputs/syslog).
+
+
 ## For More Information
 
 The following resources may be useful when configuring logging:

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/persistence/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/persistence/index.md b/docs/guide/ops/persistence/index.md
index 26696aa..1a8fe55 100644
--- a/docs/guide/ops/persistence/index.md
+++ b/docs/guide/ops/persistence/index.md
@@ -7,6 +7,7 @@ children:
 - { section: Object Store Persistence }
 - { section: Rebinding to State }
 - { section: Writing Persistable Code }
+- { section: Persisted State Backup }
 ---
 
 Brooklyn can be configured to persist its state so that the Brooklyn server can be restarted,

@@ -63,17 +64,42 @@ The state is written to the given path. The file structure under that
path is:
 * `./enrichers/`
 
 In each of those directories, an XML file will be created per item - for example a file per
-entity in ./entities/. This file will capture all of the state - for example, an
+entity in `./entities/`. This file will capture all of the state - for example, an
 entity's: id; display name; type; config; attributes; tags; relationships to locations, child

 entities, group membership, policies and enrichers; and dynamically added effectors and sensors.
 
+If using the default persistence dir (i.e. no `--persistenceDir` was specified), then Brooklyn
will
+write its state to `~/.brooklyn/brooklyn-persisted-state/data`. Copies of this directory
+will be automatically created in `~/.brooklyn/brooklyn-persisted-state/backups/` each time
Brooklyn 
+is restarted (or if a standby Brooklyn instances takes over as master).
+
+A custom directory for Brooklyn state can also be configured in `brooklyn.properties` using:
+    
+    # For all Brooklyn files
+    brooklyn.base.dir=/path/to/base/dir
+    
+    # Sub-directory of base.dir for writing persisted state (if relative). If directory
+    # starts with "/" (or "~/", or something like "c:\") then assumed to be absolute. 
+    brooklyn.persistence.dir=data
+
+    # Sub-directory of base.dir for creating backup directories (if relative). If directory
+    # starts with "/" (or "~/", or something like "c:\") then assumed to be absolute. 
+    brooklyn.persistence.backups.dir=backups
+
+This `base.dir` will also include temporary files such as the OSGi cache.
+
+If `persistence.dir` is not specified then it will use the sub-directory
+`brooklyn-persisted-state/data` of the `base.dir`. If the `backups.dir` is not specified
+the backup directories will be created in the sub-directory `backups` of the persistence
dir.
+
 
 Object Store Persistence
 ------------------------
 
 Brooklyn can persist its state to any Object Store API that jclouds supports including 
 S3, Swift and Azure. This gives access to any compatible Object Store product or cloud provider
-including AWS-S3, SoftLayer, Rackspace, HP and Microsoft Azure.
+including AWS-S3, SoftLayer, Rackspace, HP and Microsoft Azure. For a complete list of supported
+providers, see [jclouds](http://jclouds.apache.org/guides/providers/#blobstore-providers).
 
 To configure the Object Store, add the credentials to `~/.brooklyn/brooklyn.properties` such
as:
 
@@ -94,10 +120,28 @@ brooklyn.location.named.softlayer-swift-ams01.credential=abcdefghijklmnopqrstuvw
 Start Brooklyn pointing at this target object store, e.g.:
 
 {% highlight bash %}
-brooklyn launch --persist auto --persistenceDir myContainerName --persistenceLocation named:softlayer-swift-ams01
+nohup brooklyn launch --persist auto --persistenceDir myContainerName --persistenceLocation
named:softlayer-swift-ams01 &
 {% endhighlight %}
 
 
+The following `brooklyn.properties` options can also be used:
+
+    # Location spec string for an object store (e.g. jclouds:swift:URL) where persisted state

+    # should be kept; if blank or not supplied, the file system is used.
+    brooklyn.persistence.location.spec=<location>
+
+    # Container name for writing persisted state
+    brooklyn.persistence.dir=/path/to/dataContainer
+
+    # Location spec string for an object store (e.g. jclouds:swift:URL) where backups of
persisted 
+    # state should be kept; defaults to the local file system.
+    brooklyn.persistence.backups.location.spec=<location>
+
+    # Container name for writing backups of persisted state;
+    # defaults to 'backups' inside the default persistence container.
+    brooklyn.persistence.backups.dir=/path/to/backupContainer
+
+
 Rebinding to State
 ------------------
 
@@ -233,3 +277,58 @@ For locations, policies and enrichers they (currently) do not have attributes.
H
 config is persisted automatically. Normally the state of a policy or enricher is transient
- 
 on rebind it starts afresh, for example with monitoring the performance or health metrics
 rather than relying on the persisted values.
+
+
+Persisted State Backup
+----------------------
+
+### File system backup
+
+When using the file system it is important to ensure it is backed up regularly.
+
+One could use `rsync` to regularly backup the contents to another server.
+
+It is also recommended to periodically create a complete archive of the state.
+A simple mechanism is to run a CRON job periodically (e.g. every 30 minutes) that creates
an
+archive of the persistence directory, and uploads that to a backup 
+facility (e.g. to S3).
+
+Optionally, to avoid excessive load on the Brooklyn server, the archive-generation could
be done 
+on another "data" server. This could get a copy of the data via an `rsync` job.
+
+An example script to be invoked by CRON is shown below:
+
+    DATE=`date "+%Y%m%d.%H%M.%S"`
+    BACKUP_FILENAME=/path/to/archives/back-${DATE}.tar.gz
+    DATA_DIR=/path/to/base/dir/data
+    
+    tar --exclude '*/backups/*' -czvf $BACKUP_FILENAME $DATA_DIR
+    # For s3cmd installation see http://s3tools.org/repositories
+    s3cmd put $BACKUP_FILENAME s3://mybackupbucket
+    rm $BACKUP_FILENAME
+
+
+### Object store backup
+
+Object Stores will normally handle replication. However, many such object stores do not handle

+versioning (i.e. to allow access to an old version, if an object has been incorrectly changed
or 
+deleted).
+
+The state can be downloaded periodically from the object store, archived and backed up. 
+
+An example script to be invoked by CRON is shown below:
+
+    DATE=`date "+%Y%m%d.%H%M.%S"`
+    BACKUP_FILENAME=/path/to/archives/back-${DATE}.tar.gz
+    TEMP_DATA_DIR=/path/to/tempdir
+    
+    amp copy-state \
+            --persistenceLocation named:my-persistence-location \
+            --persistenceDir /path/to/bucket \
+            --destinationDir $TEMP_DATA_DIR
+
+    tar --exclude '*/backups/*' -czvf $BACKUP_FILENAME $TEMP_DATA_DIR
+    # For s3cmd installation see http://s3tools.org/repositories
+    s3cmd put $BACKUP_FILENAME s3://mybackupbucket
+    rm $BACKUP_FILENAME
+    rm -r $TEMP_DATA_DIR

http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/requirements.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/requirements.md b/docs/guide/ops/requirements.md
new file mode 100644
index 0000000..52b96e4
--- /dev/null
+++ b/docs/guide/ops/requirements.md
@@ -0,0 +1,59 @@
+---
+title: Requirements
+layout: website-normal
+---
+
+## Server Specification
+
+The size of server required by Brooklyn depends on the amount of activity. This includes:
+
+* the number of entities/VMs being managed
+* the number of VMs being deployed concurrently
+* the amount of management and monitoring required per entity
+
+For dev/test or when there are only a handful of VMs being managed, a small VM is sufficient.
+For example, an AWS m3.medium with one vCPU, 3.75GiB RAM and 4GB disk.
+
+For larger production uses, a more appropriate machine spec would be two or more cores,
+at least 8GB RAM and 20GB disk. The disk is just for logs, a small amount of persisted state,
and
+any binaries for custom blueprints/integrations.
+
+
+## Supported Operating Systems
+
+The recommended operating system is CentOS 6.x or RedHat 6.x.
+
+Brooklyn has also been tested on Ubuntu 12.04 and OS X.
+
+
+## Software Requirements
+
+Brooklyn requires Java (JRE or JDK), version 6 or version 7. The most recent version 7 is
recommended.
+OpenJDK is recommended. Brooklyn has also been tested on IBM J9 and Oracle's JVM.
+
+* check your `iptables` or other firewall service, making sure that incoming connections
on port 8443 is not blocked
+* check that the [linux kernel entropy](increase-entropy.html) is sufficient
+
+
+## Configuration Requirements
+
+### Ports
+
+The ports used by Brooklyn are:
+
+* 8443 for https, to expose the web-console and REST api.
+* 8081 for http, to expose the web-console and REST api.
+
+Whether to use https rather than http is configurable using the CLI option `--https`; 
+the port to use is configurable using the CLI option `--port <port>`.
+See [CLI](cli.html) documentation for more details.
+
+To enable remote Brooklyn access, ensure these ports are open in the firewall.
+For example, to open port 8443 in iptables, ues the command:
+
+    /sbin/iptables -I INPUT -p TCP --dport 8443 -j ACCEPT
+
+
+### Linux Kernel Entropy
+
+Check that the [linux kernel entropy](increase-entropy.html) is sufficient.


Mime
View raw message