Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8F120200C57 for ; Sat, 1 Apr 2017 06:05:16 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8AD42160B8C; Sat, 1 Apr 2017 04:05:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 10D7F160B80 for ; Sat, 1 Apr 2017 06:05:14 +0200 (CEST) Received: (qmail 6990 invoked by uid 500); 1 Apr 2017 04:05:14 -0000 Mailing-List: contact commits-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list commits@asterixdb.apache.org Received: (qmail 6981 invoked by uid 99); 1 Apr 2017 04:05:14 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Apr 2017 04:05:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 14043DFE34; Sat, 1 Apr 2017 04:05:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: buyingyi@apache.org To: commits@asterixdb.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: asterixdb git commit: Update ansible and aws docs. Date: Sat, 1 Apr 2017 04:05:14 +0000 (UTC) archived-at: Sat, 01 Apr 2017 04:05:16 -0000 Repository: asterixdb Updated Branches: refs/heads/master dc1893f96 -> 4d3cec76f Update ansible and aws docs. Change-Id: I2bf4dab747a149f6fd9f1a74cf862043171cf03e Reviewed-on: https://asterix-gerrit.ics.uci.edu/1647 Reviewed-by: Yingyi Bu Sonar-Qube: Jenkins Integration-Tests: Jenkins Tested-by: Jenkins Project: http://git-wip-us.apache.org/repos/asf/asterixdb/repo Commit: http://git-wip-us.apache.org/repos/asf/asterixdb/commit/4d3cec76 Tree: http://git-wip-us.apache.org/repos/asf/asterixdb/tree/4d3cec76 Diff: http://git-wip-us.apache.org/repos/asf/asterixdb/diff/4d3cec76 Branch: refs/heads/master Commit: 4d3cec76fd9ff64099a025c232f8649df21661c9 Parents: dc1893f Author: Till Westmann Authored: Fri Mar 31 18:51:43 2017 -0700 Committer: Yingyi Bu Committed: Fri Mar 31 21:04:52 2017 -0700 ---------------------------------------------------------------------- asterixdb/asterix-doc/pom.xml | 6 + .../src/main/installation/ansible.md | 131 +++++++++++++ .../src/main/installation/ansible_title.md | 24 +++ .../asterix-doc/src/main/installation/aws.md | 192 +++++++++++++++++++ .../src/main/installation/aws_title.md | 23 +++ .../asterix-doc/src/site/markdown/ansible.md | 136 ------------- asterixdb/asterix-doc/src/site/markdown/aws.md | 168 ---------------- 7 files changed, 376 insertions(+), 304 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/pom.xml ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/pom.xml b/asterixdb/asterix-doc/pom.xml index 16db906..b7fc379 100644 --- a/asterixdb/asterix-doc/pom.xml +++ b/asterixdb/asterix-doc/pom.xml @@ -61,6 +61,12 @@ + + + + + + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/src/main/installation/ansible.md ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/src/main/installation/ansible.md b/asterixdb/asterix-doc/src/main/installation/ansible.md new file mode 100644 index 0000000..056871f --- /dev/null +++ b/asterixdb/asterix-doc/src/main/installation/ansible.md @@ -0,0 +1,131 @@ + + +## Introduction +This installation option provides several wrapped [Ansible](https://www.ansible.com/)-based +scripts to deploy, start, stop, and erase an AsterixDB instance on a multi-node cluster without requiring +users to interact with each individual node in the cluster. + +## Prerequisites + * Supported operating systems: **Linux** and **MacOS** + + * Install pip on your client machine: + + CentOS + + $ sudo yum install python-pip + + Ubuntu + + $ sudo apt-get install python-pip + + macOS + + $ brew install pip + + * Install Ansible, boto, and boto3 on your client machine: + + $ pip install ansible + $ pip install boto + $ pip install boto3 + + Note that you might need `sudo` depending on your system configuration. + + **Make sure that the version of Ansible is no less than 2.2.1.0**: + + $ ansible --version + ansible 2.2.1.0 + + * Download the AsterixDB distribution package, unzip it, and navigate to `opt/ansible/` + + $ cd opt/ansible + + The following files and directories are in the directory `opt/ansible`: + + README bin conf yaml + + `bin` contains scripts that deploy, start, stop and erase a multi-node AsterixDB cluster, according to + the configuration specified in files under `conf`, and `yaml` contains internal Ansible scripts that the shell + scripts in `bin` use. + + +## Cluster Configuration + + * **Nodes and account**. Edit the inventory file `conf/inventory` when necessary. + You mostly only need to specify the node DNS names (or IPs) for the cluster controller, i.e., the master node, + in the **[cc]** section, and node controllers, i.e., slave nodes, in the **[ncs]** section. + The following example configures a cluster with two slave nodes (172.0.1.11 and 172.0.1.12) and + one master node (172.0.1.10). + + [cc] + 172.0.1.10 + + [ncs] + 172.0.1.11 + 172.0.1.12 + + **Configure passwordless ssh from your current client that runs the scripts to all nodes listed + in `conf/inventory` as well as `localhost`.** + If the ssh user account for target machines is different from your current username, please uncomment + and edit the following two lines: + + ;[all:vars] + ;ansible_ssh_user= + + If you want to specify advanced Ansible builtin variables, please refer to the + [Ansible documentation](http://docs.ansible.com/ansible/intro_inventory.html). + + * **Remote working directories**. Edit `conf/instance_settings.yml` to change the remote binary directories + when necessary. By default, the binary directory will be under the home directory (as the value of + Ansible builtin variable ansible_env.HOME) of the ssh user account on each node. + + # The name of the product being used. + product: asterixdb + + # The parent directory for the working directory. + basedir: "{{ ansible_env.HOME }}" + + # The working directory. + binarydir: "{{ basedir }}/{{ product }}" + + +## Cluster Lifecycle Management + * Deploy the binary to all nodes: + + $ bin/deploy.sh + + * Every time before starting the AsterixDB cluster, you can edit the instance configuration file + `conf/instance/cc.conf`, except that IP addresses/DNS names are generated and cannot + be changed. All available parameters and their usage can be found [here](ncservice.html#Parameters). + + * Launch your AsterixDB cluster: + + $ bin/start.sh + + Now you can use the multi-node AsterixDB cluster by opening the master node + listed in `conf/inventory` at port `19001` (which can be customized in `conf/instance/cc.conf`) + in your browser. + + * If you want to stop the the multi-node AsterixDB cluster, run the following script: + + $ bin/stop.sh + + * If you want to remove the binary on all nodes, run the following script: + + $ bin/erase.sh http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/src/main/installation/ansible_title.md ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/src/main/installation/ansible_title.md b/asterixdb/asterix-doc/src/main/installation/ansible_title.md new file mode 100644 index 0000000..307580a --- /dev/null +++ b/asterixdb/asterix-doc/src/main/installation/ansible_title.md @@ -0,0 +1,24 @@ + + +* [Introduction](#Introduction) +* [Prerequisites](#Prerequisites) +* [Cluster Configuration](#config) +* [Cluster Lifecycle Management](#lifecycle) + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/src/main/installation/aws.md ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/src/main/installation/aws.md b/asterixdb/asterix-doc/src/main/installation/aws.md new file mode 100644 index 0000000..8b6602b --- /dev/null +++ b/asterixdb/asterix-doc/src/main/installation/aws.md @@ -0,0 +1,192 @@ + + +## Introduction + Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the + Ansible cluster installation scripts as described [here](ansible.html) separately to manage the + lifecycle of an AsterixDB cluster on those EC2 instances. + + However, via this installation option, we provide a combo solution for automating both AWS EC2 + and AsterixDB, where you can run only one script to deploy, start, stop, and terminate + an AsterixDB cluster on AWS. + +## Prerequisites + * Supported operating systems for the client: **Linux** and **MacOS** + + * Supported operating systems for Amazon Web Services instances: **Linux** + + * Install pip on your client machine: + + CentOS + + $ sudo yum install python-pip + + Ubuntu + + $ sudo apt-get install python-pip + + macOS + + $ brew install pip + + * Install Ansible, boto, and boto3 on your client machine: + + $ pip install ansible + $ pip install boto + $ pip install boto3 + + Note that you might need `sudo` depending on your system configuration. + + **Make sure that the version of Ansible is no less than 2.2.1.0**: + + $ ansible --version + ansible 2.2.1.0 + + * Download the AsterixDB distribution package, unzip it, navigate to `opt/aws/` + + $ cd opt/aws + + The following files and directories are in the directory `opt/aws`: + + README bin conf yaml + + `bin` contains scripts that start and terminate an AWS-based cluster instance, according to the configuration + specified in files under `conf`, and `yaml` contains internal Ansible scripts that the shell scripts in `bin` use. + + * Create an AWS account and an IAM user. + + Set up a security group that you'd like to use for your AWS cluster. + **The security group should at least allow all TCP connections from anywhere.** + Provide the name of the security group as the value for the `group` field in `conf/aws_settings.yml`. + + * Retrieve your AWS EC2 key pair name and use that as the `keypair` in `conf/aws_settings.yml`; + + retrieve your AWS IAM `access key ID` and use that as the `access_key_id` in `conf/aws_settings.yml`; + + retrieve your AWS IAM `secret access key` and use that as the `secret_access_key` in `conf/aws_settings.yml`. + + Note that you can only read or download `access key ID` and `secret access key` once from your AWS console. + If you forget them, you have to create new keys and delete the old ones. + + * Configure your ssh setting by editing `~/.ssh/config` and adding the following entry: + + Host *.amazonaws.com + IdentityFile + + Note that \ should be replaced by the path to the file that stores the private key for the + key pair that you uploaded to AWS and used in `conf/aws_settings`. For example: + + Host *.amazonaws.com + IdentityFile ~/.ssh/id_rsa + +### Cluster Configuration + * **AWS settings**. Edit `conf/instance_settings.yml`. The meaning of each parameter is listed as follows: + + # The OS image id for ec2 instances. + image: ami-76fa4116 + + # The data center region for ec2 instances. + region: us-west-2 + + # The tag for each ec2 machine. Use different tags for isolation. + tag: scale_test + + # The name of a security group that appears in your AWS console. + group: default + + # The name of a key pair that appears in your AWS console. + keypair: + + # The AWS access key id for your IAM user. + access_key_id: + + # The AWS secret key for your IAM user. + secret_access_key: + + # The AWS instance type. A full list of available types are listed at: + # https://aws.amazon.com/ec2/instance-types/ + instance_type: t2.micro + + # The number of ec2 instances that construct a cluster. + count: 3 + + # The user name. + user: ec2-user + + # Whether to reuse one slave machine to host the master process. + cc_on_nc: false + + **As described in [prerequisites](#Prerequisites), the following parameters must be customized:** + + # The tag for each ec2 machine. Use different tags for isolation. + tag: scale_test + + # The name of a security group that appears in your AWS console. + group: default + + # The name of a key pair that appears in your AWS console. + keypair: + + # The AWS access key id for your IAM user. + access_key_id: + + # The AWS secrety key for your IAM user. + secret_access_key: + + * **Remote working directories**. Edit `conf/instance_settings.yml` to change the instance binary directories + when necessary. By default, the binary directory will be under the home directory (as the value of + Ansible builtin variable ansible_env.HOME) of the ssh user account on each node. + + # The parent directory for the working directory. + basedir: "{{ ansible_env.HOME }}" + + # The working directory. + binarydir: "{{ basedir }}/{{ product }}" + + +### Cluster Lifecycle Management + * Allocate AWS EC2 nodes (the number of nodes is specified in `conf/instance_settings.yml`) + and deploy the binary to all allocated EC2 nodes: + + bin/deploy.sh + + * Before starting the AsterixDB cluster, you the instance configuration file `conf/instance/cc.conf` + can be modified with the exception of the IP addresses/DNS names which are are generated and cannot + be changed. All available parameters and their usage can be found [here](ncservice.html#Parameters). + + * Launch your AsterixDB cluster on EC2: + + bin/start.sh + + Now you can use the multi-node AsterixDB cluster on EC2 by by opening the master node + listed in `conf/instance/inventory` at port `19001` (which can be customized in `conf/instance/cc.conf`) + in your browser. + + * If you want to stop the AWS-based AsterixDB cluster, run the following script: + + bin/stop.sh + + Note that this only stops AsterixDB but does not stop the EC2 nodes. + + * If you want to terminate the EC2 nodes that run the AsterixDB cluster, run the following script: + + bin/terminate.sh + + **Note that it will destroy everything in the AsterixDB cluster you installed and terminate all EC2 nodes + for the cluster.** http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/src/main/installation/aws_title.md ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/src/main/installation/aws_title.md b/asterixdb/asterix-doc/src/main/installation/aws_title.md new file mode 100644 index 0000000..abf01c9 --- /dev/null +++ b/asterixdb/asterix-doc/src/main/installation/aws_title.md @@ -0,0 +1,23 @@ + + +* [Introduction](#Introduction) +* [Prerequisites](#Prerequisites) +* [Cluster Configuration](#config) +* [Cluster Lifecycle Management](#lifecycle) http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/src/site/markdown/ansible.md ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/src/site/markdown/ansible.md b/asterixdb/asterix-doc/src/site/markdown/ansible.md deleted file mode 100644 index fa3f483..0000000 --- a/asterixdb/asterix-doc/src/site/markdown/ansible.md +++ /dev/null @@ -1,136 +0,0 @@ - - -* [Introduction](#Introduction) -* [Prerequisites](#Prerequisites) -* [Configuration and parameters](#config) -* [Manage the lifecycle of your instance](#lifecycle) - -## Introduction -This installation option wraps the basic, low-level installation binaries described in the [NCService -installation option](ncservice.html), and provides several wrapped [Ansible](https://www.ansible.com/)-based -scripts to deploy, start, stop, and erase an AsterixDB instance on a multi-node cluster without requiring -users to interact with each individual node in the cluster. - -## Prerequisites - * Supported operating systems: **Linux** and **MacOS** - - * Install pip on your client machine: - - CentOS: sudo yum install python-pip - Ubuntu: sudo apt-get install python-pip - MacOS: brew install pip - - * Install Ansible, boto, and boto3 on your client machine: - - pip install ansible - pip install boto - pip install boto3 - - **Make sure that the version of Ansible is no less than 2.2.1.0.** - - * Download a released [simple server package](http://asterixdb.apache.org/download.html). - - Alternatively, you can follow the [instruction](https://github.com/apache/asterixdb#build-from-source) to - build from source. - - * In the extracted directory from the `simple server package`, navigate to `opt/ansible/` - - $cd opt/ansible - - The following files and directories are in the directory `opt/ansible`: - - README bin conf yaml - - `bin` contains scripts that deploy, start, stop and erase a multi-node AsterixDB instance, according to - the configuration specified in files under `conf/`. `yaml` contains internal Ansible scripts that the shell - scripts in `bin` use. - - -## Configuration and parameters - * **Parameters**. Edit the instance configuration file `conf/cc.conf` when necessary. - You can add/update whatever parameters in the **[common]** and **[nc]** sections (except IPs and ports). - For example: - - [common] - log.level=INFO - - [nc] - txn.log.dir=txnlog - iodevices=iodevice - command=asterixnc - - More parameters and their usage can be found [here](ncservice.html#Parameters). - Note that with this installation option, all parameters in the **[cc]** and **[ncservice]** - section will use defaults and cannot be changed. - - - * **Nodes and account**. Edit the inventory file `conf/inventory` when necessary. - You mostly only need to sepecify the node DNS names (or IPs) for the cluster controller, i.e., the master node, - in the **[cc]** section, and node controllers, i.e., slave nodes, in the **[ncs]** section. - The following example configures a cluster with two slave nodes (172.0.1.11 and 172.0.1.12) and - one master node (172.0.1.10). - - [cc] - 172.0.1.10 - - [ncs] - 172.0.1.11 - 172.0.1.12 - - **Configure passwordless ssh from your current client that runs the scripts to all nodes listed - in `conf/inventory`.** - If the ssh user account for target machines is different from your current username, please uncomment - and edit the following two lines: - - ;[all:vars] - ;ansible_ssh_user= - - If you want to specify advanced Ansible builtin variables, please refer to the - [Ansible documentation](http://docs.ansible.com/ansible/intro_inventory.html). - - * **Remote working directories**. Edit `conf/instance_settings.yml` to change the instance binary directories - when necessary. By default, the binary directory will be under the home directory (as the value of - Ansible builtin variable ansible_env.HOME) of the ssh user account on each node. - - # The parent directory for the working directory. - basedir: "{{ ansible_env.HOME }}" - - # The working directory. - binarydir: "{{ basedir }}/{{ product }}" - - -## Manage the lifecycle of your instance - * Deploy the AsterixDB binary to all nodes: - - bin/deploy.sh - - * Launch your cluster instance: - - bin/start.sh - - Now you can use the multi-node AsterixDB instance. - - * If you want to stop the the multi-node AsterixDB instance, run the following script: - - bin/stop.sh - - * If you want to remove the AsterixDB binary on all nodes, run the following script: - - bin/erase.sh http://git-wip-us.apache.org/repos/asf/asterixdb/blob/4d3cec76/asterixdb/asterix-doc/src/site/markdown/aws.md ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-doc/src/site/markdown/aws.md b/asterixdb/asterix-doc/src/site/markdown/aws.md deleted file mode 100644 index 272beb0..0000000 --- a/asterixdb/asterix-doc/src/site/markdown/aws.md +++ /dev/null @@ -1,168 +0,0 @@ - - -* [Introduction](#Introduction) -* [Prerequisites](#Prerequisites) -* [Configuration](#config) -* [Manage the lifecycle of your instance](#lifecycle) - -## Introduction - Note that you can always manually launch a number of Amazon Web Services EC2 instances and then run the - Ansible cluster installation scripts as described [here](ansible.html) separately to manage the - lifecycle of an AsterixDB instance on those EC2 instances. - - However, via this installation option, we provide a combo solution for automating both AWS EC2 - and AsterixDB, where you can run only one script to start/stop an AsterixDB instance on AWS. - -## Prerequisites - * Supported operating systems for the client: **Linux** and **MacOS** - - * Supported operating systems for Amazon Web Services instances: **Linux** - - * Install pip on your client machine: - - CentOS: sudo yum install python-pip - Ubuntu: sudo apt-get install python-pip - MacOS: brew install pip - - * Install Ansible, boto, and boto3 on your client machine: - - pip install ansible - pip install boto - pip install boto3 - - **Make sure that the version of Ansible is no less than 2.2.1.0.** - - * Download a released [simple server package](http://asterixdb.apache.org/download.html). - - Alternatively, you can follow the [instruction](https://github.com/apache/asterixdb#build-from-source) to - build from source. - - * In the extracted directory from the `simple server package`, navigate to `opt/aws/` - - $cd opt/aws - - The following files and directories are in the directory `opt/aws`: - - README bin conf yaml - - `bin` contains scripts that start and terminate an AWS-based cluster instance, according to the configuration - specified in files under `conf/`. `yaml` contains internal Ansible scripts that the shell scripts in `bin` use. - - * Create an AWS account and an IAM user. - - Set up a security group that you'd like to use for your AWS cluster. - **The security group should at least allow all TCP connection from anywhere.** - Fill `group` in `conf/aws_settings.yml` by the name of the security group. - - * Retrieve your AWS EC2 key pair name and fill that for `keypair` `conf/aws_settings.yml`; - - retrieve your AWS IAM `access key ID` and fill that for `access_key_id` in `conf/aws_settings.yml`; - - retrieve your AWS IAM `secret access key` and fill that for `secret_access_key` in `conf/aws_settings.yml`. - - Note that you can only read or download `access key ID` and `secret access key` once from your AWS console. - If you forget them, you have to create new keys again and delete the old ones. - - * Configure your ssh setting by editing `~/.ssh/config` and adding the following entry: - - Host *.amazonaws.com - IdentityFile - - Note that \ should be replaced by the path to the file that stores the private key for the - key pair that you uploaded to AWS and used in `conf/aws_settings`. For example: - - Host *.amazonaws.com - IdentityFile ~/.ssh/id_rsa - -### Configuration - * **AWS settings**. Edit conf/instance_settings.yml. The meaning of each parameter is listed as follows: - - # The OS image id for ec2 instances. - image: ami-76fa4116 - - # The data center region for ec2 instances. - region: us-west-2 - - # The tag for each ec2 machine. - tag: scale_test - - # The name of a security group that appears in your AWS console. - group: default - - # The name of a key pair that appears in your AWS console. - keypair: - - # The AWS access key id for your IAM user. - access_key_id: - - # The AWS secrety key for your IAM user. - secret_access_key: - - # The AWS instance type. A full list of available types are listed at: - # https://aws.amazon.com/ec2/instance-types/ - instance_type: t2.micro - - # The number of ec2 instances that construct a cluster. - count: 3 - - # The user name. - user: ec2-user - - # Whether to reuse one nc machine to host cc. - cc_on_nc: false - - **As described in [prerequisites](#Prerequisites), the following parameters must be customized:** - - # The name of a security group that appears in your AWS console. - group: default - - # The name of a key pair that appears in your AWS console. - keypair: - - # The AWS access key id for your IAM user. - access_key_id: - - # The AWS secrety key for your IAM user. - secret_access_key: - - * **Remote working directories**. Edit conf/instance_settings.yml to change the instance binary directories - when necessary. By default, the binary directory will be under the home directory (as the value of - Ansible builtin variable ansible_env.HOME) of the ssh user account on each node. - - # The parent directory for the working directory. - basedir: "{{ ansible_env.HOME }}" - - # The working directory. - binarydir: "{{ basedir }}/{{ product }}" - - -### Manage the lifecycle of your instance - * Start an AWS-based AsterixDB cluster: - - bin/start.sh - - Now you can use the cluster instance through the public IP or DNS name of the master node. - - * If you want to terminate the AsterixDB instance, run the following script: - - bin/stop.sh - - **Note that it will destroy everything in the AsterixDB instance you installed and terminate all AWS nodes - for the cluster.**