hawq-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yo...@apache.org
Subject incubator-hawq-docs git commit: HAWQ-1372 - doc config change without cluster restart for ambari-managed clusters (closes #97)
Date Fri, 10 Mar 2017 20:02:26 GMT
Repository: incubator-hawq-docs
Updated Branches:
  refs/heads/develop 9d1b6017a -> 8a8bb7137


HAWQ-1372 - doc config change without cluster restart for ambari-managed clusters (closes
#97)


Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/commit/8a8bb713
Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/tree/8a8bb713
Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/diff/8a8bb713

Branch: refs/heads/develop
Commit: 8a8bb7137aec20079437da7dd1c054da905bd1bb
Parents: 9d1b601
Author: Lisa Owen <lowen@pivotal.io>
Authored: Fri Mar 10 12:02:19 2017 -0800
Committer: David Yozie <yozie@apache.org>
Committed: Fri Mar 10 12:02:19 2017 -0800

----------------------------------------------------------------------
 markdown/admin/MasterMirroring.html.md.erb      |   5 +-
 markdown/admin/ambari-admin.html.md.erb         | 138 +++++++++++++------
 .../config_hawq_bestpractices.html.md.erb       |   6 +-
 3 files changed, 100 insertions(+), 49 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/8a8bb713/markdown/admin/MasterMirroring.html.md.erb
----------------------------------------------------------------------
diff --git a/markdown/admin/MasterMirroring.html.md.erb b/markdown/admin/MasterMirroring.html.md.erb
index 80aa242..090f882 100644
--- a/markdown/admin/MasterMirroring.html.md.erb
+++ b/markdown/admin/MasterMirroring.html.md.erb
@@ -152,7 +152,10 @@ The standby can become out-of-date if the log synchronization process
between th
 
 To resynchronize the standby with the master:
 
-1. If you use Ambari to manage your cluster, follow the instructions in [Removing the HAWQ
Standby Master](ambari-admin.html#amb-remove-standby).
+1. If you use Ambari to manage your cluster:
+
+    1. Follow the instructions in [Removing the HAWQ Standby Master](ambari-admin.html#amb-remove-standby)
to remove the HAWQ Standby Master node.
+    2. Follow the instructions in [Adding a HAWQ Standby Master](ambari-admin.html#amb-add-standby)
to re-add the Standby Master.
 
 2. If you do not use Ambari, execute the following command on the HAWQ master:
 

http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/8a8bb713/markdown/admin/ambari-admin.html.md.erb
----------------------------------------------------------------------
diff --git a/markdown/admin/ambari-admin.html.md.erb b/markdown/admin/ambari-admin.html.md.erb
index d080b2e..901840a 100644
--- a/markdown/admin/ambari-admin.html.md.erb
+++ b/markdown/admin/ambari-admin.html.md.erb
@@ -182,20 +182,23 @@ First ensure that the new node(s) has been configured per the instructions
found
    4. Click **Confirm Add** to acknowledge the component to add. Click **OK** when the task
completes.
    5. In the Components summary, select **Add > PXF**.
    6. Click **Confirm Add** to acknowledge the component to add. Click **OK** when the task
completes.
-17. (Optional) If you are using hash tables, adjust the **Default buckets for Hash Distributed
tables** setting (`default_hash_table_bucket_number`) on the HAWQ service's **Configs >
Settings** tab. Update this property's value by multiplying the new number of nodes in the
cluster by the appropriate number indicated below.
-
-    |Number of Nodes After Expansion|Suggested default\_hash\_table\_bucket\_number value|
-    |---------------|------------------------------------------|
-    |<= 85|6 \* \#nodes|
-    |\> 85 and <= 102|5 \* \#nodes|
-    |\> 102 and <= 128|4 \* \#nodes|
-    |\> 128 and <= 170|3 \* \#nodes|
-    |\> 170 and <= 256|2 \* \#nodes|
-    |\> 256 and <= 512|1 \* \#nodes|
-    |\> 512|512|
-18.  Ambari requires the HAWQ service to be restarted in order to apply the configuration
changes. If you need to apply the configuration *without* restarting HAWQ (for dynamic cluster
expansion), then you can use the HAWQ CLI commands described in [Manually Updating the HAWQ
Configuration](#manual-config-steps) *instead* of following this step.
-    <br/><br/>Stop and then start the HAWQ service to apply your configuration
changes via Ambari. Select **Service Actions > Stop**, followed by **Service Actions >
Start** to ensure that the HAWQ Master starts before the newly-added segment. During the HAWQ
startup, Ambari exchanges ssh keys for the `gpadmin` user, and applies the new configuration.
-    >**Note:** Do not use the **Restart All** service action to complete this step.
+17. (Optional) If you are using hash tables, adjust the **Default buckets for Hash Distributed
tables** setting (`default_hash_table_bucket_number`).
+
+    1. Open the HAWQ service's **Configs > Settings** tab. Update the `default_hash_table_bucket_number`
property's value by multiplying the new number of nodes in the cluster by the appropriate
number indicated below.
+
+        |Number of Nodes After Expansion|Suggested default\_hash\_table\_bucket\_number value|
+        |---------------|------------------------------------------|
+        |<= 85|6 \* \#nodes|
+        |\> 85 and <= 102|5 \* \#nodes|
+        |\> 102 and <= 128|4 \* \#nodes|
+        |\> 128 and <= 170|3 \* \#nodes|
+        |\> 170 and <= 256|2 \* \#nodes|
+        |\> 256 and <= 512|1 \* \#nodes|
+        |\> 512|512|
+    18.  Ambari requires you to restart the HAWQ service in order to apply the configuration
changes. If you need to apply the configuration *without* restarting HAWQ (for dynamic cluster
expansion), then use the procedure described in [Workaround for Dynamic Cluster Expansion](#manual-config-steps)
*instead* of following this step.
+        <br/><br/>Stop and then start the HAWQ service to apply your configuration
changes via Ambari. Select **Service Actions > Stop**, followed by **Service Actions >
Start** to ensure that the HAWQ Master starts before the newly-added segment. During the HAWQ
startup, Ambari exchanges ssh keys for the `gpadmin` user, and applies the new configuration.
+        **Note:** Do *not* use the **Restart All** service action to complete this step.
+<a id="step-10"></a>
 19.  Consider the impact of rebalancing HDFS to other components, such as HBase, before you
complete this step.
     <br/><br/>Rebalance your HDFS data by selecting the **HDFS** service and
then choosing **Service Actions > Rebalance HDFS**. Follow the Ambari instructions to complete
the rebalance action.
 20.  Speed up the clearing of the metadata cache by first selecting the **HAWQ** service
and then selecting **Service Actions > Clear HAWQ's HDFS Metadata Cache**.
@@ -206,35 +209,36 @@ First ensure that the new node(s) has been configured per the instructions
found
 
 23.  (Optional.) If you enabled temporary password-based authentication while preparing/configuring
your HAWQ host systems, turn off password-based authentication as described in [Apache HAWQ
System Requirements](../requirements/system-requirements.html#topic_pwdlessssh).
 
-#### <a id="manual-config-steps"></a>Manually Updating the HAWQ Configuration
-If you need to expand your HAWQ cluster without restarting the HAWQ service, follow these
steps to manually apply the new HAWQ configuration. (Use these steps *instead* of following
Step 7 in the above procedure.):
+#### <a id="manual-config-steps"></a>Workaround for Dynamic Cluster Expansion

+If you need to expand your HAWQ cluster without restarting the HAWQ service, follow this
procedure to manually apply the HAWQ configuration changes required for this operation. (Perform
these steps *instead of* following Step 9b in the above procedure.)
+
+1. Log in to the HAWQ master host as a HAWQ administrator and source greenplum_path.sh:
+
+    ``` shell
+    $ ssh gpadmin@<master>
+    gpadmin@master$ source /usr/local/hawq/greenplum_path.sh
+    ```
+
+2. Set the configuration parameter to the new value:
+
+    ``` shell
+    gpadmin@master$ hawq config -c default_hash_table_bucket_number -v <new_value>
+    ```
+        
+3. Reload the HAWQ configuration; this operation does not restart the cluster:
+
+    ``` shell
+    gpadmin@master$ hawq stop cluster --reload
+    ```
+    
+4. Verify that the parameter was updated:
 
-1.  Update your configuration to use the new `default_hash_table_bucket_number` value that
you calculated:
-  1. SSH into the HAWQ master host as the `gpadmin` user:
-    ```shell
-    $ ssh gpadmin@<HAWQ_MASTER_HOST>
-    ```
-   2. Source the `greenplum_path.sh` file to update the shell environment:
-    ```shell
-    $ source /usr/local/hawq/greenplum_path.sh
-    ```
-   3. Verify the current value of `default_hash_table_bucket_number`:
-    ```shell
-    $ hawq config -s default_hash_table_bucket_number
-    ```
-   4. Update `default_hash_table_bucket_number` to the new value that you calculated:
-    ```shell
-    $ hawq config -c default_hash_table_bucket_number -v <new_value>
-    ```
-   5. Reload the configuration without restarting the cluster:
-    ```shell
-    $ hawq stop cluster -u
-    ```
-   6. Verify that the `default_hash_table_bucket_number` value was updated:
     ```shell
-    $ hawq config -s default_hash_table_bucket_number
-    ```
-2.  Edit the `/usr/local/hawq/etc/slaves` file and add the new HAWQ hostname(s) to the end
of the file. Separate multiple hosts with new lines. For example, after adding host4 and host5
to a cluster already contains hosts 1-3, the updated file contents would be:
+    gpadmin@master$ hawq config -s default_hash_table_bucket_number
+    ```
+
+
+5.  Edit the `/usr/local/hawq/etc/slaves` file and add the new HAWQ hostname(s) to the end
of the file. Identify each host on a separate line. For example, after adding host4 and host5
to a cluster already containing hosts 1-3, the `slaves` file would appear similar to the following:
 
      ```
      host1
@@ -242,8 +246,9 @@ If you need to expand your HAWQ cluster without restarting the HAWQ service,
fol
      host3
      host4
      host5
-     ```
-3.  Continue with Step 8 in the previous procedure, [Expanding the HAWQ Cluster](#amb-expand).
 When the HAWQ service is ready to be restarted via Ambari, Ambari will refresh the new configurations.
+     ```
+ 
+6.  Continue with [Step 10](#step-10) in the previous procedure, [Expanding the HAWQ Cluster](#amb-expand).
 When the HAWQ service is ready to be restarted via Ambari, Ambari will refresh the new configurations.
 
 ## <a id="amb-activate-standby"></a>Activating the HAWQ Standby Master
 Activating the HAWQ Standby Master promotes the standby host as the new HAWQ Master host.
The previous HAWQ Master configuration is automatically removed from the cluster.
@@ -306,7 +311,7 @@ This service action enables you to remove the HAWQ Standby Master component
in s
 
 ### When to Perform
 * Execute this procedure if you need to decommission or replace theHAWQ Standby Master host.
-* Execute this procedure and then add the HAWQ Standby Master once again, if the HAWQ Standby
Master is unable to synchronize with the HAWQ Master and you need to reinitialize the service.
+* Execute this procedure as a first step when [Resynchronizing the Standby with the Master](MasterMirroring.html#resync_master).
 * Execute this procedure during a scheduled maintenance period, because it requires a restart
of the HAWQ service.
 
 ### Procedure
@@ -320,7 +325,7 @@ This service action enables you to remove the HAWQ Standby Master component
in s
 
 7. Click **Complete** after the Wizard finishes all tasks.
 
-      **Important:** After the Wizard completes, your HAWQ cluster no longer includes a HAWQ
Standby Master host. As a best practice, follow the instructions in [Adding a HAWQ Standby
Master](#amb-add-standby) to configure a new one.
+      **Important:** After the Wizard completes, your HAWQ cluster no longer includes a HAWQ
Standby Master host. As a best practice, follow the instructions in [Adding a HAWQ Standby
Master](#amb-add-standby) to configure a new one.
 
 ## <a id="hdp-upgrade"></a>Upgrading the HDP Stack
 
@@ -368,6 +373,49 @@ All of the listed steps are mandatory. This ensures that HAWQ service
remains fu
     4. Restart HAWQ service to propagate the configuration change to all Ambari agents.
 
     This will synchronize the password on the host machines with the password that you specified
in Ambari.
+    
+## <a id="manual-cfg-norestart"></a>Updating HAWQ Configuration Without Cluster
Restart
+There may be circumstances, such as during dynamic cluster expansion, when you must update
HAWQ configuration parameters but cannot tolerate cluster downtime in your Ambari-managed
cluster. In these situations, perform the following workaround to update and reload HAWQ configuration
changes without a complete cluster restart.
+
+**Note**: Use this procedure for setting or updating only those HAWQ server configuration
parameters with a set classification of `reload`.
+
+1.  Use the Ambari console UI to set and save the desired HAWQ configuration parameter(s)
via the **HAWQ** service **Settings**, **Configs >Settings**, and/or **Configs >Advanced**
tabs. Make sure to note the configuration parameter names and new values. You may need to
hover the mouse over the configuration field to identify the HAWQ parameter name.
+    
+    **Note**: After updating the configuration parameters, do *not* restart the HAWQ service.

+
+1.  Use HAWQ command line utilities to update the same HAWQ configuration parameters:
+    1. Log in to the HAWQ master host as a HAWQ administrator and source greenplum_path.sh:
+
+        ``` shell
+        $ ssh gpadmin@<master>
+        gpadmin@master$ source /usr/local/hawq/greenplum_path.sh
+        ```
+        
+    2. Identify the current value of the HAWQ configuration parameter:
+
+        ``` shell
+        gpadmin@master$ hawq config -s <parameter_name>
+        ```
+        
+    3. Set the configuration parameter to the new value:
+
+        ``` shell
+        gpadmin@master$ hawq config -c <parameter_name> -v <new_value>
+        ```
+        
+        Perform Steps 2 and 3 for each configuration parameter you set or updated via Ambari.
+        
+3. Reload the HAWQ configuration; this operation does not restart the cluster:
+
+    ``` shell
+    gpadmin@master$ hawq stop cluster --reload
+    ```
+    
+4. Verify that each parameter was updated:
+
+    ```shell
+    gpadmin@master$ hawq config -s <parameter_name>
+    ```
 
 ## <a id="gpadmin-setup-alert"></a>Setting Up Alerts
  

http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/8a8bb713/markdown/bestpractices/config_hawq_bestpractices.html.md.erb
----------------------------------------------------------------------
diff --git a/markdown/bestpractices/config_hawq_bestpractices.html.md.erb b/markdown/bestpractices/config_hawq_bestpractices.html.md.erb
index e4b27b0..cba9824 100644
--- a/markdown/bestpractices/config_hawq_bestpractices.html.md.erb
+++ b/markdown/bestpractices/config_hawq_bestpractices.html.md.erb
@@ -22,11 +22,11 @@ under the License.
 -->
 
 
-The configuration parameters for HAWQ are maintained in the `$GPHOME/etc/hawq-site.xml` file.
This file resides on all HAWQ instances and can be modified either via Ambari or by using
the HAWQ command line interface. Use a consistent strategy (Ambari or command line interface)
to maintain the contents of `hawq-site.xml`. 
+HAWQ configuration parameters are maintained in the `$GPHOME/etc/hawq-site.xml` file. This
file resides on all HAWQ instances and is modified either via Ambari or when using the HAWQ
command line interface `hawq config` command. Use a consistent strategy (Ambari or command
line) to maintain the contents of `hawq-site.xml`. 
 
-For Ambari-managed clusters, always use Ambari for configuring `hawq-site.xml` parameters;
restarting or reconfiguring HAWQ via Ambari will overwrite any configuration changes to `hawq-site.xml`
made outside the Ambari interface.
+For Ambari-managed clusters, always use Ambari for configuring `hawq-site.xml` parameters;
restarting or reconfiguring HAWQ via Ambari will overwrite any configuration changes to `hawq-site.xml`
made outside of the Ambari interface. If you cannot tolerate cluster downtime when updating
HAWQ configuration parameters, refer to [Updating HAWQ Configuration Without Cluster Restart](../admin/ambari-admin.html#manual-cfg-norestart)
for a specific procedure for Ambari-managed clusters.
 
 If you manage your cluster using command line tools instead of Ambari, use a consistent `hawq-site.xml`
file to configure your entire cluster. 
 
-**Note:** Do not configure HAWQ parameters through the `postgresql.conf` file. Use only `hawq-site.xml`
to configure your HAWQ cluster. Any parameters defined in `hawq-site.xml` will overwrite parameters
in `postgresql.conf`. 
+**Note:** Do not use the `postgresql.conf` file to configure HAWQ configuration parameters;
HAWQ uses only `hawq-site.xml` for this purpose. 
 


Mime
View raw message