Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Content-Type: multipart/alternative;
 boundary="===============6703605055868437886=="
MIME-Version: 1.0
Subject: Re: Review Request 27250: Proper handling exporting of 0.0.0.0
 addresses with Blueprints
From: "John Speidel" <jspeidel@hortonworks.com>
To: "John Speidel" <jspeidel@hortonworks.com>,
 "Nate Cole" <ncole@hortonworks.com>
Cc: "Michael Harp" <michael.harp@teradata.com>,
 "Ambari" <dev@ambari.apache.org>,
 "Robert Nettleton" <rnettleton@hortonworks.com>
Date: Tue, 28 Oct 2014 00:31:36 -0000
Message-ID: <20141028003136.7137.63469@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "John Speidel" <noreply@reviews.apache.org>
References: <20141027225814.7143.15814@reviews.apache.org>
In-Reply-To: <20141027225814.7143.15814@reviews.apache.org>
Reply-To: "John Speidel" <jspeidel@hortonworks.com>

--===============6703605055868437886==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


> On Oct. 27, 2014, 10:58 p.m., Michael Harp wrote:
> > There are several other properties that we typically set to 0.0.0.0, should they be added?
> >             "hbase.master.ipc.address": "0.0.0.0",
> >             "hbase.regionserver.ipc.address": "0.0.0.0"
> >             "dfs.namenode.rpc-bind-host": "0.0.0.0",
> >             "dfs.namenode.http-bind-host": "0.0.0.0",
> >             "dfs.namenode.servicerpc-bind-host": "0.0.0.0",
> >             "dfs.namenode.secondary.http-address": "0.0.0.0:50090"
> >             "yarn.resourcemanager.webapp.address": "0.0.0.0:8088",
> >             "yarn.nodemanager.webapp.bind-host": "0.0.0.0",
> >             "yarn.nodemanager.webapp.https.bind-host": "0.0.0.0",
> >             "yarn.resourcemanager.admin.bind-host": "0.0.0.0",
> >             "yarn.resourcemanager.bind-host": "0.0.0.0",
> >             "yarn.resourcemanager.resource-tracker.bind-host": "0.0.0.0",
> >             "yarn.resourcemanager.scheduler.bind-host": "0.0.0.0",
> >             "yarn.timeline-service.webapp.bind-host": "0.0.0.0",
> >             "yarn.timeline-service.webapp.https.bind-host": "0.0.0.0",
> >             "yarn.timeline-service.bind-host": "0.0.0.0",
> >             "yarn.timeline-service.address": "0.0.0.0:10200"
> >             "mapreduce.jobhistory.admin.bind-host": "0.0.0.0",
> >             "mapreduce.jobhistory.webapp.bind-host": "0.0.0.0",
> >             "mapreduce.jobhistory.webapp.https.bind-host": "0.0.0.0",
> >             "mapreduce.jobhistory.bind-host": "0.0.0.0"

Michael,

For this issue, some service properties were not being exported to blueprint because they were known to contain topology related information (hostname/ip) and there was no host that matched the host/ip in the cluster.  Because of this, the blueprint processor removed these properties from the exported set as it thought that they were pointing to external non-managed hosts.  This patch simply ensures that any of these topology related properties which have a value of "0.0.0.0" will be exported to blueprint.

>From the set of properties that you have included above, only a few are included in any Ambari HDP stack.  For these properties that are included in an Ambari HDP stack, this patch will ensure that they are exported to blueprint.  For the others in the set, the vast majority, they would not have been affected by this issue since they are not "known" to be topology related properties so they wouldn't have be checked to see if they pointed to a managed host.

On a related matter, because of limitations in the currently available stack definition syntax, the blueprint processor has explicit knowledge of which properties are topology related.  These properties are automatically updated to reflect the concrete cluster topology during cluster provisioning via a blueprint.  It is only these properties which are affected by this issue.  The special knowledge baked into the blueprint processor for these properites will be moved to the stack definitions where it belongs in a future release.  This will make it a lot easier to add custom stack definitions.


- John


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27250/#review58711
-----------------------------------------------------------


On Oct. 27, 2014, 9:58 p.m., Robert Nettleton wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27250/
> -----------------------------------------------------------
> 
> (Updated Oct. 27, 2014, 9:58 p.m.)
> 
> 
> Review request for Ambari, John Speidel and Nate Cole.
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch implements a fix for AMBARI-7988.
> 
> The Blueprints config processor does not currently allow service
>   config properties with the IP address of "0.0.0.0" to be
>   included in an exported Blueprint.
> 
> The following properties:
> 
> yarn.timeline-service.address,
> yarn.timeline-service.webapp.address,
> yarn.timeline-service.webapp.https.address
> 
> are not included in an exported Blueprint after a successful
>   cluster deployment the Yarn Application Timeline Service
>   enabled.
> 
> The problem occurs because the default values for these
>   properties is typically "0.0.0.0:portNumber".  This is
>   a special IP address that the current Blueprints processor
>   does no recognize as a valid service address.  Currently,
>   the processor will remove any properties that require
>   host topology information, but don't have any valid hostname
>   included in the property.
> 
> This patch resolves the problem by adding some handling
>   code to the Blueprint export processing.  If a property, such
>   as the ones listed above, uses the default "0.0.0.0" address, then
>   this property is now allowed in the exported blueprint, as this
>   is a valid server address, but is not directly tied to the specific
>   server topology of a given cluster.  If the user overrides these
>   properties, then the hostname substitution process/export process
>   will work as before.
> 
> This patch also implements a new unit test to verify this change.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 4183b9d 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 04c75a1 
> 
> Diff: https://reviews.apache.org/r/27250/diff/
> 
> 
> Testing
> -------
> 
> 1. Ran the ambari-server unit tests with this patch applied (both trunk and 1.7.0), and all unit tests are passing with this change.
> 2. Manually verified the fix on 1.7.0. 
> 3. Manually verified the fix on trunk.
> 
> 
> Thanks,
> 
> Robert Nettleton
> 
>


--===============6703605055868437886==--