ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Speidel <jspei...@hortonworks.com>
Subject Re: Review Request 43050: Atlas Integration: Support deploying latest Atlas(which depends on kafka) using Ambari
Date Tue, 02 Feb 2016 16:25:44 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43050/#review117413
-----------------------------------------------------------



Although the issue that I raised is important, I think that it is ok to merge the patch as
is and to resolve this issue in a follow up patch.


ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
(line 150)
<https://reviews.apache.org/r/43050/#comment178594>

    The mechanism of setting the topology related properties(kafka and zk) in the agent will
result in issues with blueprint installs.
    
    Because each node in a blueprint provisioned cluster is provisioned independently of other
nodes, the blueprint topology manager needs to block provisioning of all nodes until all hosts
required for configuration topology resolution are known.  This is determined in BlueprintConfigurationProcessor.
 Because these configurations are not registered with the blueprint configuration processor
it is possible for the atlas host to be provisioned prior to the Kafka hosts being known which
result in the failure of the agent to set these topology related properties.
    
    In BlueprintConfigurationProcessor you will need too register updaters for all of the
topology related properties, in this case "atlas.kafka.bootstrap.servers" and "atlas.kafka.zookeeper.connect".
    
    For example:
    atlasPropsMap.put("atlas.kafka.bootstrap.servers", new MultipleHostTopologyUpdater("KAFKA_BROKER"));


- John Speidel


On Feb. 1, 2016, 5:05 p.m., Tom Beerbower wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43050/
> -----------------------------------------------------------
> 
> (Updated Feb. 1, 2016, 5:05 p.m.)
> 
> 
> Review request for Ambari, John Speidel, Nate Cole, and Robert Levas.
> 
> 
> Bugs: AMBARI-14853
>     https://issues.apache.org/jira/browse/AMBARI-14853
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Three additional steps need to be done to to install Atlas 0.6 via Ambari.
> 
> 1. Add new Atlas Kafka related properties to the Atlas configuration ‘application.properties’
> 
>     atlas.notification.embedded" : false,
>     atlas.kafka.data = /tmp 
>     atlas.kafka.bootstrap.servers = c6401.ambari.apache.org:6667
>     atlas.kafka.zookeeper.connect = c6401.ambari.apache.org:2181
>     atlas.kafka.hook.group.id = atlas
>     atlas.kafka.entities.group.id = entities
> 
> 
> * Note: 
> For “atlas.kafka.bootstrap.servers” and “atlas.kafka.zookeeper.connect”, modify
host names based on your cluster topology.  
> The directory specified in “atlas.kaka.data” should exist.
> 
> 2. Add an export of HADOOP_CLASSPATH which includes the required atlas directories to
hive-env.xml in the 2.3 HDP stack
>     export HADOOP_CLASSPATH=/etc/atlas/conf:/usr/hdp/current/atlas-server/hook/hive:${HADOOP_CLASSPATH}
> 
> *Note:
> It is important that the atlas directories are prepended to the existing classpath.
> 
> 3. Restart the Atlas and Hive services after the cluster is fully provisioned
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/application-properties.xml
82dacb6 
>   ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/metainfo.xml 2600fc4

>   ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
1a0c67b 
>   ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
6db42c9 
>   ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
a2131b0 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/hive-env.xml
92c0c03 
>   ambari-server/src/test/python/stacks/2.3/configs/default.json 21bff13 
> 
> Diff: https://reviews.apache.org/r/43050/diff/
> 
> 
> Testing
> -------
> 
> Manual test and verify configuration and Atlas operation.
> 
> mvn clean test : all tests pass
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:02 h
> [INFO] Finished at: 2016-02-01T11:56:24-05:00
> [INFO] Final Memory: 44M/1696M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message