ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Beerbower" <tbeerbo...@hortonworks.com>
Subject Re: Review Request 32414: Ability to set rack awareness
Date Tue, 24 Mar 2015 22:18:39 GMT


> On March 24, 2015, 5:52 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py,
line 288
> > <https://reviews.apache.org/r/32414/diff/1/?file=903411#file903411line288>
> >
> >     Should these be added to Script.py or another class, in case other services
also want to inherit it?
> 
> Tom Beerbower wrote:
>     Thanks for the review!
>     
>     This makes sense.  It really shouldn't be tied to a single service.  
>     
>     Maybe one issue with moving it... The current code accesses some params defined in
the HDFS params.py. 
>     
>     
>         def create_topology_mapping():
>           import params
>     
>           # placing the mappings file in the same folder where the topology script is
located
>           File(params.net_topology_mapping_data_file_path,
>                content=Template("topology_mappings.data.j2"),
>                owner=params.hdfs_user,
>                group=params.user_group
>           )
>           
>     If this code is moved to a common script, where do the params come from?
> 
> Alejandro Fernandez wrote:
>     I'm ok with it as it, as long as a future patch moves it to a shared class. I know
Script.py has some issues with circular dependencies that are kinda ugly right now.

Yeah, I'm looking at that now.  Unfortunately I can't leave it as is.  As Nate pointed out,
this patch will only update the topology mapping on the host that is running NN.  It needs
to be done on any host running a service that is rack sensitive.  So, I need to move it to
a common place and have it run on start/restart.  

Thanks.


- Tom


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32414/#review77604
-----------------------------------------------------------


On March 23, 2015, 8:35 p.m., Tom Beerbower wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32414/
> -----------------------------------------------------------
> 
> (Updated March 23, 2015, 8:35 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, John Speidel, and Nate Cole.
> 
> 
> Bugs: AMBARI-6466
>     https://issues.apache.org/jira/browse/AMBARI-6466
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> 1. Generate topology script used by Hadoop to determine the rack location of nodes.
> 2. Update core-site.xml to point to script.
> 3. Provide mechanism for marking services that need restart when host rack info changes.
> 4. React to host resource rack changes to ...
>     a. mark services needing restart.
>     b. update host/rack data for topology script
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/CustomServiceOrchestrator.py 8c21503 
>   ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
982f10f 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
6d35a36 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
98390fd 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
3a359e5 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java ab1ede5

>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 939a496

>   ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java de84f35

>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/configuration/core-site.xml
c0cd6b3 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 916d9b0

>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/files/topology_script.py
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
615dd54 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py
d20e8e0 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/templates/topology_mappings.data.j2
PRE-CREATION 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 974165c

>   ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
8f7c199 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
208218c 
>   ambari-server/src/test/java/org/apache/ambari/server/state/ServiceInfoTest.java PRE-CREATION

>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 1919958 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.json d1701bd 
>   ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json b3e61bc

>   ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json 4bbb0f5

>   ambari-server/src/test/python/stacks/2.0.6/configs/ha_default.json 652ca7b 
>   ambari-server/src/test/python/stacks/2.0.6/configs/ha_secured.json 6c06f35 
>   ambari-server/src/test/python/stacks/2.0.6/configs/secured.json b0b0c04 
>   ambari-server/src/test/resources/stacks/HDP/2.0.7/services/HDFS/metainfo.xml 69f13e9

> 
> Diff: https://reviews.apache.org/r/32414/diff/
> 
> 
> Testing
> -------
> 
> Manual testing to verify that topology script is configured and available, services (HDFS,
MAPRED2) are marked for restart when a change is made to host rack information, and topology
script can be invoked to get updated rack information.
> 
> New unit tests added.
> 
> All unit tests pass.
> 
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:36 min
> [INFO] Finished at: 2015-03-23T15:47:10-04:00
> [INFO] Final Memory: 43M/600M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message