ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sid Wagle" <swa...@hortonworks.com>
Subject Re: Review Request 38186: AMS - stack advisor should recommend topology smarts alongwith optimistic settings
Date Wed, 09 Sep 2015 22:53:35 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38186/#review98289
-----------------------------------------------------------



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/hadoop-metrics2-hbase.properties.j2
(line 42)
<https://reviews.apache.org/r/38186/#comment154634>

    It is fine to hard-code this and set it to 30 seconds since only consumer is Ambari Support
/ Dev.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 238)
<https://reviews.apache.org/r/38186/#comment154635>

    typo in method name.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 284)
<https://reviews.apache.org/r/38186/#comment154640>

    Based on the plot I feel collector heap size might be over subscribed, how does the plot
look with changing coefficient to say 0.5?
    
    Reason is: We only need just enough memory to handle Phoenix SELECT and UPSERT SELECT
queries. The memory utilization for these are controlled by the batchSize config, which we
should tune maybe with version 2 of this patch, still keeping the collector heap small, so
for hbase heap of 4 GB the collector should be 1 GB.
    (I think simple rule could be 25% of hbase heap > 512m)
    
    For a hbase heap of 32 GB, we allocate 8 GB to collector.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 685)
<https://reviews.apache.org/r/38186/#comment154648>

    This is great !
    However, the message suggests that allocate remaining memory to collector and hbase heap.
Can be made little unambiguous by specifiying again, 0.25 * hbase_heap = collector and remainging
to hbase heap.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 761)
<https://reviews.apache.org/r/38186/#comment154654>

    Lets change text to:
    It is not recommended to use root partition for ..


- Sid Wagle


On Sept. 9, 2015, 5:23 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38186/
> -----------------------------------------------------------
> 
> (Updated Sept. 9, 2015, 5:23 p.m.)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
> 
> 
> Bugs: AMBARI-13037
>     https://issues.apache.org/jira/browse/AMBARI-13037
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Goals:
> Metrics Collector is placed on a less busy master
> AMS HBase root dir is set to a separate mount if available, WARN user if the default
mount point say "/" is selected when some other partition is available.
> Set hbase.tmpdir to separate mount or "/" (different from rootdir) if such an option
is available.
> Fine tune memory settings for clusters under 100 nodes but that have resource avaialble,
we see that sometimes 50 node settings are too conservative for a 40 node cluster but they
are fine for a 10 node.
> Tune the metrics collection frequency (This is experimental item, I am not sure this
will help in reducing load)
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-timelineservice/src/main/resources/scripts/ams_query.py
7fea3df 
>   ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/params.py
45fa367 
>   ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/templates/hadoop-metrics2-accumulo.properties.j2
4ab6635 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml
b2cfccd 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
fa30e19 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
245802c 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/hadoop-metrics2-hbase.properties.j2
3f404eb 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
2ee21d4 
>   ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/params.py
dce73b3 
>   ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/templates/flume-metrics2.properties.j2
bc18043 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
2633640 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
78ee5fa 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
fc46aa4 
>   ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py
3bec0d8 
>   ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/templates/storm-metrics2.properties.j2
4553224 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
0fc87a5 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2
f75947e 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py cce0fc5

>   ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 2bb5aad 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py f09488f 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 3f5bc99 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 7443b51 
> 
> Diff: https://reviews.apache.org/r/38186/diff/
> 
> 
> Testing
> -------
> 
> ----------------------------------------------------------------------
> Ran 261 tests in 9.103s
> 
> OK
> ----------------------------------------------------------------------
> Total run:814
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message