Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Content-Type: multipart/alternative;
 boundary="===============6230490961852620499=="
MIME-Version: 1.0
Subject: Re: Review Request 40929: AMBARI-14140 : Test and Adopt FIFO
 compaction policy for AMS high load tables
From: "Aravindan Vijayan" <avijayan@hortonworks.com>
To: "Sid Wagle" <swagle@hortonworks.com>,
 "Sumit Mohanty" <smohanty@hortonworks.com>,
 "Dmytro Sen" <dsen@hortonworks.com>
Cc: "Aravindan Vijayan" <avijayan@hortonworks.com>,
 "Ambari" <dev@ambari.apache.org>
Date: Fri, 04 Dec 2015 00:55:29 -0000
Message-ID: <20151204005529.1719.79364@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Aravindan Vijayan" <noreply@reviews.apache.org>
References: <20151203215941.1718.59895@reviews.apache.org>
In-Reply-To: <20151203215941.1718.59895@reviews.apache.org>
Reply-To: "Aravindan Vijayan" <avijayan@hortonworks.com>

--===============6230490961852620499==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40929/
-----------------------------------------------------------

(Updated Dec. 4, 2015, 12:55 a.m.)


Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Changes
-------

Changed the AMS HBase normalizer period to 10 minutes. 
Added the AMS HBase normalizer configs in ams-hbase-site to UpgradeCatalog.
Changed stub file location to {AMS_COLLECTOR_PID_DIR}
Changed names of the configs as suggested in review comments. 

Retested manually.


Bugs: AMBARI-14140
    https://issues.apache.org/jira/browse/AMBARI-14140


Repository: ambari


Description
-------

The following patch adopts the FIFO compaction policy for the AMS high load tables - METRIC_RECORD, METRIC_RECORD_MINUTE, METRIC_AGGREGATE.

Rationale for adopting FIFO policy
FIFO compaction policy selects only files which have all cells expired. The column family MUST have non-default TTL.
Since currently AMS does not do any real compaction, it does not use CPU and IO (disk and network), it does not evict hot data from a block cache. The expected result is improved throughput and latency both write and read.

Work done
Added UI configs and methods to enable and disable compaction policy. The new UI config is hbase.fifo.compaction.policy.enabled, which can take true/false values. The enabling/disabling is done through HBase shell command line. 

If hbase.fifo.compaction.policy.enabled = true
  If 'fifo_enabled' stub file is not present
     enable_fifo_compaction + create stub file

If hbase.fifo.compaction.policy.enabled = false
  If fifo_enabled' stub file is present
     disable_fifo_compaction + remove stub file

The above check has been added to the HBase normalization enabling mechanism as well.


Diffs (updated)
-----

  ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector 52fe9ea 
  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog213.java b614b02 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml 5abde3f 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml d91bce1 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml 04b7806 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py 302b376 
  ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog213Test.java ee01ab0 

Diff: https://reviews.apache.org/r/40929/diff/


Testing
-------

Manually tested different workflows from the UI.

Submitted patch through Apache.

Relevant unit tests pass. Other unit tests currently running.


Thanks,

Aravindan Vijayan


--===============6230490961852620499==--