ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aravindan Vijayan" <avija...@hortonworks.com>
Subject Review Request 40929: AMBARI-14140 : Test and Adopt FIFO compaction policy for AMS high load tables
Date Thu, 03 Dec 2015 21:59:41 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40929/
-----------------------------------------------------------

Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-14140
    https://issues.apache.org/jira/browse/AMBARI-14140


Repository: ambari


Description
-------

The following patch adopts the FIFO compaction policy for the AMS high load tables - METRIC_RECORD,
METRIC_RECORD_MINUTE, METRIC_AGGREGATE.

Rationale for adopting FIFO policy
FIFO compaction policy selects only files which have all cells expired. The column family
MUST have non-default TTL.
Since currently AMS does not do any real compaction, it does not use CPU and IO (disk and
network), it does not evict hot data from a block cache. The expected result is improved throughput
and latency both write and read.

Work done
Added UI configs and methods to enable and disable compaction policy. The new UI config is
hbase.fifo.compaction.policy.enabled, which can take true/false values. The enabling/disabling
is done through HBase shell command line. 

If hbase.fifo.compaction.policy.enabled = true
  If 'fifo_enabled' stub file is not present
     enable_fifo_compaction + create stub file

If hbase.fifo.compaction.policy.enabled = false
  If fifo_enabled' stub file is present
     disable_fifo_compaction + remove stub file

The above check has been added to the HBase normalization enabling mechanism as well.


Diffs
-----

  ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector 52fe9ea

  ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog213.java b614b02

  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-env.xml
5abde3f 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
04b7806 
  ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
302b376 
  ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog213Test.java
ee01ab0 

Diff: https://reviews.apache.org/r/40929/diff/


Testing
-------

Manually tested different workflows from the UI.

Submitted patch through Apache.

Relevant unit tests pass. Other unit tests currently running.


Thanks,

Aravindan Vijayan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message