ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-12983) Optimize aggregator queries by performing GROUP BY on server
Date Thu, 03 Sep 2015 20:24:47 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729713#comment-14729713
] 

Hudson commented on AMBARI-12983:
---------------------------------

FAILURE: Integrated in Ambari-branch-2.1 #475 (See [https://builds.apache.org/job/Ambari-branch-2.1/475/])
AMBARI-12983. Optimize aggregator queries by performing GROUP BY on server. (swagle) (swagle:
http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=67cbe2529a0e02c54b7f1e87340fdac98dd93173)
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java
* ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITMetricAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java
* ambari-metrics/ambari-metrics-timelineservice/pom.xml
* ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java
* ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITClusterAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
* ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
* ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java
* ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
* ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java


> Optimize aggregator queries by performing GROUP BY on server
> ------------------------------------------------------------
>
>                 Key: AMBARI-12983
>                 URL: https://issues.apache.org/jira/browse/AMBARI-12983
>             Project: Ambari
>          Issue Type: Task
>          Components: ambari-metrics
>    Affects Versions: 2.1.1
>            Reporter: Siddharth Wagle
>            Assignee: Siddharth Wagle
>            Priority: Critical
>             Fix For: 2.1.2
>
>
> Query to do calculate host level time aggregates.
> *Sample*
> {code}
> select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME,
HOSTNAME, APP_ID from METRIC_RECORD WHERE METRIC_NAME IN ('bytes_in','rpc.rpc.RpcAuthorizationSuccesses')
SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME,
APP_ID limit 50;
> 0: jdbc:phoenix:localhost:61181:/hbase> select SUM(METRIC_SUM), SUM(METRIC_COUNT),
MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE METRIC_NAME
IN ('bytes_in','rpc.rpc.RpcAuthorizationSuccesses') AND SERVER_TIME > 1440106830000 AND
SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID limit 50;
> +-----------------+-------------------+-----------------+-----------------+-------------+------------+------------+
> | SUM(METRIC_SUM) | SUM(METRIC_COUNT) | MAX(METRIC_MAX) | MIN(METRIC_MIN) | METRIC_NAME
|  HOSTNAME  |   APP_ID   |
> +-----------------+-------------------+-----------------+-----------------+-------------+------------+------------+
> | 1546242.5521201263 | 24                | 564689.2744108599 | 616.7147803365921 | bytes_in
   | ams-test-1.c.pramod-thangali.internal | HOST       |
> | 1521023.3303401857 | 24                | 825825.3247578086 | 220.66636815664341 | bytes_in
   | ams-test-2.c.pramod-thangali.internal | HOST       |
> | 153685.92452883872 | 24                | 25389.11630078307 | 448.15683323834327 | bytes_in
   | ams-test-3.c.pramod-thangali.internal | HOST       |
> | 0.0             | 14                | 0.0             | 0.0             | rpc.rpc.RpcAuthorizationSuccesses
| ams-test-1.c.pramod-thangali.internal | datanode   |
> | 17.0            | 14                | 2.0             | 0.0             | rpc.rpc.RpcAuthorizationSuccesses
| ams-test-1.c.pramod-thangali.internal | namenode   |
> | 0.0             | 14                | 0.0             | 0.0             | rpc.rpc.RpcAuthorizationSuccesses
| ams-test-2.c.pramod-thangali.internal | datanode   |
> +-----------------+-------------------+-----------------+-----------------+-------------+------------+------------+
> {code}
> *Query plan*:
> {code}
> 0: jdbc:phoenix:localhost:61181:/hbase> explain select SUM(METRIC_SUM), SUM(METRIC_COUNT),
MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE SERVER_TIME
> 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID;
> +------------+
> |    PLAN    |
> +------------+
> | CLIENT PARALLEL 1-WAY FULL SCAN OVER METRIC_RECORD |
> |     SERVER FILTER BY (SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000)
|
> |     SERVER AGGREGATE INTO DISTINCT ROWS BY [METRIC_NAME, HOSTNAME, APP_ID] |
> | CLIENT MERGE SORT |
> +------------+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message