Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D269F200BAE for ; Fri, 14 Oct 2016 05:56:26 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CC465160AF6; Fri, 14 Oct 2016 03:56:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C45F9160AE4 for ; Fri, 14 Oct 2016 05:56:25 +0200 (CEST) Received: (qmail 43010 invoked by uid 500); 14 Oct 2016 03:56:24 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 43001 invoked by uid 99); 14 Oct 2016 03:56:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Oct 2016 03:56:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 729A5C1375 for ; Fri, 14 Oct 2016 03:56:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Nlw-29M3O5a4 for ; Fri, 14 Oct 2016 03:56:22 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id B168B5F119 for ; Fri, 14 Oct 2016 03:56:20 +0000 (UTC) Received: (qmail 42816 invoked by uid 99); 14 Oct 2016 03:56:19 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Oct 2016 03:56:19 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id A454DDFE80; Fri, 14 Oct 2016 03:56:19 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jinhuwu@apache.org To: commits@eagle.incubator.apache.org Message-Id: <140b0cc0c57344aead588dfc5365696a@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: incubator-eagle git commit: [EAGLE-618] migration eagle-jpm-aggregation to application framework Date: Fri, 14 Oct 2016 03:56:19 +0000 (UTC) archived-at: Fri, 14 Oct 2016 03:56:27 -0000 Repository: incubator-eagle Updated Branches: refs/heads/master 0bca234ff -> 7bd5d1d2a [EAGLE-618] migration eagle-jpm-aggregation to application framework Author: wujinhu Closes #504 from wujinhu/EAGLE-618. Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/7bd5d1d2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/7bd5d1d2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/7bd5d1d2 Branch: refs/heads/master Commit: 7bd5d1d2adb909eef7f59424f758fc0818fb683b Parents: 0bca234 Author: wujinhu Authored: Fri Oct 14 11:56:03 2016 +0800 Committer: wujinhu Committed: Fri Oct 14 11:56:03 2016 +0800 ---------------------------------------------------------------------- .../jpm/aggregation/AggregationApplication.java | 18 ++- .../jpm/aggregation/AggregationConfig.java | 10 -- ...gregation.AggregationApplicationProvider.xml | 115 +++++++++++++++++++ .../src/main/resources/application.conf | 23 +--- eagle-topology-assembly/pom.xml | 5 + ...org.apache.eagle.app.spi.ApplicationProvider | 3 +- 6 files changed, 137 insertions(+), 37 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/7bd5d1d2/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationApplication.java ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationApplication.java b/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationApplication.java index 3c40f58..0577070 100644 --- a/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationApplication.java +++ b/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationApplication.java @@ -31,22 +31,30 @@ import java.util.*; public class AggregationApplication extends StormApplication { @Override public StormTopology execute(Config config, StormEnvironment environment) { - AggregationConfig aggregationConfig = AggregationConfig.getInstance(config); - //TODO - List metricNames = config.getStringList("aggregate.counters.metrics"); - List groupByColumns = config.getStringList("aggregate.counters.groupByColumns"); + List metricNames = new ArrayList<>(); + String[] metricNamesArr = config.getString("aggregate.counters.metrics").split(","); + for (int i = 0; i < metricNamesArr.length; i++) { + metricNames.add(metricNamesArr[i]); + } + List groupByColumns = new ArrayList<>(); + String[] groupByColumnsArr = config.getString("aggregate.counters.groupBys").split(","); + for (int i = 0; i < groupByColumnsArr.length; i++) { + groupByColumns.add(groupByColumnsArr[i]); + } + Map>> metrics = new HashMap<>(); for (String metric : metricNames) { metrics.put(metric, new ArrayList<>()); for (String cols : groupByColumns) { - metrics.get(metric).add(Arrays.asList(cols.replaceAll(" ", "").split(","))); + metrics.get(metric).add(Arrays.asList(cols.replaceAll(" ", "").split("&"))); } } TopologyBuilder topologyBuilder = new TopologyBuilder(); String spoutName = "mrHistoryAggregationSpout"; String boltName = "mrHistoryAggregationBolt"; + AggregationConfig aggregationConfig = AggregationConfig.getInstance(config); int parallelism = aggregationConfig.getConfig().getInt("envContextConfig.parallelismConfig." + spoutName); int tasks = aggregationConfig.getConfig().getInt("envContextConfig.tasks." + spoutName); if (parallelism > tasks) { http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/7bd5d1d2/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationConfig.java ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationConfig.java b/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationConfig.java index c50de92..b527ddb 100644 --- a/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationConfig.java +++ b/eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/AggregationConfig.java @@ -26,12 +26,6 @@ import java.io.Serializable; public class AggregationConfig implements Serializable { private static final Logger LOG = LoggerFactory.getLogger(AggregationConfig.class); - public String getEnv() { - return env; - } - - private String env; - public ZKStateConfig getZkStateConfig() { return zkStateConfig; } @@ -67,7 +61,6 @@ public class AggregationConfig implements Serializable { public static class JobExtractorConfig implements Serializable { public String site; - public int readTimeoutSeconds; public long aggregationDuration; } @@ -104,10 +97,8 @@ public class AggregationConfig implements Serializable { */ private void init(Config config) { this.config = config; - this.env = config.getString("envContextConfig.env"); //parse eagle job extractor this.jobExtractorConfig.site = config.getString("jobExtractorConfig.site"); - this.jobExtractorConfig.readTimeoutSeconds = config.getInt("jobExtractorConfig.readTimeOutSeconds"); this.jobExtractorConfig.aggregationDuration = config.getLong("jobExtractorConfig.aggregationDuration"); //parse eagle zk @@ -126,7 +117,6 @@ public class AggregationConfig implements Serializable { this.eagleServiceConfig.password = config.getString("eagleProps.eagleService.password"); LOG.info("Successfully initialized MRHistoryJobConfig"); - LOG.info("env: " + this.env); LOG.info("zookeeper.quorum: " + this.zkStateConfig.zkQuorum); LOG.info("zookeeper.property.clientPort: " + this.zkStateConfig.zkPort); LOG.info("eagle.service.host: " + this.eagleServiceConfig.eagleServiceHost); http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/7bd5d1d2/eagle-jpm/eagle-jpm-aggregation/src/main/resources/META-INF/providers/org.apache.eagle.jpm.aggregation.AggregationApplicationProvider.xml ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-aggregation/src/main/resources/META-INF/providers/org.apache.eagle.jpm.aggregation.AggregationApplicationProvider.xml b/eagle-jpm/eagle-jpm-aggregation/src/main/resources/META-INF/providers/org.apache.eagle.jpm.aggregation.AggregationApplicationProvider.xml new file mode 100644 index 0000000..237a437 --- /dev/null +++ b/eagle-jpm/eagle-jpm-aggregation/src/main/resources/META-INF/providers/org.apache.eagle.jpm.aggregation.AggregationApplicationProvider.xml @@ -0,0 +1,115 @@ + + + + + MR_JPM_AGGREGATION_APP + Map Job Monitoring Aggregation Application + 0.5.0-incubating + + + jobExtractorConfig.site + Site ID + sandbox + + + workers + storm worker number + 4 + + + envContextConfig.parallelismConfig.mrHistoryAggregationSpout + 1 + + + envContextConfig.tasks.mrHistoryAggregationSpout + 1 + + + envContextConfig.parallelismConfig.mrHistoryAggregationBolt + 6 + + + envContextConfig.tasks.mrHistoryAggregationBolt + 6 + + + jobExtractorConfig.aggregationDuration + seconds, each bolt process metrics from [start, start + aggregationDuration] + 3600 + + + zkStateConfig.zkQuorum + sandbox.hortonworks.com:2181 + + + zkStateConfig.zkPort + 2181 + + + zkStateConfig.zkRoot + /aggregation/mr/sandbox + + + zkStateConfig.zkSessionTimeoutMs + 15000 + + + zkStateConfig.zkRetryTimes + 3 + + + zkStateConfig.zkRetryInterval + 20000 + + + eagleProps.eagleService.host + eagleProps.eagleService.host + sandbox.hortonworks.com + + + eagleProps.eagleService.port + eagleProps.eagleService.port + 9099 + + + eagleProps.eagleService.username + eagleProps.eagleService.username + admin + + + eagleProps.eagleService.password + eagleProps.eagleService.password + secret + + + aggregate.counters.metrics + cpu_milliseconds, file_bytes_read, file_bytes_written, hdfs_bytes_read, hdfs_bytes_written, hdfs_read_ops, hdfs_write_ops + + + aggregate.counters.groupBys + groupBys that each metric need to aggregate. If group by many columns, then split them by & + site&jobType, site&user, site + + + + + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/7bd5d1d2/eagle-jpm/eagle-jpm-aggregation/src/main/resources/application.conf ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-aggregation/src/main/resources/application.conf b/eagle-jpm/eagle-jpm-aggregation/src/main/resources/application.conf index 94e06b3..98fc009 100644 --- a/eagle-jpm/eagle-jpm-aggregation/src/main/resources/application.conf +++ b/eagle-jpm/eagle-jpm-aggregation/src/main/resources/application.conf @@ -15,9 +15,6 @@ { "envContextConfig" : { - "env" : "cluster", - "topologyName" : "mrHistoryAggregation", - "stormConfigFile" : "storm.yaml", "parallelismConfig" : { "mrHistoryAggregationSpout" : 1, "mrHistoryAggregationBolt" : 6 @@ -30,7 +27,6 @@ "jobExtractorConfig" : { "site" : "sandbox", - "readTimeOutSeconds" : 10, "aggregationDuration" : 3600 #seconds, each bolt process metrics from [start, start + aggregationDuration] }, @@ -44,8 +40,6 @@ }, "eagleProps" : { - "mailHost" : "abc.com", - "mailDebug" : "true", "eagleService": { "host": "sandbox.hortonworks.com:2181", "port": 9099, @@ -56,21 +50,8 @@ "aggregate" : { "counters" : { - "metrics": [ - "cpu_milliseconds", - "file_bytes_read", - "file_bytes_written", - "hdfs_bytes_read", - "hdfs_bytes_written", - "hdfs_read_ops", - "hdfs_write_ops" - ], - - "groupByColumns": [ - "site, jobType", - "site, user", - "site" - ] + "metrics" : "cpu_milliseconds,file_bytes_read, file_bytes_written, hdfs_bytes_read, hdfs_bytes_written, hdfs_read_ops, hdfs_write_ops", + "groupBys": "site&jobType, site&user, site" } }, http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/7bd5d1d2/eagle-topology-assembly/pom.xml ---------------------------------------------------------------------- diff --git a/eagle-topology-assembly/pom.xml b/eagle-topology-assembly/pom.xml index 80ca265..7af6f96 100644 --- a/eagle-topology-assembly/pom.xml +++ b/eagle-topology-assembly/pom.xml @@ -62,6 +62,11 @@ eagle-jpm-web ${project.version} + + org.apache.eagle + eagle-jpm-aggregation + ${project.version} + http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/7bd5d1d2/eagle-topology-assembly/src/resources/META-INF/services/org.apache.eagle.app.spi.ApplicationProvider ---------------------------------------------------------------------- diff --git a/eagle-topology-assembly/src/resources/META-INF/services/org.apache.eagle.app.spi.ApplicationProvider b/eagle-topology-assembly/src/resources/META-INF/services/org.apache.eagle.app.spi.ApplicationProvider index 989886f..56292d2 100644 --- a/eagle-topology-assembly/src/resources/META-INF/services/org.apache.eagle.app.spi.ApplicationProvider +++ b/eagle-topology-assembly/src/resources/META-INF/services/org.apache.eagle.app.spi.ApplicationProvider @@ -17,4 +17,5 @@ org.apache.eagle.security.hbase.HBaseAuditLogAppProvider org.apache.eagle.app.example.ExampleApplicationProvider org.apache.eagle.app.jpm.JPMWebApplicationProvider org.apache.eagle.jpm.mr.history.MRHistoryJobApplicationProvider -org.apache.eagle.jpm.mr.running.MRRunningJobApplicationProvider \ No newline at end of file +org.apache.eagle.jpm.mr.running.MRRunningJobApplicationProvider +org.apache.eagle.jpm.aggregation.AggregationApplicationProvider \ No newline at end of file