Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 25F7E200B67 for ; Tue, 16 Aug 2016 10:32:01 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 24557160AA8; Tue, 16 Aug 2016 08:32:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id EC55B160A76 for ; Tue, 16 Aug 2016 10:31:58 +0200 (CEST) Received: (qmail 40000 invoked by uid 500); 16 Aug 2016 08:31:58 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 39991 invoked by uid 99); 16 Aug 2016 08:31:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Aug 2016 08:31:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 967051806EB for ; Tue, 16 Aug 2016 08:31:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.646 X-Spam-Level: X-Spam-Status: No, score=-4.646 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id V4BPuOW_siP4 for ; Tue, 16 Aug 2016 08:31:43 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with SMTP id BBEB25FC36 for ; Tue, 16 Aug 2016 08:31:41 +0000 (UTC) Received: (qmail 39700 invoked by uid 99); 16 Aug 2016 08:31:41 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Aug 2016 08:31:41 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2C1F8E04BE; Tue, 16 Aug 2016 08:31:41 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: ralphsu@apache.org To: commits@eagle.incubator.apache.org Date: Tue, 16 Aug 2016 08:31:42 -0000 Message-Id: <0dddde8193374263a323e31ffdbfbed3@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [2/3] incubator-eagle git commit: EAGLE-421: JMX Metric APP use new application framework archived-at: Tue, 16 Aug 2016 08:32:01 -0000 EAGLE-421: JMX Metric APP use new application framework Remove the topology code since JMX app could be done in new alert engine directly. Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/1517c132 Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/1517c132 Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/1517c132 Branch: refs/heads/develop Commit: 1517c132658408952e790e27565496b9587da06b Parents: b4732cb Author: Ralph, Su Authored: Mon Aug 15 11:28:56 2016 +0800 Committer: Ralph, Su Committed: Mon Aug 15 11:28:56 2016 +0800 ---------------------------------------------------------------------- eagle-hadoop-metric/pom.xml | 14 +- .../hadoop/metric/HadoopJmxApplication.java | 74 ---- .../eagle/hadoop/metric/JsonParserBolt.java | 62 ---- .../src/main/resources/application.conf | 51 --- .../resources/capacityused-policy-import.sh | 51 --- .../src/main/resources/datasources.json | 19 + .../src/main/resources/eagle-env.sh | 44 --- .../src/main/resources/hadoop-metric-init.sh | 180 ---------- .../src/main/resources/hadoopjmx.yaml | 18 - .../src/main/resources/hastate-policy-import.sh | 51 --- .../lastcheckpointtime-policy-import.sh | 51 --- .../src/main/resources/log4j.properties | 35 -- .../resources/missingblock-policy-import.sh | 51 --- .../main/resources/namenodelag-policy-import.sh | 49 --- .../main/resources/nodecount-policy-import.sh | 51 --- .../src/main/resources/policies.json | 169 +++++++++ .../src/main/resources/publishments.json | 33 ++ .../resources/safemodecheck-policy-import.sh | 51 --- .../sanbox/capacityused-policy-import.sh | 51 --- .../resources/sanbox/hastate-policy-import.sh | 51 --- .../sanbox/lastcheckpointtime-policy-import.sh | 51 --- .../sanbox/missingblock-policy-import.sh | 51 --- .../sanbox/namenodehanoactive-policy-import.sh | 52 --- ...nodehawithmorethanoneactive-policy-import.sh | 53 --- .../sanbox/namenodelag-policy-import.sh | 49 --- .../resources/sanbox/nodecount-policy-import.sh | 51 --- .../resourcemanagerhanoactive-policy-import.sh | 53 --- ...agerhawithmorethanoneactive-policy-import.sh | 54 --- .../sanbox/safemodecheck-policy-import.sh | 51 --- .../src/main/resources/streamdefinitions.json | 47 +++ .../metric/HadoopJmxMetricDeserializerTest.java | 36 -- .../hadoop/metric/TestHadoopMetricSiddhiQL.java | 354 ------------------- .../src/test/resources/cassandra.json | 13 - 33 files changed, 271 insertions(+), 1800 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/pom.xml ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/pom.xml b/eagle-hadoop-metric/pom.xml index 15eea00..b397e37 100644 --- a/eagle-hadoop-metric/pom.xml +++ b/eagle-hadoop-metric/pom.xml @@ -15,7 +15,8 @@ See the License for the specific language governing permissions and limitations under the License. --> - + eagle-parent org.apache.eagle @@ -25,15 +26,6 @@ eagle-hadoop-metric - - org.apache.eagle - eagle-stream-process-api - ${project.version} - - - org.apache.eagle - eagle-app-base - ${project.version} - + http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/HadoopJmxApplication.java ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/HadoopJmxApplication.java b/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/HadoopJmxApplication.java deleted file mode 100644 index 40d1a24..0000000 --- a/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/HadoopJmxApplication.java +++ /dev/null @@ -1,74 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.eagle.hadoop.metric; - -import backtype.storm.generated.StormTopology; -import backtype.storm.topology.BoltDeclarer; -import backtype.storm.topology.IRichSpout; -import backtype.storm.topology.TopologyBuilder; -import backtype.storm.tuple.Fields; -import com.typesafe.config.Config; -import com.typesafe.config.ConfigFactory; -import org.apache.eagle.app.StormApplication; -import org.apache.eagle.app.environment.impl.StormEnvironment; -import org.apache.eagle.app.sink.StormStreamSink; -import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSpoutProvider; -import storm.kafka.StringScheme; - -/** - * Since 8/12/16. - * This application just pass through data from jmx metric - * For persistence or alert purpose, it is not necessary to start application - * But keep this application in case of future business process - * - * Note: this application should be run as multiple instances based on different topic for data source - */ -public class HadoopJmxApplication extends StormApplication { - public final static String SPOUT_TASK_NUM = "topology.numOfSpoutTasks"; - public final static String PARSER_TASK_NUM = "topology.numOfParserTasks"; - public final static String SINK_TASK_NUM = "topology.numOfSinkTasks"; - - @Override - public StormTopology execute(Config config, StormEnvironment environment) { - int numOfSpoutTasks = config.getInt(SPOUT_TASK_NUM); - int numOfParserTasks = config.getInt(PARSER_TASK_NUM); - int numOfSinkTasks = config.getInt(SINK_TASK_NUM); - - TopologyBuilder builder = new TopologyBuilder(); - - KafkaSpoutProvider provider = new KafkaSpoutProvider(); - IRichSpout spout = provider.getSpout(config); - builder.setSpout("ingest", spout, numOfSpoutTasks); - - JsonParserBolt bolt = new JsonParserBolt(); - BoltDeclarer boltDeclarer = builder.setBolt("parserBolt", bolt, numOfParserTasks); - boltDeclarer.fieldsGrouping("ingest", new Fields(StringScheme.STRING_SCHEME_KEY)); - - StormStreamSink sinkBolt = environment.getStreamSink("hadoop_jmx_stream",config); - BoltDeclarer kafkaBoltDeclarer = builder.setBolt("kafkaSink", sinkBolt, numOfSinkTasks); - kafkaBoltDeclarer.fieldsGrouping("parserBolt", new Fields("f1")); - - return builder.createTopology(); - } - - public static void main(String[] args){ - Config config = ConfigFactory.load(); - HadoopJmxApplication app = new HadoopJmxApplication(); - app.run(config); - } -} http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/JsonParserBolt.java ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/JsonParserBolt.java b/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/JsonParserBolt.java deleted file mode 100644 index 7ca5ba6..0000000 --- a/eagle-hadoop-metric/src/main/java/org/apache/eagle/hadoop/metric/JsonParserBolt.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.eagle.hadoop.metric; - -import backtype.storm.task.OutputCollector; -import backtype.storm.task.TopologyContext; -import backtype.storm.topology.OutputFieldsDeclarer; -import backtype.storm.topology.base.BaseRichBolt; -import backtype.storm.tuple.Fields; -import backtype.storm.tuple.Tuple; -import org.codehaus.jackson.map.ObjectMapper; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.Arrays; -import java.util.Map; - -/** - * Since 8/14/16. - */ -public class JsonParserBolt extends BaseRichBolt { - private Logger LOG = LoggerFactory.getLogger(JsonParserBolt.class); - private OutputCollector collector; - private ObjectMapper mapper = new ObjectMapper(); - - @Override - public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { - this.collector = collector; - } - - @Override - public void execute(Tuple input) { - String msg = input.getString(0); - try { - Map ret = mapper.readValue(msg, Map.class); - collector.emit(Arrays.asList(ret)); - }catch(Exception ex){ - LOG.error("error in passing json message", ex); - }finally{ - collector.ack(input); - } - } - - @Override - public void declareOutputFields(OutputFieldsDeclarer declarer) { - declarer.declare(new Fields("msg")); - } -} http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/application.conf ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/application.conf b/eagle-hadoop-metric/src/main/resources/application.conf deleted file mode 100644 index e75355f..0000000 --- a/eagle-hadoop-metric/src/main/resources/application.conf +++ /dev/null @@ -1,51 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -{ - "appId" : "HadoopJmxApplication", - "mode" : "LOCAL", - "siteId" : "testsite", - "topology" : { - "numOfSpoutTasks" : 2, - "numOfParserTasks" : 2, - "numOfSinkTasks" : 2 - }, - "dataSourceConfig": { - "topic" : "jmx_metric", - "zkConnection" : "server.eagle.apache.org:2181", - "zkConnectionTimeoutMS" : 15000, - "consumerGroupId" : "EagleConsumer", - "fetchSize" : 1048586, - "transactionZKServers" : "server.eagle.apache.org", - "transactionZKPort" : 2181, - "transactionZKRoot" : "/consumers", - "transactionStateUpdateMS" : 2000 - "schemeCls" : "storm.kafka.StringScheme" - }, - "eagleProps" : { - "eagleService": { - "host": "localhost", - "port": 9090, - "username": "admin", - "password": "secret" - } - }, - "dataSinkConfig": { - "topic" : "jmx_metric_parsed", - "brokerList" : "server.eagle.apache.org:6667", - "serializerClass" : "kafka.serializer.StringEncoder", - "keySerializerClass" : "kafka.serializer.StringEncoder" - } -} http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/capacityused-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/capacityused-policy-import.sh b/eagle-hadoop-metric/src/main/resources/capacityused-policy-import.sh deleted file mode 100644 index 71e2756..0000000 --- a/eagle-hadoop-metric/src/main/resources/capacityused-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: capacityUsedPolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "capacityUsedPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.fsnamesystemstate.capacityused\\\" and convert(value, \\\"long\\\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/datasources.json ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/datasources.json b/eagle-hadoop-metric/src/main/resources/datasources.json new file mode 100644 index 0000000..5c10696 --- /dev/null +++ b/eagle-hadoop-metric/src/main/resources/datasources.json @@ -0,0 +1,19 @@ +[ + { + "name": "hadoop_jmx_datasource", + "type": "KAFKA", + "properties": { + }, + "topic": "hadoop_jmx_metrics", + "schemeCls": "org.apache.eagle.alert.engine.scheme.JsonScheme", + "codec": { + "streamNameSelectorProp": { + "userProvidedStreamName" : "hadoopJmxMetricEventStream", + "streamNameFormat":"%s" + }, + "streamNameSelectorCls":"org.apache.eagle.alert.engine.scheme.JsonStringStreamNameSelector", + "timestampColumn": "timestamp", + "timestampFormat": "" + } + } +] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/eagle-env.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/eagle-env.sh b/eagle-hadoop-metric/src/main/resources/eagle-env.sh deleted file mode 100755 index 79ff5fa..0000000 --- a/eagle-hadoop-metric/src/main/resources/eagle-env.sh +++ /dev/null @@ -1,44 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# set EAGLE_HOME -export EAGLE_HOME=$(dirname $0)/.. - -# The java implementation to use. please use jdk 1.7 or later -# export JAVA_HOME=${JAVA_HOME} -# export JAVA_HOME=/usr/java/jdk1.7.0_80/ - -# nimbus.host, default is localhost -export EAGLE_NIMBUS_HOST=localhost - -# EAGLE_SERVICE_HOST, default is `hostname -f` -export EAGLE_SERVICE_HOST=localhost - -# EAGLE_SERVICE_PORT, default is 9099 -export EAGLE_SERVICE_PORT=9099 - -# EAGLE_SERVICE_USER -export EAGLE_SERVICE_USER=admin - -# EAGLE_SERVICE_PASSWORD -export EAGLE_SERVICE_PASSWD=secret - -export EAGLE_CLASSPATH=$EAGLE_HOME/conf -# Add eagle shared library jars -for file in $EAGLE_HOME/lib/share/*;do - EAGLE_CLASSPATH=$EAGLE_CLASSPATH:$file -done http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/hadoop-metric-init.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/hadoop-metric-init.sh b/eagle-hadoop-metric/src/main/resources/hadoop-metric-init.sh deleted file mode 100755 index 849f462..0000000 --- a/eagle-hadoop-metric/src/main/resources/hadoop-metric-init.sh +++ /dev/null @@ -1,180 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh - -##################################################################### -# Import stream metadata for HDFS -##################################################################### - -## AlertDataSource: data sources bound to sites -echo "Importing AlertDataSourceService for persist... " - -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=SiteApplicationService" \ - -d ' - [ - { - "tags":{ - "site":"sandbox", - "application":"hadoopJmxMetricDataSource" - }, - "enabled": true, - "config": "web.druid.coordinator=coordinatorHost:port\nweb.druid.broker=brokerHost:port" - } - ] - ' - -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=ApplicationDescService" \ - -d ' - [ - { - "tags":{ - "application":"hadoopJmxMetricDataSource" - }, - "description":"hadoop jmx metric monitoring", - "alias":"JmxMetricMonitor", - "groupName":"METRIC", - "config":"{}", - "features":["common","metadata"] - } - ] - ' - -## AlertStreamService: alert streams generated from data source -echo "" -echo "Importing AlertStreamService for HDFS... " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertStreamService" \ - -d ' - [ - { - "prefix":"alertStream", - "tags":{ - "application":"hadoopJmxMetricDataSource", - "streamName":"hadoopJmxMetricEventStream" - }, - "description":"hadoop" - } - ] - ' - -## AlertExecutorService: what alert streams are consumed by alert executor -echo "" -echo "Importing AlertExecutorService for HDFS... " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertExecutorService" \ - -d ' - [ - { - "prefix":"alertExecutor", - "tags":{ - "application":"hadoopJmxMetricDataSource", - "alertExecutorId":"hadoopJmxMetricAlertExecutor", - "streamName":"hadoopJmxMetricEventStream" - }, - "description":"aggregate executor for hadoop jmx metric event stream" - } - ] - ' - -## AlertStreamSchemaService: schema for event from alert stream -echo "" -echo "Importing AlertStreamSchemaService for HDFS... " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ -"http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertStreamSchemaService" \ - -d ' - [ - { - "prefix": "alertStreamSchema", - "tags": { - "application": "hadoopJmxMetricDataSource", - "streamName": "hadoopJmxMetricEventStream", - "attrName": "host" - }, - "attrDescription": "the host that current metric comes form", - "attrType": "string", - "category": "", - "attrValueResolver": "" - }, - { - "prefix": "alertStreamSchema", - "tags": { - "application": "hadoopJmxMetricDataSource", - "streamName": "hadoopJmxMetricEventStream", - "attrName": "timestamp" - }, - "attrDescription": "the metric timestamp", - "attrType": "long", - "category": "", - "attrValueResolver": "" - }, - { - "prefix": "alertStreamSchema", - "tags": { - "application": "hadoopJmxMetricDataSource", - "streamName": "hadoopJmxMetricEventStream", - "attrName": "metric" - }, - "attrDescription": "the metric name", - "attrType": "string", - "category": "", - "attrValueResolver": "" - }, - { - "prefix": "alertStreamSchema", - "tags": { - "application": "hadoopJmxMetricDataSource", - "streamName": "hadoopJmxMetricEventStream", - "attrName": "component" - }, - "attrDescription": "the component that the metric comes from", - "attrType": "string", - "category": "", - "attrValueResolver": "" - }, - { - "prefix": "alertStreamSchema", - "tags": { - "application": "hadoopJmxMetricDataSource", - "streamName": "hadoopJmxMetricEventStream", - "attrName": "site" - }, - "attrDescription": "the site that the metric belongs to", - "attrType": "string", - "category": "", - "attrValueResolver": "" - }, - { - "prefix": "alertStreamSchema", - "tags": { - "application": "hadoopJmxMetricDataSource", - "streamName": "hadoopJmxMetricEventStream", - "attrName": "value" - }, - "attrDescription": "the metric value in string presentation", - "attrType": "double", - "category": "", - "attrValueResolver": "" - } - ] - ' - -## Finished -echo "" -echo "Finished initialization for eagle topology" http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/hadoopjmx.yaml ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/hadoopjmx.yaml b/eagle-hadoop-metric/src/main/resources/hadoopjmx.yaml deleted file mode 100644 index a68a323..0000000 --- a/eagle-hadoop-metric/src/main/resources/hadoopjmx.yaml +++ /dev/null @@ -1,18 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -topology.workers: 1 -topology.acker.executors: 1 -topology.tasks: 1 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/hastate-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/hastate-policy-import.sh b/eagle-hadoop-metric/src/main/resources/hastate-policy-import.sh deleted file mode 100644 index b125e37..0000000 --- a/eagle-hadoop-metric/src/main/resources/hastate-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: haStatePolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "haStatePolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from every a = hadoopJmxMetricEventStream[metric==\\\"hadoop.namenode.fsnamesystem.hastate\\\"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host == a.host and (convert(a.value, \\\"long\\\") != convert(value, \\\"long\\\"))] within 10 min select a.host, a.value as oldHaState, b.value as newHaState, b.timestamp as timestamp, b.metric as metric, b.component as component, b.site as site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/lastcheckpointtime-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/lastcheckpointtime-policy-import.sh b/eagle-hadoop-metric/src/main/resources/lastcheckpointtime-policy-import.sh deleted file mode 100644 index 333367c..0000000 --- a/eagle-hadoop-metric/src/main/resources/lastcheckpointtime-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: lastCheckPointTimePolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "lastCheckPointTimePolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.dfs.lastcheckpointtime\\\" and (convert(value, \\\"long\\\") + 18000000) < timestamp]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"last check point time lag found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/log4j.properties ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/log4j.properties b/eagle-hadoop-metric/src/main/resources/log4j.properties deleted file mode 100644 index 149caa7..0000000 --- a/eagle-hadoop-metric/src/main/resources/log4j.properties +++ /dev/null @@ -1,35 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -log4j.rootLogger=INFO, stdout, DRFA - -eagle.log.dir=./logs -eagle.log.file=eagle.log - -# standard output -log4j.appender.stdout=org.apache.log4j.ConsoleAppender -log4j.appender.stdout.layout=org.apache.log4j.PatternLayout -log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %p [%t] %c{2}[%L]: %m%n - -# Daily Rolling File Appender -log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender -log4j.appender.DRFA.File=${eagle.log.dir}/${eagle.log.file} -log4j.appender.DRFA.DatePattern=yyyy-MM-dd -## 30-day backup -# log4j.appender.DRFA.MaxBackupIndex=30 -log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout - -# Pattern format: Date LogLevel LoggerName LogMessage -log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c{2}[%L]: %m%n \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/missingblock-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/missingblock-policy-import.sh b/eagle-hadoop-metric/src/main/resources/missingblock-policy-import.sh deleted file mode 100644 index 9a5e924..0000000 --- a/eagle-hadoop-metric/src/main/resources/missingblock-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: missingBlockPolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "missingBlockPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.dfs.missingblocks\\\" and convert(value, \\\"long\\\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/namenodelag-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/namenodelag-policy-import.sh b/eagle-hadoop-metric/src/main/resources/namenodelag-policy-import.sh deleted file mode 100644 index 45de4b8..0000000 --- a/eagle-hadoop-metric/src/main/resources/namenodelag-policy-import.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - -echo "" -echo "Importing Policy: NameNodeLagPolicy" -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "NameNodeLagPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from every a = hadoopJmxMetricEventStream[metric==\\\"hadoop.namenode.journaltransaction.lastappliedorwrittentxid\\\"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host != a.host and (max(convert(a.value, \\\"long\\\")) + 100) <= max(convert(value, \\\"long\\\"))] within 5 min select a.host as hostA, a.value as transactIdA, b.host as hostB, b.value as transactIdB insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"name node lag found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/nodecount-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/nodecount-policy-import.sh b/eagle-hadoop-metric/src/main/resources/nodecount-policy-import.sh deleted file mode 100644 index 589a44e..0000000 --- a/eagle-hadoop-metric/src/main/resources/nodecount-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: dataNodeCountPolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "dataNodeCountPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from every (e1 = hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.fsnamesystemstate.numlivedatanodes\\\" ]) -> e2 = hadoopJmxMetricEventStream[metric == e1.metric and host == e1.host and (convert(e1.value, \\\"long\\\") - 5) >= convert(value, \\\"long\\\") ] within 5 min select e1.metric, e1.host, e1.value as highNum, e1.timestamp as start, e2.value as lowNum, e2.timestamp as end insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"node count joggling found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/policies.json ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/policies.json b/eagle-hadoop-metric/src/main/resources/policies.json new file mode 100644 index 0000000..811ffb0 --- /dev/null +++ b/eagle-hadoop-metric/src/main/resources/policies.json @@ -0,0 +1,169 @@ +[ + { + "name": "haStatePolicy", + "description": "ha state check for hadoop name node", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from every a = hadoopJmxMetricEventStream[metric==\"hadoop.namenode.fsnamesystem.hastate\"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host == a.host and (convert(a.value, \"long\\\") != convert(value, \"long\"))] within 10 min select a.host, a.value as oldHaState, b.value as newHaState, b.timestamp as timestamp, b.metric as metric, b.component as component, b.site as site insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + }, + + { + "name": "capacityUsedPolicy", + "description": "capacity usage check for hadoop cluster", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from hadoopJmxMetricEventStream[metric == \"hadoop.namenode.fsnamesystemstate.capacityused\" and convert(value, \"long\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + }, + + { + "name": "lastCheckPointTimePolicy", + "description": "last check point interval check for hadoop name node pair", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from hadoopJmxMetricEventStream[metric == \"hadoop.namenode.dfs.lastcheckpointtime\" and (convert(value, \"long\") + 18000000) < timestamp]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + }, + + { + "name": "missingBlockPolicy", + "description": "missing block number check for hadoop cluster", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from hadoopJmxMetricEventStream[metric == \"hadoop.namenode.dfs.missingblocks\" and convert(value, \"long\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + }, + + { + "name": "namenodeTxLagPolicy", + "description": "name node tx log lag check", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from every a = hadoopJmxMetricEventStream[metric==\"hadoop.namenode.journaltransaction.lastappliedorwrittentxid\"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host != a.host and (max(convert(a.value, \"long\")) + 100) <= max(convert(value, \"long\"))] within 5 min select a.host as hostA, a.value as transactIdA, b.host as hostB, b.value as transactIdB insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + }, + + { + "name": "nodecountPolicy", + "description": "data node number check for hadoop cluster", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from every (e1 = hadoopJmxMetricEventStream[metric == \"hadoop.namenode.fsnamesystemstate.numlivedatanodes\" ]) -> e2 = hadoopJmxMetricEventStream[metric == e1.metric and host == e1.host and (convert(e1.value, \"long\") - 5) >= convert(value, \"long\") ] within 5 min select e1.metric, e1.host, e1.value as highNum, e1.timestamp as start, e2.value as lowNum, e2.timestamp as end insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + }, + + { + "name": "nameNodeSafeModeCheckPolicy", + "description": "safe mode check for name node", + "definition": { + "inputStreams": [ + "hadoopJmxMetricEventStream" + ], + "outputStreams": [ + "tmp" + ], + "type": "siddhi", + "value": " from hadoopJmxMetricEventStream[component==\"namenode\" and metric == \"hadoop.namenode.fsnamesystemstate.fsstate\" and convert(value, \"long\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; " + }, + "partitionSpec": [ + { + "streamId": "hadoopJmxMetricEventStream", + "type": "GROUPBY", + "columns": [ + "host" + ] + } + ] + } +] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/publishments.json ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/publishments.json b/eagle-hadoop-metric/src/main/resources/publishments.json new file mode 100644 index 0000000..10d16a4 --- /dev/null +++ b/eagle-hadoop-metric/src/main/resources/publishments.json @@ -0,0 +1,33 @@ +[ + { + "name":"jmxStreamOutput_email", + "type":"org.apache.eagle.alert.engine.publisher.impl.AlertEmailPublisher", + "policyIds": [ + "perfmon_cpu_host_check", "perfmon_cpu_pool_check" + ], + "properties": { + "subject":"Hadoop Jmx Metrics Alert", + "template":"", + "sender": "sender@corp.com", + "recipients": "services@corp.com", + "smtp.server":"smtp.mailhost.com", + "connection": "plaintext", + "smtp.port": "25" + }, + "dedupIntervalMin" : "PT1M", + "serializer" : "org.apache.eagle.alert.engine.publisher.impl.StringEventSerializer" + }, + { + "type": "org.apache.eagle.alert.engine.publisher.impl.AlertKafkaPublisher", + "name":"jmxStreamOutput_kafka", + "policyIds": [ + "perfmon_cpu_host_check" + ], + "dedupIntervalMin": "PT1M", + "properties":{ + "kafka_broker":"localhost:9092", + "topic":"jmx_alerts" + }, + "serializer" : "org.apache.eagle.alert.engine.publisher.impl.StringEventSerializer" + } +] http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/safemodecheck-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/safemodecheck-policy-import.sh b/eagle-hadoop-metric/src/main/resources/safemodecheck-policy-import.sh deleted file mode 100644 index 32a6bee..0000000 --- a/eagle-hadoop-metric/src/main/resources/safemodecheck-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: safeModePolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "safeModePolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[component==\\\"namenode\\\" and metric == \\\"hadoop.namenode.fsnamesystemstate.fsstate\\\" and convert(value, \\\"long\\\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/capacityused-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/capacityused-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/capacityused-policy-import.sh deleted file mode 100755 index ad52275..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/capacityused-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: capacityUsedPolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "capacityUsedPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.fsnamesystemstate.capacityused\\\" and convert(value, \\\"long\\\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/hastate-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/hastate-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/hastate-policy-import.sh deleted file mode 100755 index 7801c09..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/hastate-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: haStatePolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "haStatePolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from every a = hadoopJmxMetricEventStream[metric==\\\"hadoop.namenode.fsnamesystem.hastate\\\"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host == a.host and (convert(a.value, \\\"long\\\") != convert(value, \\\"long\\\"))] within 10 min select a.host, a.value as oldHaState, b.value as newHaState, b.timestamp as timestamp, b.metric as metric, b.component as component, b.site as site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/lastcheckpointtime-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/lastcheckpointtime-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/lastcheckpointtime-policy-import.sh deleted file mode 100755 index d3811aa..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/lastcheckpointtime-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: lastCheckPointTimePolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "lastCheckPointTimePolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.dfs.lastcheckpointtime\\\" and (convert(value, \\\"long\\\") + 18000000) < timestamp]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"last check point time lag found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/missingblock-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/missingblock-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/missingblock-policy-import.sh deleted file mode 100755 index be51597..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/missingblock-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: missingBlockPolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "missingBlockPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.dfs.missingblocks\\\" and convert(value, \\\"long\\\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/namenodehanoactive-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/namenodehanoactive-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/namenodehanoactive-policy-import.sh deleted file mode 100755 index 33a7210..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/namenodehanoactive-policy-import.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## - -echo "" -echo "Importing policy: NamenodeHAHasNoActive " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "NamenodeHAHasNoActive", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.hastate.active.count\\\" and value == 0 ]select * insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/namenodehawithmorethanoneactive-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/namenodehawithmorethanoneactive-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/namenodehawithmorethanoneactive-policy-import.sh deleted file mode 100755 index 2afa09e..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/namenodehawithmorethanoneactive-policy-import.sh +++ /dev/null @@ -1,53 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## - -echo "" -echo "Importing policy: NamenodeHAWithMoreThanOneActive " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "NamenodeHAWithMoreThanOneActive", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.hastate.active.count\\\" and value > 1]select * insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 - http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/namenodelag-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/namenodelag-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/namenodelag-policy-import.sh deleted file mode 100755 index 2ccd37e..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/namenodelag-policy-import.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - -echo "" -echo "Importing Policy: NameNodeLagPolicy" -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "NameNodeLagPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from every a = hadoopJmxMetricEventStream[metric==\\\"hadoop.namenode.journaltransaction.lastappliedorwrittentxid\\\"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host != a.host and (max(convert(a.value, \\\"long\\\")) + 100) <= max(convert(value, \\\"long\\\"))] within 5 min select a.host as hostA, a.value as transactIdA, b.host as hostB, b.value as transactIdB insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"name node lag found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/nodecount-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/nodecount-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/nodecount-policy-import.sh deleted file mode 100755 index 90e0114..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/nodecount-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: dataNodeCountPolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "dataNodeCountPolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from every (e1 = hadoopJmxMetricEventStream[metric == \\\"hadoop.namenode.fsnamesystemstate.numlivedatanodes\\\" ]) -> e2 = hadoopJmxMetricEventStream[metric == e1.metric and host == e1.host and (convert(e1.value, \\\"long\\\") - 5) >= convert(value, \\\"long\\\") ] within 5 min select e1.metric, e1.host, e1.value as highNum, e1.timestamp as start, e2.value as lowNum, e2.timestamp as end insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"node count joggling found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhanoactive-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhanoactive-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhanoactive-policy-import.sh deleted file mode 100755 index 95b9c35..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhanoactive-policy-import.sh +++ /dev/null @@ -1,53 +0,0 @@ -# -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## - -echo "" -echo "Importing policy: ResourceManagerHAHasNoActive " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "ResourceManagerHAHasNoActive", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.resourcemanager.hastate.active.count\\\" and value == 0 ]select * insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhawithmorethanoneactive-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhawithmorethanoneactive-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhawithmorethanoneactive-policy-import.sh deleted file mode 100755 index 269003c..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/resourcemanagerhawithmorethanoneactive-policy-import.sh +++ /dev/null @@ -1,54 +0,0 @@ -# -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## - -echo "" -echo "Importing policy: ResourceManagerHAHasMoreThanOneActive " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "ResourceManagerHAHasMoreThanOneActive", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine", - "description":" Resource Manager HA Has More than one Active" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[metric == \\\"hadoop.resourcemanager.hastate.active.count\\\" and value > 1 ]select * insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/sanbox/safemodecheck-policy-import.sh ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/sanbox/safemodecheck-policy-import.sh b/eagle-hadoop-metric/src/main/resources/sanbox/safemodecheck-policy-import.sh deleted file mode 100755 index 8f1d14f..0000000 --- a/eagle-hadoop-metric/src/main/resources/sanbox/safemodecheck-policy-import.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with` -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -source $(dirname $0)/eagle-env.sh -source $(dirname $0)/hadoop-metric-init.sh - - -##### add policies ########## -echo "" -echo "Importing policy: safeModePolicy " -curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \ - "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertDefinitionService" \ - -d ' - [ - { - "prefix": "alertdef", - "tags": { - "site": "sandbox", - "application": "hadoopJmxMetricDataSource", - "policyId": "safeModePolicy", - "alertExecutorId": "hadoopJmxMetricAlertExecutor", - "policyType": "siddhiCEPEngine" - }, - "description": "jmx metric ", - "policyDef": "{\"expression\":\"from hadoopJmxMetricEventStream[component==\\\"namenode\\\" and metric == \\\"hadoop.namenode.fsnamesystemstate.fsstate\\\" and convert(value, \\\"long\\\") > 0]#window.externalTime(timestamp ,10 min) select metric, host, value, timestamp, component, site insert into tmp; \",\"type\":\"siddhiCEPEngine\"}", - "enabled": true, - "dedupeDef": "{\"alertDedupIntervalMin\":10,\"emailDedupIntervalMin\":10}", - "notificationDef": "[{\"notificationType\":\"eagleStore\"},{\"sender\":\"eagle@apache.org\",\"recipients\":\"eagle@apache.org\",\"subject\":\"missing block found.\",\"flavor\":\"email\",\"id\":\"email_1\",\"tplFileName\":\"\"}]" - } - ] - ' - - ## Finished -echo "" -echo "Finished initialization for eagle topology" - -exit 0 http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/main/resources/streamdefinitions.json ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/main/resources/streamdefinitions.json b/eagle-hadoop-metric/src/main/resources/streamdefinitions.json new file mode 100644 index 0000000..66a2c98 --- /dev/null +++ b/eagle-hadoop-metric/src/main/resources/streamdefinitions.json @@ -0,0 +1,47 @@ +[ + { + "streamId": "hadoopJmxMetricEventStream", + "dataSource": "hadoop_jmx_datasource", + "description": "the data stream for hadoop jmx metrics", + "validate": false, + "timeseries": false, + "columns": [ + { + "name": "host", + "type": "STRING", + "defaultValue": "", + "required": true + }, + { + "name": "timestamp", + "type": "LONG", + "defaultValue": 0, + "required": true + }, + { + "name": "metric", + "type": "STRING", + "defaultValue": "default_hadoop_jmx_metric_name", + "required": true + }, + { + "name": "component", + "type": "STRING", + "defaultValue": "namenode", + "required": true + }, + { + "name": "site", + "type": "STRING", + "defaultValue": "hadoop", + "required": true + }, + { + "name": "value", + "type": "DOUBLE", + "defaultValue": 0.0, + "required": true + } + ] + } +] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/1517c132/eagle-hadoop-metric/src/test/java/org/apache/eagle/hadoop/metric/HadoopJmxMetricDeserializerTest.java ---------------------------------------------------------------------- diff --git a/eagle-hadoop-metric/src/test/java/org/apache/eagle/hadoop/metric/HadoopJmxMetricDeserializerTest.java b/eagle-hadoop-metric/src/test/java/org/apache/eagle/hadoop/metric/HadoopJmxMetricDeserializerTest.java deleted file mode 100644 index 4c7fe6d..0000000 --- a/eagle-hadoop-metric/src/test/java/org/apache/eagle/hadoop/metric/HadoopJmxMetricDeserializerTest.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package org.apache.eagle.hadoop.metric; - -import org.junit.Test; - -/** - * Created on 1/19/16. - */ -public class HadoopJmxMetricDeserializerTest { - @Test - public void test() { -// HadoopJmxMetricDeserializer des = new HadoopJmxMetricDeserializer(null); -// -// String m = "{\"host\": \"hostname-1\", \"timestamp\": 1453208956395, \"metric\": \"hadoop.namenode.dfs.lastwrittentransactionid\", \"component\": \"namenode\", \"site\": \"sandbox\", \"value\": \"49716\"}"; -// Object obj = des.deserialize(m.getBytes()); -// Assert.assertTrue(obj instanceof Map); -// Map metric = (Map) obj; -// Assert.assertEquals("hostname-1" ,metric.get("host")); -// Assert.assertEquals(1453208956395l ,metric.get("timestamp")); - } -}