Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1BCF7200B3E for ; Wed, 7 Sep 2016 19:42:19 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1A325160AD3; Wed, 7 Sep 2016 17:42:19 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 43821160AC1 for ; Wed, 7 Sep 2016 19:42:17 +0200 (CEST) Received: (qmail 14321 invoked by uid 500); 7 Sep 2016 17:42:16 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 14312 invoked by uid 99); 7 Sep 2016 17:42:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Sep 2016 17:42:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id F22F3C0D53 for ; Wed, 7 Sep 2016 17:42:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.646 X-Spam-Level: X-Spam-Status: No, score=-4.646 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.426] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id AKw-pjZbKhd1 for ; Wed, 7 Sep 2016 17:42:01 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 697D060D7C for ; Wed, 7 Sep 2016 17:41:58 +0000 (UTC) Received: (qmail 11879 invoked by uid 99); 7 Sep 2016 17:41:57 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Sep 2016 17:41:57 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 60EBFE08B8; Wed, 7 Sep 2016 17:41:57 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: yonzhang2012@apache.org To: commits@eagle.incubator.apache.org Date: Wed, 07 Sep 2016 17:42:00 -0000 Message-Id: In-Reply-To: <7201ad3b09274d489e31c344a4940ff4@git.apache.org> References: <7201ad3b09274d489e31c344a4940ff4@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [04/52] [abbrv] incubator-eagle git commit: [EAGLE-484][EAGLE-497] Add override equals method and fix code style of Oozie archived-at: Wed, 07 Sep 2016 17:42:19 -0000 [EAGLE-484][EAGLE-497] Add override equals method and fix code style of Oozie EAGLE-497 Fix code style of Oozie - Fix code style of Oozie https://issues.apache.org/jira/browse/EAGLE-497 Author: r7raul1984 Closes #382 from r7raul1984/EAGLE-497. Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/6a55b59f Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/6a55b59f Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/6a55b59f Branch: refs/heads/master Commit: 6a55b59f0a53f6e832318691ac1dee5a6d0b90e7 Parents: 0bde482 Author: r7raul1984 Authored: Wed Aug 24 20:20:31 2016 +0800 Committer: Hao Chen Committed: Wed Aug 24 20:20:31 2016 +0800 ---------------------------------------------------------------------- .../coordination/model/PolicyWorkerQueue.java | 22 ++++++++++++++++ .../coordination/model/StreamRouterSpec.java | 16 ++++++++++++ .../alert/coordination/model/WorkSlot.java | 23 +++++++++++++++++ .../security/entity/OozieResourceEntity.java | 12 ++++----- .../OozieResourceSensitivityAPIEntity.java | 2 +- .../oozie/parse/OozieAuditLogApplication.java | 25 +++++++++--------- .../parse/OozieAuditLogKafkaDeserializer.java | 9 +++---- .../oozie/parse/OozieAuditLogParser.java | 25 +++++++++--------- .../oozie/parse/OozieAuditLogParserBolt.java | 6 ++--- .../OozieResourceSensitivityDataJoinBolt.java | 17 +++++------- .../OozieSensitivityDataEnrichLCM.java | 7 +++-- .../OozieResourceSensitivityDataJoiner.java | 2 +- .../BadOozieMetadataAccessConfigException.java | 6 ++--- .../oozie/dao/OozieMetadataAccessConfig.java | 26 +++++++++---------- .../dao/OozieMetadataAccessConfigDAOImpl.java | 7 ++--- .../oozie/dao/OozieMetadataDAOImpl.java | 3 ++- .../oozie/dao/OozieSensitivityMetadataDAO.java | 3 ++- .../dao/OozieSensitivityMetadataDAOImpl.java | 27 ++++++++++---------- .../TestOozieResourceSensitivityDataJoiner.java | 8 +----- 19 files changed, 153 insertions(+), 93 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/PolicyWorkerQueue.java ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/PolicyWorkerQueue.java b/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/PolicyWorkerQueue.java index 43780f3..e32d13c 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/PolicyWorkerQueue.java +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/PolicyWorkerQueue.java @@ -18,8 +18,11 @@ package org.apache.eagle.alert.coordination.model; import java.util.ArrayList; import java.util.List; +import java.util.Objects; +import org.apache.commons.collections.CollectionUtils; import org.apache.commons.lang3.StringUtils; +import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.eagle.alert.engine.coordinator.StreamPartition; /** @@ -56,6 +59,7 @@ public class PolicyWorkerQueue { this.partition = partition; } + public StreamPartition getPartition() { return partition; } @@ -72,6 +76,24 @@ public class PolicyWorkerQueue { this.workers = workers; } + @Override + public boolean equals(Object other) { + if (this == other) { + return true; + } + if (!(other instanceof PolicyWorkerQueue)) { + return false; + } + PolicyWorkerQueue that = (PolicyWorkerQueue) other; + return Objects.equals(partition, that.partition) && + CollectionUtils.isEqualCollection(workers, that.workers); + } + + @Override + public int hashCode() { + return new HashCodeBuilder().append(partition).append(workers).build(); + } + public String toString() { return "[" + StringUtils.join(workers, ",") + "]"; } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/StreamRouterSpec.java ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/StreamRouterSpec.java b/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/StreamRouterSpec.java index 773ae56..195c8d9 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/StreamRouterSpec.java +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/StreamRouterSpec.java @@ -18,7 +18,9 @@ package org.apache.eagle.alert.coordination.model; import java.util.ArrayList; import java.util.List; +import java.util.Objects; +import org.apache.commons.collections.CollectionUtils; import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.eagle.alert.engine.coordinator.StreamPartition; @@ -72,4 +74,18 @@ public class StreamRouterSpec { return String.format("StreamRouterSpec[streamId=%s,partition=%s, queue=[%s]]", this.getStreamId(), this.getPartition(), this.getTargetQueue()); } + + @Override + public boolean equals(Object other) { + if (this == other) { + return true; + } + if (!(other instanceof StreamRouterSpec)) { + return false; + } + StreamRouterSpec that = (StreamRouterSpec) other; + return Objects.equals(streamId, that.streamId) && + Objects.equals(partition, that.partition) && + CollectionUtils.isEqualCollection(targetQueue, that.targetQueue); + } } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/WorkSlot.java ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/WorkSlot.java b/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/WorkSlot.java index 96016f4..3f6f36d 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/WorkSlot.java +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-common/src/main/java/org/apache/eagle/alert/coordination/model/WorkSlot.java @@ -17,6 +17,10 @@ package org.apache.eagle.alert.coordination.model; +import org.apache.commons.lang3.builder.HashCodeBuilder; + +import java.util.Objects; + /** * A slot is simply a bolt. */ @@ -44,11 +48,30 @@ public class WorkSlot { } + public WorkSlot(String topo, String boltId) { this.topologyName = topo; this.boltId = boltId; } + @Override + public boolean equals(Object other) { + if (this == other) { + return true; + } + if (!(other instanceof WorkSlot)) { + return false; + } + WorkSlot workSlot = (WorkSlot) other; + return Objects.equals(topologyName, workSlot.topologyName) && + Objects.equals(boltId, workSlot.boltId); + } + + @Override + public int hashCode() { + return new HashCodeBuilder().append(topologyName).append(boltId).build(); + } + public String toString() { return "(" + topologyName + ":" + boltId + ")"; } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceEntity.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceEntity.java b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceEntity.java index 0506dd2..4dc2249 100644 --- a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceEntity.java +++ b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceEntity.java @@ -32,7 +32,7 @@ public class OozieResourceEntity implements Serializable { } public OozieResourceEntity(String jobId, String name) { - this(jobId,name,null); + this(jobId, name, null); } public String getJobId() { @@ -72,11 +72,11 @@ public class OozieResourceEntity implements Serializable { @Override public String toString() { - return "OozieResourceEntity{" + - "jobId='" + jobId + '\'' + - ", name='" + name + '\'' + - ", sensitiveType='" + sensitiveType + '\'' + - '}'; + return "OozieResourceEntity{" + + "jobId='" + jobId + '\'' + + ", name='" + name + '\'' + + ", sensitiveType='" + sensitiveType + '\'' + + '}'; } } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceSensitivityAPIEntity.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceSensitivityAPIEntity.java b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceSensitivityAPIEntity.java index 82953f5..767a4fb 100644 --- a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceSensitivityAPIEntity.java +++ b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/entity/OozieResourceSensitivityAPIEntity.java @@ -30,7 +30,7 @@ import org.codehaus.jackson.map.annotate.JsonSerialize; public class OozieResourceSensitivityAPIEntity extends TaggedLogAPIEntity { private static final long serialVersionUID = 2L; /** - * sensitivityType can be multi-value attribute, and values can be separated by "|" + * sensitivityType can be multi-value attribute, and values can be separated by "|". */ @Column("a") private String sensitivityType; http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogApplication.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogApplication.java b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogApplication.java index 4204a17..23fef45 100644 --- a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogApplication.java +++ b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogApplication.java @@ -17,6 +17,12 @@ */ package org.apache.eagle.security.oozie.parse; +import org.apache.eagle.app.StormApplication; +import org.apache.eagle.app.environment.impl.StormEnvironment; +import org.apache.eagle.app.sink.StormStreamSink; +import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSpoutProvider; +import org.apache.eagle.security.oozie.parse.sensitivity.OozieResourceSensitivityDataJoinBolt; + import backtype.storm.generated.StormTopology; import backtype.storm.topology.BoltDeclarer; import backtype.storm.topology.IRichSpout; @@ -24,20 +30,15 @@ import backtype.storm.topology.TopologyBuilder; import backtype.storm.tuple.Fields; import com.typesafe.config.Config; import com.typesafe.config.ConfigFactory; -import org.apache.eagle.app.StormApplication; -import org.apache.eagle.app.environment.impl.StormEnvironment; -import org.apache.eagle.app.sink.StormStreamSink; -import org.apache.eagle.security.oozie.parse.sensitivity.OozieResourceSensitivityDataJoinBolt; -import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSpoutProvider; /** * Since 8/12/16. */ public class OozieAuditLogApplication extends StormApplication { - public final static String SPOUT_TASK_NUM = "topology.numOfSpoutTasks"; - public final static String PARSER_TASK_NUM = "topology.numOfParserTasks"; - public final static String JOIN_TASK_NUM = "topology.numOfJoinTasks"; - public final static String SINK_TASK_NUM = "topology.numOfSinkTasks"; + public static final String SPOUT_TASK_NUM = "topology.numOfSpoutTasks"; + public static final String PARSER_TASK_NUM = "topology.numOfParserTasks"; + public static final String JOIN_TASK_NUM = "topology.numOfJoinTasks"; + public static final String SINK_TASK_NUM = "topology.numOfSinkTasks"; @Override public StormTopology execute(Config config, StormEnvironment environment) { @@ -48,7 +49,6 @@ public class OozieAuditLogApplication extends StormApplication { int numOfSpoutTasks = config.getInt(SPOUT_TASK_NUM); int numOfParserTask = config.getInt(PARSER_TASK_NUM); int numOfJoinTasks = config.getInt(JOIN_TASK_NUM); - int numOfSinkTasks = config.getInt(SINK_TASK_NUM); builder.setSpout("ingest", spout, numOfSpoutTasks); @@ -61,13 +61,14 @@ public class OozieAuditLogApplication extends StormApplication { BoltDeclarer boltDeclarer = builder.setBolt("joinBolt", joinBolt, numOfJoinTasks); boltDeclarer.fieldsGrouping("parserBolt", new Fields("f1")); - StormStreamSink sinkBolt = environment.getStreamSink("oozie_audit_log_stream",config); + StormStreamSink sinkBolt = environment.getStreamSink("oozie_audit_log_stream", config); + int numOfSinkTasks = config.getInt(SINK_TASK_NUM); BoltDeclarer kafkaBoltDeclarer = builder.setBolt("kafkaSink", sinkBolt, numOfSinkTasks); kafkaBoltDeclarer.fieldsGrouping("joinBolt", new Fields("user")); return builder.createTopology(); } - public static void main(String[] args){ + public static void main(String[] args) { Config config = ConfigFactory.load(); OozieAuditLogApplication app = new OozieAuditLogApplication(); app.run(config); http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogKafkaDeserializer.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogKafkaDeserializer.java b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogKafkaDeserializer.java index 06d6a3e..b7413e4 100644 --- a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogKafkaDeserializer.java +++ b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogKafkaDeserializer.java @@ -17,7 +17,6 @@ */ package org.apache.eagle.security.oozie.parse; - import org.apache.eagle.dataproc.impl.storm.kafka.SpoutKafkaMessageDeserializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -31,7 +30,7 @@ public class OozieAuditLogKafkaDeserializer implements SpoutKafkaMessageDeserial private static Logger LOG = LoggerFactory.getLogger(OozieAuditLogKafkaDeserializer.class); private Properties props; - public OozieAuditLogKafkaDeserializer(Properties props){ + public OozieAuditLogKafkaDeserializer(Properties props) { this.props = props; } @@ -41,12 +40,12 @@ public class OozieAuditLogKafkaDeserializer implements SpoutKafkaMessageDeserial OozieAuditLogParser parser = new OozieAuditLogParser(); OozieAuditLogObject entity = null; - try{ + try { entity = parser.parse(logLine); - }catch(Exception ex){ + } catch (Exception ex) { LOG.error("Failing oozie parse audit log message", ex); } - if(entity == null){ + if (entity == null) { LOG.warn("Event ignored as it can't be correctly parsed, the log is ", logLine); return null; } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParser.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParser.java b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParser.java index 56228f8..bb88895 100644 --- a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParser.java +++ b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParser.java @@ -17,9 +17,10 @@ */ package org.apache.eagle.security.oozie.parse; -import org.apache.commons.lang.StringUtils; import org.apache.eagle.common.DateTimeUtil; +import org.apache.commons.lang.StringUtils; + import java.text.ParseException; import java.util.ArrayList; import java.util.List; @@ -37,17 +38,17 @@ public class OozieAuditLogParser { private static final String OOZIEAUDIT_FLAG = "(\\w+:\\d+)"; private static final String PREFIX_REGEX = TIMESTAMP_REGEX + WHITE_SPACE_REGEX + LOG_LEVEL_REGEX + WHITE_SPACE_REGEX; - private final static String IP = "IP"; - private final static String USER = "USER"; - private final static String GROUP = "GROUP"; - private final static String APP = "APP"; - private final static String JOBID = "JOBID"; - private final static String OPERATION = "OPERATION"; - private final static String PARAMETER = "PARAMETER"; - private final static String STATUS = "STATUS"; - private final static String HTTPCODE = "HTTPCODE"; - private final static String ERRORCODE = "ERRORCODE"; - private final static String ERRORMESSAGE = "ERRORMESSAGE"; + private static final String IP = "IP"; + private static final String USER = "USER"; + private static final String GROUP = "GROUP"; + private static final String APP = "APP"; + private static final String JOBID = "JOBID"; + private static final String OPERATION = "OPERATION"; + private static final String PARAMETER = "PARAMETER"; + private static final String STATUS = "STATUS"; + private static final String HTTPCODE = "HTTPCODE"; + private static final String ERRORCODE = "ERRORCODE"; + private static final String ERRORMESSAGE = "ERRORMESSAGE"; private static final Pattern LOG_PATTERN = constructPattern(); public OozieAuditLogObject parse(String logLine) throws Exception { http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParserBolt.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParserBolt.java b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParserBolt.java index 13f5e8c..ed59ec1 100644 --- a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParserBolt.java +++ b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/OozieAuditLogParserBolt.java @@ -36,7 +36,7 @@ import java.util.TreeMap; /** * Since 8/12/16. */ -public class OozieAuditLogParserBolt extends BaseRichBolt{ +public class OozieAuditLogParserBolt extends BaseRichBolt { private static Logger LOG = LoggerFactory.getLogger(OozieAuditLogParserBolt.class); private OutputCollector collector; @@ -72,9 +72,9 @@ public class OozieAuditLogParserBolt extends BaseRichBolt{ map.put("errorcode", entity.errorcode); map.put("errormessage", entity.errormessage); collector.emit(Arrays.asList(map)); - }catch(Exception ex){ + } catch (Exception ex) { LOG.error("error in parsing oozie audit log", ex); - }finally { + } finally { collector.ack(input); } } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieResourceSensitivityDataJoinBolt.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieResourceSensitivityDataJoinBolt.java b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieResourceSensitivityDataJoinBolt.java index 91f9e6f..156e326 100644 --- a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieResourceSensitivityDataJoinBolt.java +++ b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieResourceSensitivityDataJoinBolt.java @@ -17,17 +17,14 @@ */ package org.apache.eagle.security.oozie.parse.sensitivity; +import org.apache.eagle.security.enrich.AbstractDataEnrichBolt; +import org.apache.eagle.security.entity.OozieResourceSensitivityAPIEntity; + import backtype.storm.task.OutputCollector; -import backtype.storm.task.TopologyContext; import backtype.storm.topology.OutputFieldsDeclarer; -import backtype.storm.topology.base.BaseRichBolt; import backtype.storm.tuple.Fields; import backtype.storm.tuple.Tuple; import com.typesafe.config.Config; -import org.apache.eagle.security.enrich.AbstractDataEnrichBolt; -import org.apache.eagle.security.entity.OozieResourceSensitivityAPIEntity; -import org.apache.eagle.security.enrich.ExternalDataCache; -import org.apache.eagle.security.enrich.ExternalDataJoiner; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -37,11 +34,11 @@ import java.util.TreeMap; import java.util.regex.Pattern; public class OozieResourceSensitivityDataJoinBolt extends AbstractDataEnrichBolt { - private final static Logger LOG = LoggerFactory.getLogger(OozieResourceSensitivityDataJoinBolt.class); + private static final Logger LOG = LoggerFactory.getLogger(OozieResourceSensitivityDataJoinBolt.class); private Config config; private OutputCollector collector; - public OozieResourceSensitivityDataJoinBolt(Config config){ + public OozieResourceSensitivityDataJoinBolt(Config config) { super(config, new OozieSensitivityDataEnrichLCM(config)); } @@ -72,9 +69,9 @@ public class OozieResourceSensitivityDataJoinBolt extends AbstractDataEnrichBolt } LOG.info("After oozie resource sensitivity lookup: " + newEvent); collector.emit(Arrays.asList(newEvent.get("user"), newEvent)); - }catch(Exception ex){ + } catch (Exception ex) { LOG.error("error join external sensitivity data", ex); - }finally { + } finally { collector.ack(input); } } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieSensitivityDataEnrichLCM.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieSensitivityDataEnrichLCM.java b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieSensitivityDataEnrichLCM.java index b974c5f..eec4058 100644 --- a/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieSensitivityDataEnrichLCM.java +++ b/eagle-security/eagle-security-oozie-auditlog/src/main/java/org/apache/eagle/security/oozie/parse/sensitivity/OozieSensitivityDataEnrichLCM.java @@ -17,19 +17,22 @@ package org.apache.eagle.security.oozie.parse.sensitivity; -import com.typesafe.config.Config; + import org.apache.eagle.security.enrich.AbstractDataEnrichLCM; import org.apache.eagle.security.service.ISecurityDataEnrichServiceClient; +import com.typesafe.config.Config; + import java.util.Collection; /** * Since 8/16/16. */ public class OozieSensitivityDataEnrichLCM extends AbstractDataEnrichLCM { - public OozieSensitivityDataEnrichLCM(Config config){ + public OozieSensitivityDataEnrichLCM(Config config) { super(config); } + @Override protected Collection loadFromService(ISecurityDataEnrichServiceClient client) { return null; http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/OozieResourceSensitivityDataJoiner.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/OozieResourceSensitivityDataJoiner.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/OozieResourceSensitivityDataJoiner.java index 4bfe814..497dba6 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/OozieResourceSensitivityDataJoiner.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/OozieResourceSensitivityDataJoiner.java @@ -29,7 +29,7 @@ import java.util.Map; public class OozieResourceSensitivityDataJoiner { - private final static Logger LOG = LoggerFactory.getLogger(OozieResourceSensitivityDataJoiner.class); + private static final Logger LOG = LoggerFactory.getLogger(OozieResourceSensitivityDataJoiner.class); public List joinOozieResourceSensitivity(String site, List coordinatorJobs) { http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/BadOozieMetadataAccessConfigException.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/BadOozieMetadataAccessConfigException.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/BadOozieMetadataAccessConfigException.java index 7c8b650..63fb57e 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/BadOozieMetadataAccessConfigException.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/BadOozieMetadataAccessConfigException.java @@ -16,12 +16,12 @@ */ package org.apache.eagle.service.security.oozie.dao; -public class BadOozieMetadataAccessConfigException extends RuntimeException{ - public BadOozieMetadataAccessConfigException(Exception ex){ +public class BadOozieMetadataAccessConfigException extends RuntimeException { + public BadOozieMetadataAccessConfigException(Exception ex) { super(ex); } - public BadOozieMetadataAccessConfigException(String msg){ + public BadOozieMetadataAccessConfigException(String msg) { super(msg); } } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfig.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfig.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfig.java index d2e6a76..02115d8 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfig.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfig.java @@ -61,26 +61,26 @@ public class OozieMetadataAccessConfig { return "accessType:" + accessType + ",oozieUrl:" + oozieUrl + ",filter:" + filter + ",authType:" + authType; } - public final static class OOZIECONF { - public final static String ACCESSTYPE = "accessType"; - public final static String OOZIEURL = "oozieUrl"; - public final static String FILTER = "filter"; - public final static String AUTHTYPE = "authType"; + public static final class OozieConf { + public static final String ACCESSTYPE = "accessType"; + public static final String OOZIEURL = "oozieUrl"; + public static final String FILTER = "filter"; + public static final String AUTHTYPE = "authType"; } public static OozieMetadataAccessConfig config2Entity(Config config) { OozieMetadataAccessConfig oozieconf = new OozieMetadataAccessConfig(); - if(config.hasPath(OOZIECONF.ACCESSTYPE)) { - oozieconf.setAccessType(config.getString(OOZIECONF.ACCESSTYPE)); + if (config.hasPath(OozieConf.ACCESSTYPE)) { + oozieconf.setAccessType(config.getString(OozieConf.ACCESSTYPE)); } - if(config.hasPath(OOZIECONF.OOZIEURL)) { - oozieconf.setOozieUrl(config.getString(OOZIECONF.OOZIEURL)); + if (config.hasPath(OozieConf.OOZIEURL)) { + oozieconf.setOozieUrl(config.getString(OozieConf.OOZIEURL)); } - if(config.hasPath(OOZIECONF.FILTER)) { - oozieconf.setFilter(config.getString(OOZIECONF.FILTER)); + if (config.hasPath(OozieConf.FILTER)) { + oozieconf.setFilter(config.getString(OozieConf.FILTER)); } - if(config.hasPath(OOZIECONF.AUTHTYPE)) { - oozieconf.setAuthType(config.getString(OOZIECONF.AUTHTYPE)); + if (config.hasPath(OozieConf.AUTHTYPE)) { + oozieconf.setAuthType(config.getString(OozieConf.AUTHTYPE)); } return oozieconf; } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfigDAOImpl.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfigDAOImpl.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfigDAOImpl.java index 8cab617..320e1b4 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfigDAOImpl.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataAccessConfigDAOImpl.java @@ -16,12 +16,13 @@ */ package org.apache.eagle.service.security.oozie.dao; -import com.typesafe.config.Config; import org.apache.eagle.security.resolver.MetadataAccessConfigRepo; -public class OozieMetadataAccessConfigDAOImpl implements OozieMetadataAccessConfigDAO{ +import com.typesafe.config.Config; + +public class OozieMetadataAccessConfigDAOImpl implements OozieMetadataAccessConfigDAO { @Override - public OozieMetadataAccessConfig getConfig(String site) throws Exception{ + public OozieMetadataAccessConfig getConfig(String site) throws Exception { MetadataAccessConfigRepo repo = new MetadataAccessConfigRepo(); Config config = repo.getConfig("oozieAuditLog", site); return OozieMetadataAccessConfig.config2Entity(config); http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataDAOImpl.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataDAOImpl.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataDAOImpl.java index 65b4bcb..adbf7e6 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataDAOImpl.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieMetadataDAOImpl.java @@ -32,8 +32,9 @@ public class OozieMetadataDAOImpl implements OozieMetadataDAO { private OozieMetadataAccessConfig config; public OozieMetadataDAOImpl(OozieMetadataAccessConfig config) { - if (config.getAccessType() == null) + if (config.getAccessType() == null) { throw new BadOozieMetadataAccessConfigException("access Type is null, options: [oozie_api]"); + } this.config = config; } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAO.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAO.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAO.java index dc16d01..972c515 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAO.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAO.java @@ -19,11 +19,12 @@ package org.apache.eagle.service.security.oozie.dao; import java.util.Map; /** - * retrieve oozie sensitivity metadata directly from table + * retrieve oozie sensitivity metadata directly from table. */ public interface OozieSensitivityMetadataDAO { // site to (resource, sensitivityType) Map> getAllOozieSensitivityMap(); + // resource to sensitivityType Map getOozieSensitivityMap(String site); } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAOImpl.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAOImpl.java b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAOImpl.java index 247eb90..c25ad6a 100644 --- a/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAOImpl.java +++ b/eagle-security/eagle-security-oozie-web/src/main/java/org/apache/eagle/service/security/oozie/dao/OozieSensitivityMetadataDAOImpl.java @@ -27,32 +27,32 @@ import java.util.HashMap; import java.util.List; import java.util.Map; -public class OozieSensitivityMetadataDAOImpl implements OozieSensitivityMetadataDAO{ +public class OozieSensitivityMetadataDAOImpl implements OozieSensitivityMetadataDAO { private static Logger LOG = LoggerFactory.getLogger(OozieSensitivityMetadataDAOImpl.class); @Override - public Map> getAllOozieSensitivityMap(){ + public Map> getAllOozieSensitivityMap() { GenericEntityServiceResource resource = new GenericEntityServiceResource(); /* parameters are: query, startTime, endTime, pageSzie, startRowkey, treeAgg, timeSeries, intervalmin, top, filterIfMissing, * parallel, metricName*/ GenericServiceAPIResponseEntity ret = resource.search("OozieResourceSensitivityService[]{*}", null, null, Integer.MAX_VALUE, null, false, false, 0L, 0, false, 0, null, false); List list = (List) ret.getObj(); - if( list == null ) + if (list == null) { return Collections.emptyMap(); + } Map> res = new HashMap>(); - for(OozieResourceSensitivityAPIEntity entity : list){ + for (OozieResourceSensitivityAPIEntity entity : list) { String site = entity.getTags().get("site"); - if(entity.getTags().containsKey("oozieResource")) { - if(res.get(site) == null){ + if (entity.getTags().containsKey("oozieResource")) { + if (res.get(site) == null) { res.put(site, new HashMap()); } Map resSensitivityMap = res.get(site); resSensitivityMap.put(entity.getTags().get("oozieResource"), entity.getSensitivityType()); - } - else { - if(LOG.isDebugEnabled()) { + } else { + if (LOG.isDebugEnabled()) { LOG.debug("An invalid sensitivity entity is detected" + entity); } } @@ -61,17 +61,18 @@ public class OozieSensitivityMetadataDAOImpl implements OozieSensitivityMetadata } @Override - public Map getOozieSensitivityMap(String site){ + public Map getOozieSensitivityMap(String site) { GenericEntityServiceResource resource = new GenericEntityServiceResource(); String queryFormat = "OozieResourceSensitivityService[@site=\"%s\"]{*}"; GenericServiceAPIResponseEntity ret = resource.search(String.format(queryFormat, site), null, null, Integer.MAX_VALUE, null, false, false, 0L, 0, false, 0, null, false); List list = (List) ret.getObj(); - if( list == null ) + if (list == null) { return Collections.emptyMap(); + } Map resSensitivityMap = new HashMap(); - for(OozieResourceSensitivityAPIEntity entity : list){ - if(entity.getTags().containsKey("oozieResource")) { + for (OozieResourceSensitivityAPIEntity entity : list) { + if (entity.getTags().containsKey("oozieResource")) { resSensitivityMap.put(entity.getTags().get("oozieResource"), entity.getSensitivityType()); } } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/6a55b59f/eagle-security/eagle-security-oozie-web/src/test/java/org/apache/eagle/service/security/oozie/TestOozieResourceSensitivityDataJoiner.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-oozie-web/src/test/java/org/apache/eagle/service/security/oozie/TestOozieResourceSensitivityDataJoiner.java b/eagle-security/eagle-security-oozie-web/src/test/java/org/apache/eagle/service/security/oozie/TestOozieResourceSensitivityDataJoiner.java index 50cd1f1..ac3cc0b 100644 --- a/eagle-security/eagle-security-oozie-web/src/test/java/org/apache/eagle/service/security/oozie/TestOozieResourceSensitivityDataJoiner.java +++ b/eagle-security/eagle-security-oozie-web/src/test/java/org/apache/eagle/service/security/oozie/TestOozieResourceSensitivityDataJoiner.java @@ -16,11 +16,7 @@ */ package org.apache.eagle.service.security.oozie; -import org.apache.eagle.log.entity.GenericServiceAPIResponseEntity; import org.apache.eagle.security.entity.OozieResourceEntity; -import org.apache.eagle.security.entity.OozieResourceSensitivityAPIEntity; -import org.apache.eagle.service.generic.GenericEntityServiceResource; -import org.apache.eagle.service.security.oozie.dao.OozieSensitivityMetadataDAO; import org.apache.eagle.service.security.oozie.dao.OozieSensitivityMetadataDAOImpl; import org.apache.oozie.client.CoordinatorJob; import org.apache.oozie.client.rest.JsonTags; @@ -41,9 +37,7 @@ import java.util.List; import java.util.Map; import static org.easymock.EasyMock.expect; -import static org.powermock.api.easymock.PowerMock.createMock; -import static org.powermock.api.easymock.PowerMock.expectNew; -import static org.powermock.api.easymock.PowerMock.replay; +import static org.powermock.api.easymock.PowerMock.*; @RunWith(PowerMockRunner.class) @PrepareForTest(OozieResourceSensitivityDataJoiner.class)