Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E624F200B5C for ; Thu, 28 Jul 2016 05:13:37 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E30CB160A93; Thu, 28 Jul 2016 03:13:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BF1C0160A90 for ; Thu, 28 Jul 2016 05:13:36 +0200 (CEST) Received: (qmail 80068 invoked by uid 500); 28 Jul 2016 03:13:35 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 80059 invoked by uid 99); 28 Jul 2016 03:13:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jul 2016 03:13:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 70BA9C0778 for ; Thu, 28 Jul 2016 03:13:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.506 X-Spam-Level: X-Spam-Status: No, score=-4.506 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-1.287, WEIRD_PORT=0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id MoHeC1UWw7si for ; Thu, 28 Jul 2016 03:13:30 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with SMTP id A7F2A5FBF9 for ; Thu, 28 Jul 2016 03:13:23 +0000 (UTC) Received: (qmail 78119 invoked by uid 99); 28 Jul 2016 03:13:20 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jul 2016 03:13:20 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id CB07AE02A2; Thu, 28 Jul 2016 03:13:20 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: mw@apache.org To: commits@eagle.incubator.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: incubator-eagle git commit: fix typo in files Date: Thu, 28 Jul 2016 03:13:20 +0000 (UTC) archived-at: Thu, 28 Jul 2016 03:13:38 -0000 Repository: incubator-eagle Updated Branches: refs/heads/develop 8777936d2 -> 4d7ccd18e fix typo in files Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/4d7ccd18 Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/4d7ccd18 Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/4d7ccd18 Branch: refs/heads/develop Commit: 4d7ccd18e817ac65e81fe80ca992801b6ed90174 Parents: 8777936 Author: anyway1021 Authored: Thu Jul 28 11:12:46 2016 +0800 Committer: anyway1021 Committed: Thu Jul 28 11:12:46 2016 +0800 ---------------------------------------------------------------------- .../eagle-stream-pipeline/src/test/resources/pipeline_3.conf | 2 +- .../eagle/hadoop/queue/crawler/SchedulerInfoCrawler.java | 2 +- .../eagle/jpm/mr/history/parser/JHFMRVer1EventReader.java | 2 +- .../src/main/resources/MRErrorCategory.config | 1 - .../eagle/jpm/spark/history/crawl/JHFSparkEventReader.java | 6 ++++-- .../eagle-jpm-util/src/main/resources/application.properties | 2 +- .../security/hdfs/test/HDFSResourceAccessConfigTest.java | 6 +++--- .../service/security/pwdgen/PasswordEncoderGenerator.java | 2 +- 8 files changed, 12 insertions(+), 11 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-core/eagle-data-process/eagle-stream-pipeline/src/test/resources/pipeline_3.conf ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-data-process/eagle-stream-pipeline/src/test/resources/pipeline_3.conf b/eagle-core/eagle-data-process/eagle-stream-pipeline/src/test/resources/pipeline_3.conf index b1c1955..9dc7ce3 100644 --- a/eagle-core/eagle-data-process/eagle-stream-pipeline/src/test/resources/pipeline_3.conf +++ b/eagle-core/eagle-data-process/eagle-stream-pipeline/src/test/resources/pipeline_3.conf @@ -34,7 +34,7 @@ "site" : "sandbox" "application": "HADOOP" "dataJoinPollIntervalSec" : 30 - "mailHost" : "atom.corp.ebay.com" + "mailHost" : "some.mail.server" "mailSmtpPort":"25" "mailDebug" : "true" "eagleService": { http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-jpm/eagle-hadoop-queue/src/main/java/org/apache/eagle/hadoop/queue/crawler/SchedulerInfoCrawler.java ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-hadoop-queue/src/main/java/org/apache/eagle/hadoop/queue/crawler/SchedulerInfoCrawler.java b/eagle-jpm/eagle-hadoop-queue/src/main/java/org/apache/eagle/hadoop/queue/crawler/SchedulerInfoCrawler.java index f25fa20..f7cd5b4 100755 --- a/eagle-jpm/eagle-hadoop-queue/src/main/java/org/apache/eagle/hadoop/queue/crawler/SchedulerInfoCrawler.java +++ b/eagle-jpm/eagle-hadoop-queue/src/main/java/org/apache/eagle/hadoop/queue/crawler/SchedulerInfoCrawler.java @@ -42,7 +42,7 @@ public class SchedulerInfoCrawler implements Runnable { @Override public void run() { try { - //https://apollo-phx-rm-2.vip.ebay.com:50030/ws/v1/cluster/scheduler?anonymous=true + //https://some.server.address:50030/ws/v1/cluster/scheduler?anonymous=true logger.info("Start to crawl cluster scheduler queues from " + this.urlString); SchedulerWrapper schedulerWrapper = (SchedulerWrapper) HadoopYarnResourceUtils.getObjectFromStreamWithGzip(urlString, SchedulerWrapper.class); if (schedulerWrapper == null || schedulerWrapper.getScheduler() == null) { http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-jpm/eagle-jpm-mr-history/src/main/java/org/apache/eagle/jpm/mr/history/parser/JHFMRVer1EventReader.java ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-mr-history/src/main/java/org/apache/eagle/jpm/mr/history/parser/JHFMRVer1EventReader.java b/eagle-jpm/eagle-jpm-mr-history/src/main/java/org/apache/eagle/jpm/mr/history/parser/JHFMRVer1EventReader.java index 24e3563..7fe8909 100644 --- a/eagle-jpm/eagle-jpm-mr-history/src/main/java/org/apache/eagle/jpm/mr/history/parser/JHFMRVer1EventReader.java +++ b/eagle-jpm/eagle-jpm-mr-history/src/main/java/org/apache/eagle/jpm/mr/history/parser/JHFMRVer1EventReader.java @@ -85,7 +85,7 @@ public class JHFMRVer1EventReader extends JHFEventReaderBase implements JHFMRVer String rack = null; // we get rack/hostname based on task's status if (values.get(Keys.TASK_STATUS).equals(EagleTaskStatus.SUCCESS.name())) { - // in CDH4.1.4, the hostname has the format of /default/rack128/phxdpehdc10dn0338.stratus.phx.ebay.com + // in CDH4.1.4, the hostname has the format of /default/rack128/some.server.address // if not specified, the hostname has the format of /default-rack/ String decoratedHostname = values.get(Keys.HOSTNAME); String[] tmp = decoratedHostname.split("/"); http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-jpm/eagle-jpm-mr-history/src/main/resources/MRErrorCategory.config ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-mr-history/src/main/resources/MRErrorCategory.config b/eagle-jpm/eagle-jpm-mr-history/src/main/resources/MRErrorCategory.config index ee8c0c5..de5072a 100644 --- a/eagle-jpm/eagle-jpm-mr-history/src/main/resources/MRErrorCategory.config +++ b/eagle-jpm/eagle-jpm-mr-history/src/main/resources/MRErrorCategory.config @@ -22,7 +22,6 @@ CONTAINER_KILLED_BY_APPMASTER = false|Container killed by the ApplicationMaster CONTAINER_CLEANUP_FAILURE = false|cleanup failed for container container_ USER_NOT_FOUND = false|User \S+ not found TASK_TREE_BEYOND_MEMORY_LIMIT = false|^TaskTree \S+ is running beyond memory-limits -EBAY_APPMON_LOG_GET_TRAVERSER = false|^Error: com.ebay.appmon.log.traverser.LogTraverser.getTraverser MAP_OUTPUT_LOST = false|Map output lost BEYOND_PHYSICAL_MEMORY_LIMITS = false|running beyond physical memory limits GC_OVERHEAD_LIMIT_EXCEEDED = false|GC overhead limit exceeded http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/crawl/JHFSparkEventReader.java ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/crawl/JHFSparkEventReader.java b/eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/crawl/JHFSparkEventReader.java index db5d432..99aa68d 100644 --- a/eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/crawl/JHFSparkEventReader.java +++ b/eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/crawl/JHFSparkEventReader.java @@ -110,7 +110,7 @@ public class JHFSparkEventReader { List jobConfs = conf.getStringList("basic.jobConf.additional.info"); String[] props = {"spark.yarn.app.id", "spark.executor.memory", "spark.driver.host", "spark.driver.port", - "spark.driver.memory", "ebay.job.name", "spark.scheduler.pool", "spark.executor.cores", "spark.yarn.am.memory", + "spark.driver.memory", "spark.scheduler.pool", "spark.executor.cores", "spark.yarn.am.memory", "spark.yarn.am.cores", "spark.yarn.executor.memoryOverhead", "spark.yarn.driver.memoryOverhead", "spark.yarn.am.memoryOverhead", "spark.master"}; jobConfs.addAll(Arrays.asList(props)); for (String prop : jobConfs) { @@ -158,7 +158,9 @@ public class JHFSparkEventReader { entity.getTags().put(SparkJobTagName.SPARK_APP_ID.toString(), JSONUtil.getString(event, "App ID")); entity.getTags().put(SparkJobTagName.SPARK_APP_NAME.toString(), JSONUtil.getString(event, "App Name")); entity.getTags().put(SparkJobTagName.SPARK_APP_ATTEMPT_ID.toString(), JSONUtil.getString(event, "App Attempt ID")); - entity.getTags().put(SparkJobTagName.SPARK_APP_NORM_NAME.toString(), this.getNormalizedName(JSONUtil.getString(event, "App Name"), this.app.getConfig().getConfig().get("ebay.job.name"))); + // the second argument of getNormalizeName() is changed to null because the original code contains sensitive text + // original second argument looks like: this.app.getConfig().getConfig().get("xxx"), "xxx" is the sensitive text + entity.getTags().put(SparkJobTagName.SPARK_APP_NORM_NAME.toString(), this.getNormalizedName(JSONUtil.getString(event, "App Name"), null)); entity.getTags().put(SparkJobTagName.SPARK_USER.toString(), JSONUtil.getString(event, "User")); } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-jpm/eagle-jpm-util/src/main/resources/application.properties ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-util/src/main/resources/application.properties b/eagle-jpm/eagle-jpm-util/src/main/resources/application.properties index d6f2a5b..b180a41 100644 --- a/eagle-jpm/eagle-jpm-util/src/main/resources/application.properties +++ b/eagle-jpm/eagle-jpm-util/src/main/resources/application.properties @@ -19,5 +19,5 @@ # */ # -### job name normalization rules, please refer to wiki https://wiki.vip.corp.ebay.com/display/DI/Job+name+normalization +### job name normalization rules, please refer to wiki job.name.normalization.rules.key="^(.*)[0-9]{4}/[0-9]{2}/[0-9]{2}/[0-9]{2}$ => $1~ ; ^(oozie:launcher):T=(.*):W=(.*):A=(.*):ID=(?:.*)$ => $1-$2-$3-$4~ ; ^(.*)([0-9]{10})$ => $1~ ; ^[0-9a-z]{8}-[0-9a-z]{4}-[0-9a-z]{4}-[0-9a-z]{4}-[0-9a-z]{12}$ => uuid~ ; ^(.*)(?:[0-9]{8}/[0-9]{2}_[0-9]{2})$ => $1~" \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-security/eagle-security-hdfs-web/src/test/java/org/apache/eagle/service/security/hdfs/test/HDFSResourceAccessConfigTest.java ---------------------------------------------------------------------- diff --git a/eagle-security/eagle-security-hdfs-web/src/test/java/org/apache/eagle/service/security/hdfs/test/HDFSResourceAccessConfigTest.java b/eagle-security/eagle-security-hdfs-web/src/test/java/org/apache/eagle/service/security/hdfs/test/HDFSResourceAccessConfigTest.java index bfc1df9..8ad7cc0 100644 --- a/eagle-security/eagle-security-hdfs-web/src/test/java/org/apache/eagle/service/security/hdfs/test/HDFSResourceAccessConfigTest.java +++ b/eagle-security/eagle-security-hdfs-web/src/test/java/org/apache/eagle/service/security/hdfs/test/HDFSResourceAccessConfigTest.java @@ -26,15 +26,15 @@ public class HDFSResourceAccessConfigTest { @Test public void testHDFSResourceAccessConfig() throws Exception { - String configStr = "{\"fs.defaultFS\":\"hdfs://sandbox-nn-ha\",\"dfs.nameservices\":\"sandbox-nn-ha\",\"dfs.ha.namenodes.sandbox-nn-ha\":\"nn1,nn2\",\"dfs.namenode.rpc-address.sandbox-nn-ha.nn1\":\"sandbox-nn.vip.ebay.com:8020\",\"dfs.namenode.rpc-address.sandbox-nn-ha.nn2\":\"sandbox-nn-2.vip.ebay.com:8020\",\"dfs.client.failover.proxy.provider.sandbox-nn-ha\":\"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider\",\"hadoop.security.authentication\":\"kerberos\",\"dfs.namenode.kerberos.principal\":\"hadoop/_HOST@EXAMPLE.COM\"}"; + String configStr = "{\"fs.defaultFS\":\"hdfs://sandbox-nn-ha\",\"dfs.nameservices\":\"sandbox-nn-ha\",\"dfs.ha.namenodes.sandbox-nn-ha\":\"nn1,nn2\",\"dfs.namenode.rpc-address.sandbox-nn-ha.nn1\":\"sandbox-nn.some.server.com:8020\",\"dfs.namenode.rpc-address.sandbox-nn-ha.nn2\":\"sandbox-nn-2.some.server.com:8020\",\"dfs.client.failover.proxy.provider.sandbox-nn-ha\":\"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider\",\"hadoop.security.authentication\":\"kerberos\",\"dfs.namenode.kerberos.principal\":\"hadoop/_HOST@EXAMPLE.COM\"}"; ObjectMapper mapper = new ObjectMapper(); Map configMap = mapper.readValue(configStr, Map.class); Map result = new HashMap<>(); result.put("fs.defaultFS" , "hdfs://sandbox-nn-ha"); result.put("dfs.nameservices", "sandbox-nn-ha"); result.put("dfs.ha.namenodes.sandbox-nn-ha", "nn1,nn2"); - result.put("dfs.namenode.rpc-address.sandbox-nn-ha.nn1", "sandbox-nn.vip.ebay.com:8020"); - result.put("dfs.namenode.rpc-address.sandbox-nn-ha.nn2", "sandbox-nn-2.vip.ebay.com:8020"); + result.put("dfs.namenode.rpc-address.sandbox-nn-ha.nn1", "sandbox-nn.some.server.com:8020"); + result.put("dfs.namenode.rpc-address.sandbox-nn-ha.nn2", "sandbox-nn-2.some.server.com:8020"); result.put("dfs.client.failover.proxy.provider.sandbox-nn-ha","org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"); result.put("hadoop.security.authentication", "kerberos"); result.put("dfs.namenode.kerberos.principal", "hadoop/_HOST@EXAMPLE.COM"); http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/4d7ccd18/eagle-webservice/src/main/java/org/apache/eagle/service/security/pwdgen/PasswordEncoderGenerator.java ---------------------------------------------------------------------- diff --git a/eagle-webservice/src/main/java/org/apache/eagle/service/security/pwdgen/PasswordEncoderGenerator.java b/eagle-webservice/src/main/java/org/apache/eagle/service/security/pwdgen/PasswordEncoderGenerator.java index 1a855e2..ffbdcc2 100644 --- a/eagle-webservice/src/main/java/org/apache/eagle/service/security/pwdgen/PasswordEncoderGenerator.java +++ b/eagle-webservice/src/main/java/org/apache/eagle/service/security/pwdgen/PasswordEncoderGenerator.java @@ -20,7 +20,7 @@ import org.springframework.security.crypto.bcrypt.BCryptPasswordEncoder; public class PasswordEncoderGenerator { public static void main(String[] args) { - String password = "eagle@ebay.com"; + String password = "eagle@fake.pwd"; BCryptPasswordEncoder passwordEncoder = new BCryptPasswordEncoder(); String hashedPassword = passwordEncoder.encode(password); System.out.println(hashedPassword);