Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 264A9200BBD for ; Tue, 25 Oct 2016 06:08:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 250A1160B00; Tue, 25 Oct 2016 04:08:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6B930160AEB for ; Tue, 25 Oct 2016 06:08:04 +0200 (CEST) Received: (qmail 71161 invoked by uid 500); 25 Oct 2016 04:08:03 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 71149 invoked by uid 99); 25 Oct 2016 04:08:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Oct 2016 04:08:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0769E1862A0 for ; Tue, 25 Oct 2016 04:08:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 7XgMlHifCt4V for ; Tue, 25 Oct 2016 04:08:01 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id E23D65FAC5 for ; Tue, 25 Oct 2016 04:08:00 +0000 (UTC) Received: (qmail 71117 invoked by uid 99); 25 Oct 2016 04:08:00 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Oct 2016 04:08:00 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id D632DDFB90; Tue, 25 Oct 2016 04:07:59 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: jinhuwu@apache.org To: commits@eagle.incubator.apache.org Message-Id: <2ef8f9a7f47b4625bf52dd5e7ee7994a@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: =?utf-8?q?incubator-eagle_git_commit=3A_=5BEAGLE-679=5D_Fix_missin?= =?utf-8?q?g_CoprocessorService_and_MR_Running_feeder_task_number_limi?= =?utf-8?b?4oCm?= Date: Tue, 25 Oct 2016 04:07:59 +0000 (UTC) archived-at: Tue, 25 Oct 2016 04:08:05 -0000 Repository: incubator-eagle Updated Branches: refs/heads/master 6b87fb3a7 -> 15f39705d [EAGLE-679] Fix missing CoprocessorService and MR Running feeder task number limi… Author: wujinhu Closes #558 from wujinhu/EAGLE-679. Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/15f39705 Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/15f39705 Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/15f39705 Branch: refs/heads/master Commit: 15f39705d0e4c86803d1c552810a3adf2f9b50d6 Parents: 6b87fb3 Author: wujinhu Authored: Tue Oct 25 12:07:48 2016 +0800 Committer: wujinhu Committed: Tue Oct 25 12:07:48 2016 +0800 ---------------------------------------------------------------------- .../apache/eagle/jpm/mr/running/parser/MRJobParser.java | 11 +++++++++++ eagle-topology-assembly/pom.xml | 4 ---- 2 files changed, 11 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/15f39705/eagle-jpm/eagle-jpm-mr-running/src/main/java/org/apache/eagle/jpm/mr/running/parser/MRJobParser.java ---------------------------------------------------------------------- diff --git a/eagle-jpm/eagle-jpm-mr-running/src/main/java/org/apache/eagle/jpm/mr/running/parser/MRJobParser.java b/eagle-jpm/eagle-jpm-mr-running/src/main/java/org/apache/eagle/jpm/mr/running/parser/MRJobParser.java index 0abf9a0..945e8ce 100644 --- a/eagle-jpm/eagle-jpm-mr-running/src/main/java/org/apache/eagle/jpm/mr/running/parser/MRJobParser.java +++ b/eagle-jpm/eagle-jpm-mr-running/src/main/java/org/apache/eagle/jpm/mr/running/parser/MRJobParser.java @@ -78,6 +78,7 @@ public class MRJobParser implements Runnable { private List configKeys; private static final int TOP_BOTTOM_TASKS_BY_ELAPSED_TIME = 10; private static final int FLUSH_TASKS_EVERY_TIME = 5; + private static final int MAX_TASKS_PERMIT = 5000; static { OBJ_MAPPER.configure(JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS, true); @@ -428,6 +429,16 @@ public class MRJobParser implements Runnable { } private Function fetchTasks = jobId -> { + try { + JobExecutionAPIEntity entity = this.mrJobEntityMap.get(jobId); + int taskNumber = entity.getNumTotalMaps() + entity.getNumTotalReduces(); + if (taskNumber > MAX_TASKS_PERMIT) { + LOG.info("too many tasks {}, ignore tasks", taskNumber); + return true; + } + } catch (Exception e) { + return true; + } String taskURL = app.getTrackingUrl() + Constants.MR_JOBS_URL + "/" + jobId + "/" + Constants.MR_TASKS_URL + "?" + Constants.ANONYMOUS_PARAMETER; InputStream is = null; List tasks = null; http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/15f39705/eagle-topology-assembly/pom.xml ---------------------------------------------------------------------- diff --git a/eagle-topology-assembly/pom.xml b/eagle-topology-assembly/pom.xml index a9dddb0..f5aee5f 100644 --- a/eagle-topology-assembly/pom.xml +++ b/eagle-topology-assembly/pom.xml @@ -42,10 +42,6 @@ ${project.version} - org.apache.hbase - hbase-server - - javax.servlet servlet-api