Return-Path: X-Original-To: apmail-tez-commits-archive@minotaur.apache.org Delivered-To: apmail-tez-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8A2D9182D0 for ; Thu, 18 Feb 2016 09:58:02 +0000 (UTC) Received: (qmail 67629 invoked by uid 500); 18 Feb 2016 09:57:56 -0000 Delivered-To: apmail-tez-commits-archive@tez.apache.org Received: (qmail 67503 invoked by uid 500); 18 Feb 2016 09:57:56 -0000 Mailing-List: contact commits-help@tez.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@tez.apache.org Delivered-To: mailing list commits@tez.apache.org Received: (qmail 67296 invoked by uid 99); 18 Feb 2016 09:57:55 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2016 09:57:55 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id A3AD9DFF67; Thu, 18 Feb 2016 09:57:55 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sree@apache.org To: commits@tez.apache.org Date: Thu, 18 Feb 2016 09:57:59 -0000 Message-Id: In-Reply-To: <56a3befb33c349f7bf43e58a8338f007@git.apache.org> References: <56a3befb33c349f7bf43e58a8338f007@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [05/24] tez git commit: TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services (zjffdu) TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services (zjffdu) Project: http://git-wip-us.apache.org/repos/asf/tez/repo Commit: http://git-wip-us.apache.org/repos/asf/tez/commit/2bf27de3 Tree: http://git-wip-us.apache.org/repos/asf/tez/tree/2bf27de3 Diff: http://git-wip-us.apache.org/repos/asf/tez/diff/2bf27de3 Branch: refs/heads/TEZ-2980 Commit: 2bf27de3bfbe7a54fbf2982fa50ab6b32974d6b1 Parents: 92def52 Author: Jeff Zhang Authored: Thu Jan 21 21:32:59 2016 -0800 Committer: Jeff Zhang Committed: Thu Jan 21 21:38:40 2016 -0800 ---------------------------------------------------------------------- CHANGES.txt | 2 ++ .../tez/dag/app/dag/impl/TaskAttemptImpl.java | 24 +++++++++++++------- 2 files changed, 18 insertions(+), 8 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/tez/blob/2bf27de3/CHANGES.txt ---------------------------------------------------------------------- diff --git a/CHANGES.txt b/CHANGES.txt index 4ffcf13..91c86f8 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -6,6 +6,7 @@ Release 0.8.3: Unreleased INCOMPATIBLE CHANGES ALL CHANGES: + TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services. TEZ-3036. Tez AM can hang on startup with no indication of error TEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED TEZ-2594. Fix LICENSE for missing entries for full and minimal tarballs. @@ -320,6 +321,7 @@ INCOMPATIBLE CHANGES TEZ-2949. Allow duplicate dag names within session for Tez. ALL CHANGES + TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services. TEZ-3036. Tez AM can hang on startup with no indication of error TEZ-3052. Task internal error due to Invalid event: T_ATTEMPT_FAILED at FAILED TEZ-2937. Can Processor.close() be called after closing inputs and outputs? http://git-wip-us.apache.org/repos/asf/tez/blob/2bf27de3/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java ---------------------------------------------------------------------- diff --git a/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java b/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java index dda4891..0affff2 100644 --- a/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java +++ b/tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/TaskAttemptImpl.java @@ -1785,15 +1785,23 @@ public class TaskAttemptImpl implements TaskAttempt, public void setLastEventSent(TezEvent lastEventSent) { writeLock.lock(); try { - DataEventDependencyInfo info = new DataEventDependencyInfo( + // TEZ-3066 ideally Heartbeat just happens in FAIL_IN_PROGRESS & KILL_IN_PROGRESS, + // add other states here just in case. create TEZ-3068 for a more elegant solution. + if (!EnumSet.of(TaskAttemptStateInternal.FAIL_IN_PROGRESS, + TaskAttemptStateInternal.KILL_IN_PROGRESS, + TaskAttemptStateInternal.FAILED, + TaskAttemptStateInternal.KILLED, + TaskAttemptStateInternal.SUCCEEDED).contains(getInternalState())) { + DataEventDependencyInfo info = new DataEventDependencyInfo( lastEventSent.getEventReceivedTime(), lastEventSent.getSourceInfo().getTaskAttemptID()); - // task attempt id may be null for input data information events - if (appendNextDataEvent) { - appendNextDataEvent = false; - lastDataEvents.add(info); - } else { - // over-write last event - array list makes it quick - lastDataEvents.set(lastDataEvents.size() - 1, info); + // task attempt id may be null for input data information events + if (appendNextDataEvent) { + appendNextDataEvent = false; + lastDataEvents.add(info); + } else { + // over-write last event - array list makes it quick + lastDataEvents.set(lastDataEvents.size() - 1, info); + } } } finally { writeLock.unlock();