Return-Path: 
 <mapreduce-issues-return-78161-apmail-hadoop-mapreduce-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 730E5180A5
	for <apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org>;
 Mon,  4 May 2015 09:07:11 +0000 (UTC)
Received: (qmail 96593 invoked by uid 500); 4 May 2015 09:07:06 -0000
Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org
Received: (qmail 96530 invoked by uid 500); 4 May 2015 09:07:06 -0000
Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:mapreduce-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:mapreduce-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:mapreduce-issues@hadoop.apache.org>
List-Id: <mapreduce-issues.hadoop.apache.org>
Reply-To: mapreduce-issues@hadoop.apache.org
Delivered-To: mailing list mapreduce-issues@hadoop.apache.org
Received: (qmail 96519 invoked by uid 99); 4 May 2015 09:07:06 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 May 2015 09:07:06 +0000
Date: Mon, 4 May 2015 09:07:06 +0000 (UTC)
From: "Laxman (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12826736.1430585468000.68459.1430730426172@Atlassian.JIRA>
In-Reply-To: <JIRA.12826736.1430585468000@Atlassian.JIRA>
References: <JIRA.12826736.1430585468000@Atlassian.JIRA>
 <JIRA.12826736.1430585468995@arcas>
Subject: [jira] [Commented] (MAPREDUCE-6351) Reducer hung in copy phase.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526438#comment-14526438 ] 

Laxman commented on MAPREDUCE-6351:
-----------------------------------

"Threads analysis" mentioned in description above found to be incorrect when I retrace the code flow. Pre-notification is not a problem as merger wait is guarded by size check.

However, problem exists, fetchers are not proceeding and waiting for merger to free some memory and merge doing nothing.

> Reducer hung in copy phase.
> ---------------------------
>
>                 Key: MAPREDUCE-6351
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.6.0
>            Reporter: Laxman
>         Attachments: jstat-gc.log, reducer-container-partial.log.zip, thread-dumps.out
>
>
> *Problem*
> Reducer gets stuck in copy phase and doesn't make progress for very long time. After killing this task for couple of times manually, it gets completed. 
> *Observations*
> - Verfied gc logs. Found no memory related issues. Attached the logs.
> - Verified thread dumps. Found no thread related problems. 
> - On verification of logs, fetcher threads are not copying the map outputs and they are just waiting for merge to happen.
> - Merge thread is alive and in wait state.
> *Analysis* 
> On careful observation of logs, thread dumps and code, this looks to me like a classic case of multi-threading issue. Thread goes to wait state after it has been notified. 
> Here is the suspect code flow.
> *Thread #1*
> Fetcher thread - notification comes first
> org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(Set<T>)
> {code}
>       synchronized(pendingToBeMerged) {
>         pendingToBeMerged.addLast(toMergeInputs);
>         pendingToBeMerged.notifyAll();
>       }
> {code}
> *Thread #2*
> Merge Thread - goes to wait state (Notification goes unconsumed)
> org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
> {code}
>         synchronized (pendingToBeMerged) {
>           while(pendingToBeMerged.size() <= 0) {
>             pendingToBeMerged.wait();
>           }
>           // Pickup the inputs to merge.
>           inputs = pendingToBeMerged.removeFirst();
>         }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)