Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 91844 invoked from network); 21 Sep 2009 10:10:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 21 Sep 2009 10:10:33 -0000 Received: (qmail 82773 invoked by uid 500); 21 Sep 2009 10:10:32 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 82686 invoked by uid 500); 21 Sep 2009 10:10:32 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 82676 invoked by uid 99); 21 Sep 2009 10:10:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Sep 2009 10:10:32 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Sep 2009 10:10:20 +0000 Received: from localhost (localhost [127.0.0.1]) by tobor.hpl.hp.com (Postfix) with ESMTP id 1DDB7B7CD1 for ; Mon, 21 Sep 2009 11:09:59 +0100 (BST) X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com Received: from tobor.hpl.hp.com ([127.0.0.1]) by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id A4Hv+x6LImDF for ; Mon, 21 Sep 2009 11:09:53 +0100 (BST) Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tobor.hpl.hp.com (Postfix) with ESMTPS id 3FF78B7CCF for ; Mon, 21 Sep 2009 11:09:53 +0100 (BST) MailScanner-NULL-Check: 1254132580.76735@ZTxUz5Q6IUlaVMRp/KAQgg Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158]) by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n8LA9eHH008689 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 21 Sep 2009 11:09:40 +0100 (BST) Message-ID: <4AB750E4.70300@apache.org> Date: Mon, 21 Sep 2009 11:09:40 +0100 From: Steve Loughran User-Agent: Thunderbird 2.0.0.23 (X11/20090812) MIME-Version: 1.0 To: general@hadoop.apache.org Subject: Re: hadoop hangs on reduce References: <904767.92694.qm@web38404.mail.mud.yahoo.com> <232426.10791.qm@web38405.mail.mud.yahoo.com> In-Reply-To: <232426.10791.qm@web38405.mail.mud.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-HPL-MailScanner-Information: Please contact the ISP for more information X-MailScanner-ID: n8LA9eHH008689 X-HPL-MailScanner: Found to be clean X-HPL-MailScanner-From: stevel@apache.org X-Virus-Checked: Checked by ClamAV on apache.org himanshu chandola wrote: > Just as an update. I made a dummy map job so that the map outputs a unique key for every input and hence the input to reduce is unique too. Still my reduce jobs hang at 76.02 % now (I've added a few nodes into my cluster so I suspect what was earlier 96.49 is 76.02). So this is definitely not a memory or io issue. > > Do I restart my task trackers ? (ive tried once but didnt help) > I see reduce hangs when the TT's cant talk to each other, when they can't get data from the other TTs check the value of mapred.task.tracker.report.address , that it is on an external address (not 127.0.0.1) and that the port in use is open on all the machines. -steve