Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9A644F15F for ; Mon, 22 Apr 2013 22:15:40 +0000 (UTC) Received: (qmail 82576 invoked by uid 500); 22 Apr 2013 22:15:35 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 82500 invoked by uid 500); 22 Apr 2013 22:15:35 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 82492 invoked by uid 99); 22 Apr 2013 22:15:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Apr 2013 22:15:35 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=URIBL_DBL_REDIR X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [108.166.43.97] (HELO smtp97.ord1c.emailsrvr.com) (108.166.43.97) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Apr 2013 22:15:30 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp5.relay.ord1c.emailsrvr.com (SMTP Server) with ESMTP id 916DB1B00AC for ; Mon, 22 Apr 2013 18:14:49 -0400 (EDT) X-Virus-Scanned: OK Received: by smtp5.relay.ord1c.emailsrvr.com (Authenticated sender: kaveh-AT-plutoz.com) with ESMTPSA id 3CFA71B00BF for ; Mon, 22 Apr 2013 18:14:48 -0400 (EDT) Message-ID: <5175B657.600@plutoz.com> Date: Mon, 22 Apr 2013 15:14:47 -0700 From: kaveh minooie User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130402 Thunderbird/17.0.5 MIME-Version: 1.0 To: user@hadoop.apache.org Subject: Re: common error in map tasks References: <5174F5FD.5080702@plutoz.com> <1366621933912.b5661eda@Nodemailer> <5175A717.50106@plutoz.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org thanks Chris. I only run nutch, so no to the external command. And I just checked and it happens or has happened on all the nodes at some point. I have to say thou that it doesn't cause the job to fail or anything. the map tasks that fail will finish when they are re-spawn again. it is just annoying and makes me think that some value some where in the config files are either not correct or not optimal. On 04/22/2013 02:49 PM, Chris Nauroth wrote: > I'm not aware of any Hadoop-specific meaning for exit code 126. > Typically, this is a standard Unix exit code used to indicate that a > command couldn't be executed. Some reasons for this might be that the > command is not an executable file, or the command is an executable file > but the user doesn't have execute permissions. (See below for an > example of each of these.) > > Does your job code attempt to exec an external command? Also, are the > task failures consistently happening on the same set of nodes in your > cluster? If so, then I recommend checking that the command has been > deployed and has the correct permissions on those nodes. > > Even if your code doesn't exec an external command, various parts of the > Hadoop code do this internally, so you still might have a case of a > misconfigured node. > > Hope this helps, > --Chris > > [chris@Chriss-MacBook-Pro:ttys000] hadoop-common > > ./BUILDING.txt > -bash: ./BUILDING.txt: Permission denied > [chris@Chriss-MacBook-Pro:ttys000] hadoop-common > > echo $? > 126 > > [chris@Chriss-MacBook-Pro:ttys000] test > > ls -lrt exec > -rwx------ 1 root staff 0B Apr 22 14:43 exec* > [chris@Chriss-MacBook-Pro:ttys000] test > > whoami > chris > [chris@Chriss-MacBook-Pro:ttys000] test > > ./exec > bash: ./exec: Permission denied > [chris@Chriss-MacBook-Pro:ttys000] test > > echo $? > 126 > > > > On Mon, Apr 22, 2013 at 2:09 PM, kaveh minooie > wrote: > > thanks. that is the issue, there is no other log files. when i go to > the attempt directory of that failed map task (e.g. > userlogs/job_201304191712___0015/attempt_201304191712___0015_m_000019_0 > ) it is empty. there is no other log file. thou based on the counter > value, I can say that it happens right at the beginning of the map > task (counter is only 1 ) > > > > > On 04/22/2013 02:12 AM, 姚吉龙 wrote: > > Hi > > > I have the same problem before > I think this is caused by the lack of memory shortage for map task. > It is just a suggestion,you can post your log > > > BRs > Geelong > — > Sent from Mailbox for iPhone > > > > On Mon, Apr 22, 2013 at 4:34 PM, kaveh minooie > >> wrote: > > HI > > regardless of what job I run, there are always a few map > tasks that > fail with the following, very unhelpful, message: ( that is the > entire error message) > > java.lang.Throwable: Child Error > at > org.apache.hadoop.mapred.__TaskRunner.run(TaskRunner.__java:271) > Caused by: java.io.IOException: Task process exit with > nonzero status of 126. > at > org.apache.hadoop.mapred.__TaskRunner.run(TaskRunner.__java:258) > > > I would appreciate it if someone could show me how I could > figure > out why this error keeps happening. > > thanks, > > > > -- > Kaveh Minooie > > -- Kaveh Minooie