Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 144B1D0BE for ; Tue, 14 May 2013 06:30:28 +0000 (UTC) Received: (qmail 78030 invoked by uid 500); 14 May 2013 06:30:22 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 77640 invoked by uid 500); 14 May 2013 06:30:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77626 invoked by uid 99); 14 May 2013 06:30:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 May 2013 06:30:19 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=FORGED_YAHOO_RCVD,FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [72.30.238.74] (HELO nm35-vm2.bullet.mail.bf1.yahoo.com) (72.30.238.74) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 May 2013 06:30:11 +0000 Received: from [98.139.215.143] by nm35.bullet.mail.bf1.yahoo.com with NNFMP; 14 May 2013 06:29:49 -0000 Received: from [98.139.211.200] by tm14.bullet.mail.bf1.yahoo.com with NNFMP; 14 May 2013 06:29:49 -0000 Received: from [127.0.0.1] by smtp209.mail.bf1.yahoo.com with NNFMP; 14 May 2013 06:29:49 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1368512989; bh=5pfxD9glID9o3ekjn6Ble7MFpVckcOTKo1WCp3EAjck=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:From:To:References:In-Reply-To:Subject:Date:Message-ID:MIME-Version:Content-Type:X-Mailer:Thread-Index:Content-Language; b=5SQHf9FypvpZp6OpRZnany1S/C1fyJ/dSREQ7C5ruwty7DcWOM++MgncF07hnxgIB6A14QwDW5xvnzOdD+D9Ft79ws+D5VUgigi9/hN2F3aFnckyf4q19BNx/o4BYYtMi6BJOfqATe779A8JzcxgLvjOZWEK3KQS+xmMgQptBzc= X-Yahoo-Newman-Id: 235007.59192.bm@smtp209.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: rO1DLkoVM1lZ_EIu._hyS4tjJz0w7qQGJUPHBer2StJVbYC JcuEbd4j39GZgyeh87MUj9X0pawMjPEilJYnoyRJpoM7uW83VL5ibyIehCva s6DkiWm6Mr5R8lB6s7wTc5mFUNAy6CuLZuGZXE9mrFp065oRMSqidDfL2LpR LlHycnZw.JO2Y9PX8BGZJR.o7lbFiwc4i7ArVhb.JK9DRemhm1fJWAlX4c.E m6flL1h2N0ipBGQyhuyxqucYUprr68PiI8zklqmGEmWsXHVllYQUk4XxIqgc oVdK3c84v7ZNcOtVioYmNPHUclKovKN.A1jCmuOH8NF2lj_bpshn7lq2V8jy 5YInhM1Fzp7DgsB0O.Y7_gpInNPEUp5qwJnECIFP4VqmPh8WNNDQ2hAzmhqu Y4MvL9PCWQFcUGoynOBWEdOXljEABvlyN6ty.C7WzJ51EJ92L5vSEerNGH0t wiZkNH2THCT0sHQz91PsfiJrHfpt9cRLUHZAWTQJK7SlkWq_HWvU79AUaly2 lGbiwV1GzsMCjGmhdged3ml0h3BtypTud3S.guoUApIEr.N_BjeNjQHp9qHu aB.8eX1LLtYHveNbxKwBIjM514XtBYesiyuUhgznz1_NDAY3tQd6sqGK9wNP 7AmSHIC4jrqZ6xyF64YmDlENvGOS1khbDCcDEFRBMXvIR X-Yahoo-SMTP: k2gD1GeswBAV_JFpZm8dmpTCwr4ufTKOyA-- X-Rocket-Received: from sattelite (davidparks21@113.161.75.108 with ) by smtp209.mail.bf1.yahoo.com with SMTP; 13 May 2013 23:29:49 -0700 PDT From: "David Parks" To: References: <085a01ce506b$0455fcf0$0d01f6d0$@yahoo.com> In-Reply-To: <085a01ce506b$0455fcf0$0d01f6d0$@yahoo.com> Subject: RE: JobClient: Error reading task output - after instituting a DNS server Date: Tue, 14 May 2013 13:29:40 +0700 Message-ID: <086e01ce506c$6c92e7a0$45b8b6e0$@yahoo.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_086F_01CE50A7.18F457B0" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQGNm7tQBZr2PcbLxYHIRD0AgLlBwpmFoQ0Q Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_086F_01CE50A7.18F457B0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I just hate it when I figure out a problem right after asking for help. Finding the task logs via the task tracker website identified the problem which didn't show up elsewhere. Simple mis-configuration which I did concurrently with the DNS update that threw me off track. Dave From: David Parks [mailto:davidparks21@yahoo.com] Sent: Tuesday, May 14, 2013 1:20 PM To: user@hadoop.apache.org Subject: JobClient: Error reading task output - after instituting a DNS server So we just configured a local DNS server for hostname resolution and stopped using a hosts file and now jobs fail on us. But I can't figure out why. You can see the error below, but if I run curl to any of those URLs they come back "Failed to retrieve stdout log", which doesn't look much like a DNS issue. I can ping and do nslookup from any host to any other host. This is a CDH4 cluster and the host inspector is happy as could be; also Cloudera Manager indicates all is well. When I open the task tracker website I see the first task attempt show up on the site there for maybe 10 seconds or so before it fails. Any idea what I need to look at here? Job: ==== 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process : 131 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process : 1 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0% 13/05/14 05:13:47 INFO mapred.JobClient: Task Id : attempt_201305131758_0003_m_000353_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237) 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_m_000353_0&filter=stdout 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_m_000353_0&filter=stderr 13/05/14 05:13:50 INFO mapred.JobClient: Task Id : attempt_201305131758_0003_r_000521_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237) 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_r_000521_0&filter=stdout 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_r_000521_0&filter=stderr curl of above URL: ==================== davidparks21@hadoop-meta1:~$ curl 'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013 05131758_0003_m_000353_0&filter=stdout' Error 410 Failed to retrieve stdout log for task: attempt_201305131758_0003_m_000353_0

HTTP ERROR 410

Problem accessing /tasklog. Reason: 

    Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0

Powered by Jetty://




------=_NextPart_000_086F_01CE50A7.18F457B0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I just hate it when I figure out a problem right = after asking for help.

 

Finding the task logs = via the task tracker website identified the problem which didn’t = show up elsewhere. Simple mis-configuration which I did concurrently = with the DNS update that threw me off track.

 

Dave

 

 

From:= = David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, = May 14, 2013 1:20 PM
To: = user@hadoop.apache.org
Subject: JobClient: Error reading task = output - after instituting a DNS = server

 

So we just = configured a local DNS server for hostname resolution and stopped using = a hosts file and now jobs fail on us. But I can’t figure out = why.

 

You can see the error below, but if I run curl to any = of those URLs they come back “Failed to retrieve stdout = log”, which doesn’t look much like a DNS = issue.

 

I can ping and do nslookup from any host to any other = host. This is a CDH4 cluster and the host inspector is happy as could = be; also Cloudera Manager indicates all is well.

 

When I open = the task tracker website I see the first task attempt show up on the = site there for maybe 10 seconds or so before it fails.

 

Any idea = what I need to look at here?

 

Job:

=3D=3D=3D=3D

13/05/14 05:13:40 INFO = input.FileInputFormat: Total input paths to process : = 131

13/05/14 05:13:41 INFO = input.FileInputFormat: Total input paths to process : = 1

13/05/14 05:13:42 INFO = mapred.JobClient: Running job: = job_201305131758_0003

13/05/14 05:13:43 INFO = mapred.JobClient:  map 0% reduce 0%

13/05/14 = 05:13:47 INFO mapred.JobClient: Task Id : = attempt_201305131758_0003_m_000353_0, Status : = FAILED

java.lang.Throwable: Child = Error

        at = org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)

Caused by: java.io.IOException: Task process exit with nonzero = status of 1.

        at = org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

 

13/05/14 05:13:47 WARN = mapred.JobClient: Error reading task = outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=3Dtrue&attempti= d=3Dattempt_201305131758_0003_m_000353_0&filter=3Dstdout

13/05/14 05:13:47 WARN mapred.JobClient: Error reading task = outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=3Dtrue&attempti= d=3Dattempt_201305131758_0003_m_000353_0&filter=3Dstderr

13/05/14 05:13:50 INFO mapred.JobClient: Task Id : = attempt_201305131758_0003_r_000521_0, Status : = FAILED

java.lang.Throwable: Child = Error

        at = org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)

Caused by: java.io.IOException: Task process exit with nonzero = status of 1.

        at = org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

 

13/05/14 05:13:50 WARN = mapred.JobClient: Error reading task = outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=3Dtrue&attempti= d=3Dattempt_201305131758_0003_r_000521_0&filter=3Dstdout

13/05/14 05:13:50 WARN mapred.JobClient: Error reading task = outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=3Dtrue&attempti= d=3Dattempt_201305131758_0003_r_000521_0&filter=3Dstderr

 

 

curl of = above URL:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

davidparks21@hadoop-meta1:~$ curl = 'http://hadoop-fullslot2:50060/tasklog?plaintext=3Dtrue&attemptid=3Da= ttempt_201305131758_0003_m_000353_0&filter=3Dstdout'

<html>

<head>

<meta = http-equiv=3D"Content-Type" content=3D"text/html; = charset=3DISO-8859-1"/>

<title>Error 410 Failed to retrieve stdout log for task: = attempt_201305131758_0003_m_000353_0</title>

<= p class=3DMsoNormal></head>

<body><h2>HTTP ERROR = 410</h2>

<p>Problem accessing /tasklog. = Reason:

<pre>    Failed = to retrieve stdout log for task: = attempt_201305131758_0003_m_000353_0</pre></p><hr = /><i><small>Powered by = Jetty://</small></i><br/>     =             &= nbsp;           &n= bsp;  

<br/>

<br/>

<br/>

<br/>

------=_NextPart_000_086F_01CE50A7.18F457B0--