Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D394AD587 for ; Tue, 28 Aug 2012 16:28:28 +0000 (UTC) Received: (qmail 54144 invoked by uid 500); 28 Aug 2012 16:28:27 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 54106 invoked by uid 500); 28 Aug 2012 16:28:27 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 54096 invoked by uid 99); 28 Aug 2012 16:28:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Aug 2012 16:28:27 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jamalraihan@gmail.com designates 209.85.212.48 as permitted sender) Received: from [209.85.212.48] (HELO mail-vb0-f48.google.com) (209.85.212.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Aug 2012 16:28:21 +0000 Received: by vbme21 with SMTP id e21so5628696vbm.35 for ; Tue, 28 Aug 2012 09:28:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=pKlXQWhhKxeRwT1U934R6ILBKXGq631cnJ2wCSDdIpQ=; b=c53vFKeEW3cz4GRHXcuG9oh1TvbA4p7PhVxvz0Vn9/L6vpFI+8RDVc3VIvMLuYKOjS 1rsbvLutAXCNH/npacJ0LBV8ue18b6S0pZLp2a/Sk1X3j/enVLZPlbA8aDi63h8kIPb/ Ol9TIbjoLFKOh/S2jF/tiz6Qpeom6EVDAGqJPNB2lgR6KTB3FVDhS8RzKRVxILi87iIl 6OuCEi0Wd8HVlzZnO2ormkW82oaWeiPq7dxVUlXqUas9tr8Bb+jxNsWhcbxBD73JtX4C Hy/pIzcJcgWb/53ZhxSdxD3BYZDhpzrk4keBPQzl3XsgBPP5dHos1PjoanppNULO1LUq gY3A== Received: by 10.58.65.10 with SMTP id t10mr16296312ves.48.1346171280119; Tue, 28 Aug 2012 09:28:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.210.1 with HTTP; Tue, 28 Aug 2012 09:27:39 -0700 (PDT) In-Reply-To: References: From: Raihan Jamal Date: Tue, 28 Aug 2012 09:27:39 -0700 Message-ID: Subject: Re: Unexpected end of input stream To: user@hive.apache.org Content-Type: multipart/alternative; boundary=047d7b6d7e00f5577f04c855ecbd X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d7e00f5577f04c855ecbd Content-Type: text/plain; charset=ISO-8859-1 That basically means your data was not in the correct format when you move or copied the data to HDFS. So there is one file which is corrupted, you can find the file name in your error logs. *Raihan Jamal* On Tue, Aug 28, 2012 at 7:23 AM, Kiwon Lee wrote: > Hi > > I have a lot of compressed gzip files on hdfs. > An exception has occurred at TaskTracker, during processing of MR. > If any file is invalid, may I know that? > > > 2012-08-28 09:17:56,320 INFO ExecMapper: ExecMapper: processed 0 rows: > used memory = 125190136 > 2012-08-28 09:17:56,324 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > 2012-08-28 09:17:56,326 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:ubuntu (auth:SIMPLE) cause:java.io.IOException: java.io.EOFException: > Unexpected end of input stream > 2012-08-28 09:17:56,326 WARN org.apache.hadoop.mapred.Child: Error running > child > java.io.IOException: java.io.EOFException: Unexpected end of input stream > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) > at > org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:275) > at > org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79) > at > org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: java.io.EOFException: Unexpected end of input stream > at > org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:83) > at java.io.InputStream.read(InputStream.java:82) > at > org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:209) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:173) > at > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:160) > at > org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:38) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:273) > ... 13 more > > > -- > > *Best Regards.** Ethan (Kiwon Lee)* > kiwoni.lee@gmail.com > > > --047d7b6d7e00f5577f04c855ecbd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
That basically means your data was not in the correct form= at when you move or copied the data to HDFS. So there is one file which is = corrupted, you can find the file name in your error logs.
<= br>

Raihan Jamal



On Tue, Aug 28, 2012 at 7:23 AM, Kiwon L= ee <kiwoni.lee@gmail.com> wrote:
Hi

I have a lot of compressed gzip files= on hdfs.=A0
An exception has occurred at TaskTracker, during pro= cessing of MR.
If any file is invalid, may I know that?


2012-08-28 09:17:56,320 INFO ExecMa= pper: ExecMapper: processed 0 rows: used memory =3D 125190136
201= 2-08-28 09:17:56,324 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initi= alizing logs' truncater with mapRetainSize=3D-1 and reduceRetainSize=3D= -1
2012-08-28 09:17:56,326 ERROR org.apache.hadoop.security.UserGroupInfo= rmation: PriviledgedActionException as:ubuntu (auth:SIMPLE) cause:java.io.I= OException: java.io.EOFException: Unexpected end of input stream
2012-08-28 09:17:56,326 WARN org.apache.hadoop.mapred.Child: Error run= ning child
java.io.IOException: java.io.EOFException: Unexpected = end of input stream
=A0 =A0 =A0 =A0 at org.apache.hadoop.hive.io.= HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOException= HandlerChain.java:121)
=A0 =A0 =A0 =A0 at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUti= l.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
=
=A0 =A0 =A0 =A0 at org.apache.hadoop.hive.ql.io.HiveContextAwareRecord= Reader.doNext(HiveContextAwareRecordReader.java:275)
=A0 =A0 =A0 =A0 at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNex= t(HiveRecordReader.java:79)
=A0 =A0 =A0 =A0 at org.apache.hadoop.= hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
=A0 = =A0 =A0 =A0 at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.ne= xt(HiveContextAwareRecordReader.java:108)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask$TrackedRecordReade= r.moveToNext(MapTask.java:210)
=A0 =A0 =A0 =A0 at org.apache.hado= op.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
=A0 = =A0 =A0 =A0 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTa= sk.java:393)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.MapTask.= run(MapTask.java:327)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred= .Child$4.run(Child.java:270)
=A0 =A0 =A0 =A0 at java.security.AccessController.doPrivileged(Native = Method)
=A0 =A0 =A0 =A0 at javax.security.auth.Subject.doAs(Subje= ct.java:396)
=A0 =A0 =A0 =A0 at org.apache.hadoop.security.UserGr= oupInformation.doAs(UserGroupInformation.java:1232)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.Child.main(Child.java:264)=
Caused by: java.io.EOFException: Unexpected end of input stream<= /div>
=A0 =A0 =A0 =A0 at org.apache.hadoop.io.compress.DecompressorStre= am.decompress(DecompressorStream.java:143)
=A0 =A0 =A0 =A0 at org.apache.hadoop.io.compress.DecompressorStream.re= ad(DecompressorStream.java:83)
=A0 =A0 =A0 =A0 at java.io.InputSt= ream.read(InputStream.java:82)
=A0 =A0 =A0 =A0 at org.apache.hado= op.util.LineReader.readDefaultLine(LineReader.java:209)
=A0 =A0 =A0 =A0 at org.apache.hadoop.util.LineReader.readLine(LineRead= er.java:173)
=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.LineReco= rdReader.next(LineRecordReader.java:160)
=A0 =A0 =A0 =A0 at org.a= pache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:38)
=A0 =A0 =A0 =A0 at org.apache.hadoop.hive.ql.io.HiveContextAwareRecord= Reader.doNext(HiveContextAwareRecordReader.java:273)
=A0 =A0 =A0 = =A0 ... 13 more

--

Best Regards.=A0Ethan (Kiwon = Lee)
=A0


--047d7b6d7e00f5577f04c855ecbd--