Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F217B10CFF for ; Tue, 19 Nov 2013 04:08:41 +0000 (UTC) Received: (qmail 87484 invoked by uid 500); 19 Nov 2013 04:08:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 87430 invoked by uid 500); 19 Nov 2013 04:08:32 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 87417 invoked by uid 99); 19 Nov 2013 04:08:31 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Nov 2013 04:08:31 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id A19051D3B8C; Tue, 19 Nov 2013 04:08:27 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============0721175115875372200==" MIME-Version: 1.0 Subject: Re: Review Request 15663: Hive should be able to skip header and footer rows when reading data file for a table From: "Thejas Nair" To: "Thejas Nair" Cc: "Eric Hanson" , "hive" Date: Tue, 19 Nov 2013 04:08:27 -0000 Message-ID: <20131119040827.5849.90086@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Thejas Nair" X-ReviewGroup: hive X-ReviewRequest-URL: https://reviews.apache.org/r/15663/ X-Sender: "Thejas Nair" References: <20131119013138.5850.91801@reviews.apache.org> In-Reply-To: <20131119013138.5850.91801@reviews.apache.org> Reply-To: "Thejas Nair" X-ReviewRequest-Repository: hive-git --===============0721175115875372200== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15663/#review29097 ----------------------------------------------------------- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java config params also need to be added to hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java this code should be refactored to be re-used in both places. ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java I think it is better to fail the query if it is not a number. Otherwise, user will silently get data different from what is expected. ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java need to update comment. same comment regarding exception handling as for header. ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java should this conf param go into HiveConf ? - Thejas Nair On Nov. 19, 2013, 1:31 a.m., Eric Hanson wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15663/ > ----------------------------------------------------------- > > (Updated Nov. 19, 2013, 1:31 a.m.) > > > Review request for hive and Thejas Nair. > > > Bugs: HIVE-5795 > https://issues.apache.org/jira/browse/HIVE-5795 > > > Repository: hive-git > > > Description > ------- > > Hive should be able to skip header and footer rows when reading data file for a table > > (I am uploading this on behalf of Shuaishuai Nie since he's not in the office) > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 32ab3d8 > data/files/header_footer_table_1/0001.txt PRE-CREATION > data/files/header_footer_table_1/0002.txt PRE-CREATION > data/files/header_footer_table_1/0003.txt PRE-CREATION > data/files/header_footer_table_2/2012/01/01/0001.txt PRE-CREATION > data/files/header_footer_table_2/2012/01/02/0002.txt PRE-CREATION > data/files/header_footer_table_2/2012/01/03/0003.txt PRE-CREATION > itests/qtest/pom.xml a453d8a > ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 5abcfc1 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveContextAwareRecordReader.java dd5cb6b > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 0ec6e63 > ql/src/test/org/apache/hadoop/hive/ql/io/TestHiveBinarySearchRecordReader.java 85dd975 > ql/src/test/org/apache/hadoop/hive/ql/io/TestSymlinkTextInputFormat.java 0686d9b > ql/src/test/queries/clientpositive/file_with_header_footer.q PRE-CREATION > ql/src/test/results/clientpositive/file_with_header_footer.q.out PRE-CREATION > > Diff: https://reviews.apache.org/r/15663/diff/ > > > Testing > ------- > > > Thanks, > > Eric Hanson > > --===============0721175115875372200==--