Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 62194200BEF for ; Wed, 21 Dec 2016 06:57:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 60B10160B3C; Wed, 21 Dec 2016 05:57:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AA504160B29 for ; Wed, 21 Dec 2016 06:56:59 +0100 (CET) Received: (qmail 65230 invoked by uid 500); 21 Dec 2016 05:56:58 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 65205 invoked by uid 99); 21 Dec 2016 05:56:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Dec 2016 05:56:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 9CEBB2C0059 for ; Wed, 21 Dec 2016 05:56:58 +0000 (UTC) Date: Wed, 21 Dec 2016 05:56:58 +0000 (UTC) From: "Hive QA (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-14165) Remove Hive file listing during split computation MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 21 Dec 2016 05:57:00 -0000 [ https://issues.apache.org/jira/browse/HIVE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766220#comment-15766220 ] Hive QA commented on HIVE-14165: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12844177/HIVE-14165.05.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10825 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=234) TestVectorizedColumnReaderBase - did not produce a TEST-*.xml file (likely timed out) (batchId=251) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=71) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[exim_04_evolved_parts] (batchId=29) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[str_to_map] (batchId=58) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=135) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision] (batchId=151) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=93) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exchange_partition_neg_incomplete_partition] (batchId=84) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_00_unsupported_schema] (batchId=85) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query36] (batchId=222) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query70] (batchId=222) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query86] (batchId=222) org.apache.hive.hcatalog.pig.TestHCatLoader.testReadMissingPartitionBasicNeg[3] (batchId=171) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2667/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2667/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2667/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12844177 - PreCommit-HIVE-Build > Remove Hive file listing during split computation > ------------------------------------------------- > > Key: HIVE-14165 > URL: https://issues.apache.org/jira/browse/HIVE-14165 > Project: Hive > Issue Type: Sub-task > Affects Versions: 2.1.0 > Reporter: Abdullah Yousufi > Assignee: Sahil Takiar > Attachments: HIVE-14165.02.patch, HIVE-14165.03.patch, HIVE-14165.04.patch, HIVE-14165.05.patch, HIVE-14165.patch > > > The Hive side listing in FetchOperator.java is unnecessary, since Hadoop's FileInputFormat.java will list the files during split computation anyway to determine their size. One way to remove this is to catch the InvalidInputFormat exception thrown by FileInputFormat#getSplits() on the Hive side instead of doing the file listing beforehand. > For S3 select queries on partitioned tables, this results in a 2x speedup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)