Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6B34C17C36 for ; Sun, 5 Oct 2014 12:57:35 +0000 (UTC) Received: (qmail 29470 invoked by uid 500); 5 Oct 2014 12:57:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 29401 invoked by uid 500); 5 Oct 2014 12:57:34 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 29390 invoked by uid 500); 5 Oct 2014 12:57:34 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 29387 invoked by uid 99); 5 Oct 2014 12:57:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Oct 2014 12:57:34 +0000 Date: Sun, 5 Oct 2014 12:57:34 +0000 (UTC) From: "Hive QA (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-8137) Empty ORC file handling MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159516#comment-14159516 ] Hive QA commented on HIVE-8137: ------------------------------- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12672959/HIVE-8137.patch {color:red}ERROR:{color} -1 due to 78 failed/errored test(s), 6511 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_correctness org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_union1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input23 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_filters org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_hive_626 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonblock_op_deduplicate org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_within_subquery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt9 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_exists_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notexists_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_join1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union14 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union15 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union17 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union18 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union19 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union22 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union31 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union32 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union33 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union34 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_varchar_union1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_cast org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_external_table_with_space_in_location_path org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_file_with_header_footer org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1123/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1123/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1123/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 78 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12672959 > Empty ORC file handling > ----------------------- > > Key: HIVE-8137 > URL: https://issues.apache.org/jira/browse/HIVE-8137 > Project: Hive > Issue Type: Improvement > Components: File Formats > Affects Versions: 0.13.1 > Reporter: Pankit Thapar > Fix For: 0.14.0 > > Attachments: HIVE-8137.patch > > > Hive 13 does not handle reading of a zero size Orc File properly. An Orc file is suposed to have a post-script > which the ReaderIml class tries to read and initialize the footer with it. But in case, the file is empty > or is of zero size, then it runs into an IndexOutOfBound Exception because of ReaderImpl trying to read in its constructor. > Code Snippet : > //get length of PostScript > int psLen = buffer.get(readSize - 1) & 0xff; > In the above code, readSize for an empty file is zero. > I see that ensureOrcFooter() method performs some sanity checks for footer , > so, either we can move the above code snippet to ensureOrcFooter() and throw a "Malformed ORC file exception" or we can create a dummy Reader that does not initialize footer and basically has hasNext() set to false so that it returns false on the first call. > Basically, I would like to know what might be the correct way to handle an empty ORC file in a mapred job? > Should we neglect it and not throw an exception or we can throw an exeption that the ORC file is malformed. > Please let me know your thoughts on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)