From issues-return-125140-archive-asf-public=cust-asf.ponee.io@hive.apache.org Fri Jun 22 12:09:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4F980180647 for ; Fri, 22 Jun 2018 12:09:05 +0200 (CEST) Received: (qmail 54889 invoked by uid 500); 22 Jun 2018 10:09:04 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 54878 invoked by uid 99); 22 Jun 2018 10:09:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jun 2018 10:09:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EBCACC04C5 for ; Fri, 22 Jun 2018 10:09:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -107 X-Spam-Level: X-Spam-Status: No, score=-107 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, KAM_BADIPHTTP=2, KAM_NUMSUBJECT=0.5, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 0XaxP4eyBLBv for ; Fri, 22 Jun 2018 10:09:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 029525F288 for ; Fri, 22 Jun 2018 10:09:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 08231E0CCC for ; Fri, 22 Jun 2018 10:09:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5D7BC23F9E for ; Fri, 22 Jun 2018 10:09:00 +0000 (UTC) Date: Fri, 22 Jun 2018 10:09:00 +0000 (UTC) From: "Hive QA (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-17852?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1652= 0207#comment-16520207 ]=20 Hive QA commented on HIVE-17852: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12928606/HIVE-17852.11.pat= ch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/11992/test= Report Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11992/co= nsole Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11992/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with e= xit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-06-22 10:07:38.262 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=3D/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=3D/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=3D/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/= bin:/bin:/usr/local/games:/usr/games + PATH=3D/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bi= n:/usr/local/games:/usr/games + export 'ANT_OPTS=3D-Xmx1g -XX:MaxPermSize=3D256m ' + ANT_OPTS=3D'-Xmx1g -XX:MaxPermSize=3D256m ' + export 'MAVEN_OPTS=3D-Xmx1g ' + MAVEN_OPTS=3D'-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-11992/source-prep.txt + [[ false =3D=3D \t\r\u\e ]] + mkdir -p maven ivy + [[ git =3D \s\v\n ]] + [[ git =3D \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-06-22 10:07:38.265 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at e36f6e4 HIVE-19783: Retrieve only locations in HiveMetaStore= .dropPartitionsAndGetLocations (Peter Vary, reviewed by Alexander Kolbasov = and Vihang Karajgaonkar) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at e36f6e4 HIVE-19783: Retrieve only locations in HiveMetaStore= .dropPartitionsAndGetLocations (Peter Vary, reviewed by Alexander Kolbasov = and Vihang Karajgaonkar) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-06-22 10:07:39.579 + rm -rf ../yetus_PreCommit-HIVE-Build-11992 + mkdir ../yetus_PreCommit-HIVE-Build-11992 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-11992 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11992/yetus + patchCommandPath=3D/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=3D/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/work= ing/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.ja= va:2310 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'= with conflicts. Going to apply patch with: git apply -p0 /data/hiveptest/working/scratch/build.patch:3806: trailing whitespace. =09numFiles =091 =20 /data/hiveptest/working/scratch/build.patch:3866: trailing whitespace. =09numFiles =091 =20 /data/hiveptest/working/scratch/build.patch:3870: trailing whitespace. =09totalSize =095293 =20 /data/hiveptest/working/scratch/build.patch:3909: trailing whitespace. =09numFiles =091 =20 /data/hiveptest/working/scratch/build.patch:3930: trailing whitespace. =09numFiles =091 =20 error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.ja= va:2310 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'= with conflicts. U ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java warning: squelched 46 whitespace errors warning: 51 lines add whitespace errors. + result=3D1 + '[' 1 -ne 0 ']' + rm -rf yetus_PreCommit-HIVE-Build-11992 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12928606 - PreCommit-HIVE-Build > remove support for list bucketing "stored as directories" in 3.0 > ---------------------------------------------------------------- > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug > Reporter: Sergey Shelukhin > Assignee: Laszlo Bodor > Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, HIVE-17852= .03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch, HIVE-17852.06.patch, H= IVE-17852.07.patch, HIVE-17852.08.patch, HIVE-17852.09.patch, HIVE-17852.10= .patch, HIVE-17852.11.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to = Hive tables that has to be accounted for in many places in the code where t= he files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is b= roken - e.g. the duplicate file removal from tasks, a pretty fundamental co= rrectness feature in Hive, may be broken. LB also doesn=E2=80=99t appear to= be compatible with e.g. regular bucketing. > 3) The feature hasn=E2=80=99t seen development activity in a while; it al= so doesn=E2=80=99t appear to be used a lot. > Keeping with the theme of cleaning up =E2=80=9Clegacy=E2=80=9D code for 3= .0, I was proposing we remove it. > (2) also suggested that, if needed, it might be easier to implement simil= ar functionality by adding some flexibility to partitions (which LB directo= ries look like anyway); that would also keep the logic on a higher level of= abstraction (split generation, partition pruning) as opposed to many low-l= evel places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)