Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 31CD4200B84 for ; Tue, 20 Sep 2016 18:24:17 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3064B160AC5; Tue, 20 Sep 2016 16:24:17 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 74C89160AC0 for ; Tue, 20 Sep 2016 18:24:16 +0200 (CEST) Received: (qmail 42499 invoked by uid 500); 20 Sep 2016 16:24:15 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 42474 invoked by uid 99); 20 Sep 2016 16:24:15 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Sep 2016 16:24:15 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id DD4912711C6; Tue, 20 Sep 2016 16:24:14 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============7735328175483445306==" MIME-Version: 1.0 Subject: Re: Review Request 50525: HIVE-14341: Altered skewed location is not respected for list bucketing From: Aihua Xu To: hive , Aihua Xu Date: Tue, 20 Sep 2016 16:24:14 -0000 Message-ID: <20160920162414.1653.62585@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Aihua Xu X-ReviewGroup: hive X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/50525/ X-Sender: Aihua Xu References: <20160919210212.25295.80669@reviews.apache.org> In-Reply-To: <20160919210212.25295.80669@reviews.apache.org> Reply-To: Aihua Xu X-ReviewRequest-Repository: hive-git archived-at: Tue, 20 Sep 2016 16:24:17 -0000 --===============7735328175483445306== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50525/#review149671 ----------------------------------------------------------- ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java (line 234) Actually the original one is also not overwrite but move from a temp file to the final location. While that would fail if an previous file already exists. Then we should try to rename to '_copy_1' and so. ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java (line 899) Yes. That's where this issue comes from. Without this change, even if you alter the skew location which will be saved in locationMap, it would get the old location and save it to overwrite the newer location. That is not correct. The new logic is: if we have the mapped location for the skew keys, then use it. Otherwise, get the default skew location. - Aihua Xu On Sept. 19, 2016, 9:02 p.m., Aihua Xu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50525/ > ----------------------------------------------------------- > > (Updated Sept. 19, 2016, 9:02 p.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > ------- > > HIVE-14341: Altered skewed location is not respected for list bucketing > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java e386717 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java da46854 > ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java ba4f6a7 > ql/src/test/queries/clientpositive/create_alter_list_bucketing_table1.q bf89e8f > ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 216d3be > > Diff: https://reviews.apache.org/r/50525/diff/ > > > Testing > ------- > > > Thanks, > > Aihua Xu > > --===============7735328175483445306==--