From issues-return-199362-archive-asf-public=cust-asf.ponee.io@hive.apache.org Tue Sep 22 11:35:02 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 9B0CD18063F for ; Tue, 22 Sep 2020 13:35:02 +0200 (CEST) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id C5141121D01 for ; Tue, 22 Sep 2020 11:35:01 +0000 (UTC) Received: (qmail 70388 invoked by uid 500); 22 Sep 2020 11:35:01 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 70372 invoked by uid 99); 22 Sep 2020 11:35:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Sep 2020 11:35:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B3A6F407EF for ; Tue, 22 Sep 2020 11:35:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 33866780203 for ; Tue, 22 Sep 2020 11:35:00 +0000 (UTC) Date: Tue, 22 Sep 2020 11:35:00 +0000 (UTC) From: "Marta Kuczora (Jira)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-24163) Dynamic Partitioning Insert for MM table fails during MoveTask MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-24163: --------------------------------- Summary: Dynamic Partitioning Insert for MM table fails during MoveTask (was: Dynamic Partitioning Insert fail for MM table fail during MoveTask) > Dynamic Partitioning Insert for MM table fails during MoveTask > -------------------------------------------------------------- > > Key: HIVE-24163 > URL: https://issues.apache.org/jira/browse/HIVE-24163 > Project: Hive > Issue Type: Bug > Components: Hive > Reporter: Rajkumar Singh > Assignee: Marta Kuczora > Priority: Major > Labels: pull-request-available > Fix For: 3.1.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > -- DDLs and Query > {code:java} > create table `class` (name varchar(8), sex varchar(1), age double precision, height double precision, weight double precision); > insert into table class values ('RAJ','MALE',28,12,12); > CREATE TABLE `PART1` (`id` DOUBLE,`N` DOUBLE,`Name` VARCHAR(8),`Sex` VARCHAR(1)) PARTITIONED BY(Weight string, Age > string, Height string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE; > INSERT INTO TABLE `part1` PARTITION (`Weight`,`Age`,`Height`) SELECT 0, 0, `Name`,`Sex`,`Weight`,`Age`,`Height` FROM `class`; > {code} > it fail during the MoveTask execution: > {code:java} > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: partition hdfs://hostname:8020/warehouse/tablespace/managed/hive/part1/.hive-staging_hive_2020-09-02_13-29-58_765_4475282758764123921-1/-ext-10000/tmpstats-0_FS_3 is not a directory! > at org.apache.hadoop.hive.ql.metadata.Hive.getValidPartitionsInPath(Hive.java:2769) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:2837) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.exec.MoveTask.handleDynParts(MoveTask.java:562) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:440) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) ~[hive-exec-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225) ~[hive-service-3.1.3000.7.2.0.0-237.jar:3.1.3000.7.2.0.0-237] > {code} > The reason is Task write the fsstat during the FileSinkOperator closing, HS2 ran the MoveTask to move data into the destination partition directory, while getting the partition location hive check whether destination is directory or not and failing. > -- hive set the stat location during > https://github.com/apache/hive/blob/d700ea54ec5da5364d92a9faaa58f89ea03181e0/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L8135 > which is relative to the hive-staging directory: > https://github.com/apache/hive/blob/fecad5b0f72c535ed1c53f2cc62b0d6649b651ae/ql/src/java/org/apache/hadoop/hive/ql/Context.java#L617 -- This message was sent by Atlassian Jira (v8.3.4#803005)