hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "gavin_shi (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HIVE-20064) Data cannot be inserted into a dynamic partition
Date Tue, 03 Jul 2018 09:09:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

gavin_shi reassigned HIVE-20064:
--------------------------------

    Assignee:     (was: gavin_shi)

> Data cannot be inserted into a dynamic partition
> ------------------------------------------------
>
>                 Key: HIVE-20064
>                 URL: https://issues.apache.org/jira/browse/HIVE-20064
>             Project: Hive
>          Issue Type: Bug
>          Components: SQL
>            Reporter: gavin_shi
>            Priority: Blocker
>
> Data cannot be inserted into a dynamic partition,The relevant table information is as
follows:
> The source table:
> {code:java}
> //代码占位符
> create table customer(id string,name string,gender string,state string) partitioned by(country
string);
> {code}
> The source table data:
>  
> {code:java}
> //代码占位符
> +--------------+----------------+------------------+-----------------+-------------------+--+
> | customer.id | customer.name | customer.gender | customer.state | customer.country |
> +--------------+----------------+------------------+-----------------+-------------------+--+
> | 1 | gavin | 1 | shanghai | china |
> +--------------+----------------+------------------+-----------------+-------------------+--+
> {code}
>  
> The target table:
>  
> {code:java}
> //代码占位符
> create table customer_bak(id string,name string,gender string,state string) partitioned
by(country string);
> {code}
>  
> Now I use dynamic techniques to insert data into the customer_bak table:
> {code:java}
> //代码占位符
> set hive.exec.dynamic.partition=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> insert into table customer_bak partition(country) select id,name,gender,state,country
from customer;
> {code}
> The program is always executing and neither success nor failure is shown. The information
in the hive log is as follows:
> {code:java}
> //代码占位符
> 2018-06-28T16:01:22,900 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Executing
command(queryId=hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc): insert into table
customer_bak partition(country) select id,name,gender,state,country from customer
> 2018-06-28T16:01:22,901 WARN [HiveServer2-Background-Pool: Thread-97] ql.Driver: Hive-on-MR
is deprecated in Hive 2 and may not be available in the future versions. Consider using a
different execution engine (i.e. tez, spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: WARNING:
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider
using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Query
ID = hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc
> 2018-06-28T16:01:22,903 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Total
jobs = 3
> 2018-06-28T16:01:22,904 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Launching
Job 1 out of 3
> 2018-06-28T16:01:22,907 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting
task [Stage-1:MAPRED] in serial mode
> 2018-06-28T16:01:22,908 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Number
of reduce tasks is set to 0 since there's no reduce operator
> 2018-06-28T16:01:22,929 INFO [HiveServer2-Background-Pool: Thread-97] ql.Context: New
scratch dir is hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
> 2018-06-28T16:01:22,942 INFO [HiveServer2-Background-Pool: Thread-97] mr.ExecDriver:
Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
Processing alias customer
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
Adding 2 inputs; the first input is hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
Content Summary not cached for hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,945 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
Content Summary not cached for hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa
> 2018-06-28T16:01:22,978 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
Changed input file hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa to empty
file hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
(false)
> 2018-06-28T16:01:22,979 INFO [HiveServer2-Background-Pool: Thread-97] ql.Context: New
scratch dir is hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
> 2018-06-28T16:01:22,996 INFO [HiveServer2-Background-Pool: Thread-97] exec.SerializationUtilities:
Serializing MapWork using kryo
> 2018-06-28T16:01:23,055 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
Serialized plan (via FILE) - name: null size: 4.86KB
> 2018-06-28T16:01:29,318 INFO [HiveServer2-Background-Pool: Thread-97] fs.FSStatsPublisher:
created : hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10001
> 2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
PLAN PATH = hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
> 2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
PLAN PATH = hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
> 2018-06-28T16:01:29,398 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
No plan file found: hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
> 2018-06-28T16:01:29,406 INFO [HiveServer2-Background-Pool: Thread-97] client.ConfiguredRMFailoverProxyProvider:
Failing over to rm2
> 2018-06-28T16:01:29,413 WARN [HiveServer2-Background-Pool: Thread-97] mapreduce.JobSubmitter:
Hadoop command-line option parsing not performed. Implement the Tool interface and execute
your application with ToolRunner to remedy this.
> 2018-06-28T16:01:31,721 INFO [HiveServer2-Background-Pool: Thread-97] exec.Utilities:
PLAN PATH = hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
> 2018-06-28T16:01:31,722 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat:
Total number of paths: 2, launching 1 threads to check non-combinable ones.
> 2018-06-28T16:01:31,724 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat:
CombineHiveInputSplit creating pool for hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china;
using filter path hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:31,725 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat:
CombineHiveInputSplit: pool is already created for hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036;
using filter path hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
> 2018-06-28T16:01:31,738 INFO [HiveServer2-Background-Pool: Thread-97] input.FileInputFormat:
Total input paths to process : 2
> 2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97] input.CombineFileInputFormat:
DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
> 2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat:
number of splits 1
> 2018-06-28T16:01:31,743 INFO [HiveServer2-Background-Pool: Thread-97] io.CombineHiveInputFormat:
Number of all splits 1
> 2018-06-28T16:01:31,853 INFO [HiveServer2-Background-Pool: Thread-97] mapreduce.JobSubmitter:
number of splits:1
> 2018-06-28T16:01:31,982 INFO [HiveServer2-Background-Pool: Thread-97] mapreduce.JobSubmitter:
Submitting tokens for job: job_1530599349924_0007
> 2018-06-28T16:01:32,220 INFO [HiveServer2-Background-Pool: Thread-97] impl.YarnClientImpl:
Submitted application application_1530599349924_0007
> 2018-06-28T16:01:32,226 INFO [HiveServer2-Background-Pool: Thread-97] mapreduce.Job:
The url to track the job: http://cluster3:8088/proxy/application_1530599349924_0007/
> 2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Starting
Job = job_1530599349924_0007, Tracking URL = http://cluster3:8088/proxy/application_1530599349924_0007/
> 2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Kill
Command = /home/hadoop/app/hadoop_2.6/bin/hadoop job -kill job_1530599349924_0007
> 2018-06-28T16:01:56,835 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Hadoop
job information for Stage-1: number of mappers: 1; number of reducers: 0
> 2018-06-28T16:01:56,933 WARN [HiveServer2-Background-Pool: Thread-97] mapreduce.Counters:
Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter
instead
> 2018-06-28T16:01:56,934 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: 2018-06-28
16:01:56,933 Stage-1 map = 0%, reduce = 0%
> 2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30] conf.HiveConf: Using
the default value passed in for log id: 07fd131c-6590-4081-b7ed-2f0ae5b16551
> 2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30] session.SessionState:
Updating thread name to 07fd131c-6590-4081-b7ed-2f0ae5b16551 HiveServer2-Handler-Pool: Thread-30
> 2018-06-28T16:02:05,924 INFO [07fd131c-6590-4081-b7ed-2f0ae5b16551 HiveServer2-Handler-Pool:
Thread-30] conf.HiveConf: Using the default value passed in for log id: 07fd131c-6590-4081-b7ed-2f0ae5b16551
> 2018-06-28T16:02:05,925 INFO [HiveServer2-Handler-Pool: Thread-30] session.SessionState:
Resetting thread name to HiveServer2-Handler-Pool: Thread-30
> 2018-06-28T16:02:35,131 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: 2018-06-28
16:02:35,130 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.82 sec
> 2018-06-28T16:02:39,527 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: MapReduce
Total cumulative CPU time: 2 seconds 820 msec
> 2018-06-28T16:02:39,607 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Ended
Job = job_1530599349924_0007
> 2018-06-28T16:02:39,643 INFO [HiveServer2-Background-Pool: Thread-97] exec.FileSinkOperator:
Moving tmp dir: hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/_tmp.-ext-10002
to: hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
> 2018-06-28T16:02:39,669 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting
task [Stage-7:CONDITIONAL] in serial mode
> 2018-06-28T16:02:39,688 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Stage-4
is selected by condition resolver.
> 2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Stage-3
is filtered out by condition resolver.
> 2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Stage-5
is filtered out by condition resolver.
> 2018-06-28T16:02:39,690 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting
task [Stage-4:MOVE] in serial mode
> 2018-06-28T16:02:39,691 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Moving
data to directory hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
from hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
> 2018-06-28T16:02:39,785 INFO [HiveServer2-Background-Pool: Thread-97] ql.Driver: Starting
task [Stage-0:MOVE] in serial mode
> 2018-06-28T16:02:39,786 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task: Loading
data to table default.customer_bak partition (country=null) from hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
> 2018-06-28T16:02:39,871 INFO [HiveServer2-Background-Pool: Thread-97] exec.MoveTask:
Partition is: {country=null}
> 2018-06-28T16:02:39,879 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task:
> {code}
> Why doesn't the program continue?
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message