hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliot West <tea...@gmail.com>
Subject Re: delta file compact take no effect
Date Thu, 11 Jun 2015 07:25:29 GMT
What do you see if you issue:

SHOW COMPACTIONS;

On Thursday, 11 June 2015, r7raul1984@163.com <r7raul1984@163.com> wrote:

>
> I use hive 1.1.0 on hadoop 2.5.0
> After I do some update operation on table u_data_txn.
> My table create many delta file like:
> drwxr-xr-x - hdfs hive 0 2015-02-06 22:52
> /user/hive/warehouse/u_data_txn/delta_0000001_0000001
> -rw-r--r-- 3 hdfs supergroup 346453 2015-02-06 22:52
> /user/hive/warehouse/u_data_txn/delta_0000001_0000001/bucket_00000
> -rw-r--r-- 3 hdfs supergroup 415924 2015-02-06 22:52
> /user/hive/warehouse/u_data_txn/delta_0000001_0000001/bucket_00001
> drwxr-xr-x - hdfs hive 0 2015-02-06 22:58
> /user/hive/warehouse/u_data_txn/delta_0000002_0000002
> -rw-r--r-- 3 hdfs supergroup 807 2015-02-06 22:58
> /user/hive/warehouse/u_data_txn/delta_0000002_0000002/bucket_00000
> -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 22:58
> /user/hive/warehouse/u_data_txn/delta_0000002_0000002/bucket_00001
> drwxr-xr-x - hdfs hive 0 2015-02-06 22:59
> /user/hive/warehouse/u_data_txn/delta_0000003_0000003
> -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 22:59
> /user/hive/warehouse/u_data_txn/delta_0000003_0000003/bucket_00000
> -rw-r--r-- 3 hdfs supergroup 767 2015-02-06 22:59
> /user/hive/warehouse/u_data_txn/delta_0000003_0000003/bucket_00001
> drwxr-xr-x - hdfs hive 0 2015-02-06 23:01
> /user/hive/warehouse/u_data_txn/delta_0000004_0000004
> -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:01
> /user/hive/warehouse/u_data_txn/delta_0000004_0000004/bucket_00000
> -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:01
> /user/hive/warehouse/u_data_txn/delta_0000004_0000004/bucket_00001
> drwxr-xr-x - hdfs hive 0 2015-02-06 23:03
> /user/hive/warehouse/u_data_txn/delta_0000005_0000005
> -rw-r--r-- 3 hdfs supergroup 817 2015-02-06 23:03
> /user/hive/warehouse/u_data_txn/delta_0000005_0000005/bucket_00000
> -rw-r--r-- 3 hdfs supergroup 779 2015-02-06 23:03
> /user/hive/warehouse/u_data_txn/delta_0000005_0000005/bucket_00001
> drwxr-xr-x - hdfs hive 0 2015-02-10 21:34
> /user/hive/warehouse/u_data_txn/delta_0000006_0000006
> -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:34
> /user/hive/warehouse/u_data_txn/delta_0000006_0000006/bucket_00000
> drwxr-xr-x - hdfs hive 0 2015-02-10 21:35
> /user/hive/warehouse/u_data_txn/delta_0000007_0000007
> -rw-r--r-- 3 hdfs supergroup 821 2015-02-10 21:35
> /user/hive/warehouse/u_data_txn/delta_0000007_0000007/bucket_00000
> drwxr-xr-x - hdfs hive 0 2015-03-24 01:16
> /user/hive/warehouse/u_data_txn/delta_0000008_0000008
> -rw-r--r-- 3 hdfs supergroup 1670 2015-03-24 01:16
> /user/hive/warehouse/u_data_txn/delta_0000008_0000008/bucket_00000
> -rw-r--r-- 3 hdfs supergroup 1767 2015-03-24 01:16
> /user/hive/warehouse/u_data_txn/delta_0000008_0000008/bucket_00001
>
>
>
>
>
> *I try ALTER TABLE u_data_txn COMPACT 'MAJOR'; The delta still exist. Then
> I try ALTER TABLE u_data_txn COMPACT 'MINOR'; The delta still exist. How to
>  merge delta file?*
>
> *My config is:*
> <property>
> <name>hive.support.concurrency</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.enforce.bucketing</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.exe.dynamic.partition.mode</name>
> <value>nonstrict</value>
> </property>
> <property>
> <name>hive.txn.manager</name>
> <value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>
> </property>
> <property>
> <name>hive.compactor.initiator.on</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.compactor.worker.threads</name>
> <value>4</value>
> </property>
> ------------------------------
> r7raul1984@163.com <javascript:_e(%7B%7D,'cvml','r7raul1984@163.com');>
>

Mime
View raw message