hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Kuzmenko <f1she...@gmail.com>
Subject Re: Hive compaction didn't launch
Date Wed, 27 Jul 2016 10:31:53 GMT
One more thing. I'm using Apache Storm to stream data in Hive. And when I
turned off Storm topology compactions started to work properly.

On Tue, Jul 26, 2016 at 6:28 PM, Igor Kuzmenko <f1sherox@gmail.com> wrote:

> I'm using Hive 1.2.1 transactional table. Inserting data in it via Hive
> Streaming API. After some time i expect compaction to start but it didn't
> happen:
>
> Here's part of log, which shows that compactor initiator thread doesn't
> see any delta files:
> *2016-07-26 18:06:52,459 INFO  [Thread-8]: compactor.Initiator
> (Initiator.java:run(89)) - Checking to see if we should compact
> default.data_aaa.dt=20160726*
> *2016-07-26 18:06:52,496 DEBUG [Thread-8]: io.AcidUtils
> (AcidUtils.java:getAcidState(432)) - in directory
> hdfs://sorm-master01.msk.mts.ru:8020/apps/hive/warehouse/data_aaa/dt=20160726
> base = null deltas = 0*
>
> *2016-07-26 18:06:52,496 DEBUG [Thread-8]: compactor.Initiator
> (Initiator.java:determineCompactionType(271)) - delta size: 0 base size: 0
> threshold: 0.1 will major compact: false*
>
> But in that directory there's actually 23 files:
>
> hadoop fs -ls /apps/hive/warehouse/data_aaa/dt=20160726
> Found 23 items
> -rw-r--r--   3 storm hdfs          4 2016-07-26 17:20
> /apps/hive/warehouse/data_aaa/dt=20160726/_orc_acid_version
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:22
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71741256_71741355
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:23
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71762456_71762555
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:25
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71787756_71787855
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:26
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71795756_71795855
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:27
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71804656_71804755
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:29
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71828856_71828955
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:30
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71846656_71846755
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:32
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71850756_71850855
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:33
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71867356_71867455
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:34
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71891556_71891655
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:36
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71904856_71904955
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:37
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71907256_71907355
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:39
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71918756_71918855
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:40
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71947556_71947655
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:41
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71960656_71960755
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:43
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71963156_71963255
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:44
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71964556_71964655
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:46
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_71987156_71987255
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:47
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_72015756_72015855
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:48
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_72021356_72021455
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:50
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_72048756_72048855
> drwxrwxrwx   - storm hdfs          0 2016-07-26 17:50
> /apps/hive/warehouse/data_aaa/dt=20160726/delta_72070856_72070955
>
>
> Full log here <http://pastebin.com/gHwvgRUV>.
>
> What could go wrong?
>

Mime
View raw message