hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal Vijayaraghavan <gop...@apache.org>
Subject Re: Optimize Hive Query
Date Fri, 24 Jun 2016 23:53:11 GMT

> Yes for this tables, ACID enabled.  it has only 256 files for each
>buckets. these are create only when data initially loaded in this table.

Yes, the initial load goes in as an insert DELTA too - that requires
another compaction to move into base files.

The fact that they haven't been automatically compacted yet, suggests that
the compactor isn't working for some reason (check hive metastore logs).

> One thing that I am not able to understand that its is running with 1
>MAPPER.  

The size of deltas shows up as 0, till the compaction goes through - in
Hive2, it will be -1 which will be correctly interpreted as "unknown size".

 
> | -rw-r--r--   3 H56473 hdfs  215973009 2016-06-23 17:38
>/apps/hive/warehouse/PRDDB.db/tuning_dd_key/delta_0001570_0001570/bucket_0
>0000  |

Clearly an issue due to the lack of compaction - I see a single delta with
255 buckets and no base_* files at all.

Cheers,
Gopal









Mime
View raw message