hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "@Sanjiv Singh" <sanjiv.is...@gmail.com>
Subject Re: Optimize Hive Query
Date Sun, 26 Jun 2016 20:11:15 GMT
Thanks Gopal for your inputs ....For now I have create NON ACID table and
loaded data ....see below from logs proper group splits happening .

2016-06-25 12:52:00,160 [INFO] [InputInitializer {Map 1} #0]
|tez.HiveSplitGenerator|: Number of grouped splits: 512


On compaction issue , Compaction enabled with two workers. why compaction
not happened ? will check metastore logs.

I have too many ACID tables on hive and how many worker should be
configured ? currently it is 2.

Thanks a lot once again.


Regards
Sanjiv Singh
Mob :  +091 9990-447-339

On Fri, Jun 24, 2016 at 9:14 PM, @Sanjiv Singh <sanjiv.is.on@gmail.com>
wrote:

> Thanks Gopal for your inputs. Let me run compaction explicitly on table
> then see how query works.
>
>
>
> Let
>
> Regards
> Sanjiv Singh
> Mob :  +091 9990-447-339
>
> On Fri, Jun 24, 2016 at 7:53 PM, Gopal Vijayaraghavan <gopalv@apache.org>
> wrote:
>
>>
>> > Yes for this tables, ACID enabled.  it has only 256 files for each
>> >buckets. these are create only when data initially loaded in this table.
>>
>> Yes, the initial load goes in as an insert DELTA too - that requires
>> another compaction to move into base files.
>>
>> The fact that they haven't been automatically compacted yet, suggests that
>> the compactor isn't working for some reason (check hive metastore logs).
>>
>> > One thing that I am not able to understand that its is running with 1
>> >MAPPER.
>>
>> The size of deltas shows up as 0, till the compaction goes through - in
>> Hive2, it will be -1 which will be correctly interpreted as "unknown
>> size".
>>
>>
>> > | -rw-r--r--   3 H56473 hdfs  215973009 2016-06-23 17:38
>>
>> >/apps/hive/warehouse/PRDDB.db/tuning_dd_key/delta_0001570_0001570/bucket_0
>> >0000  |
>>
>> Clearly an issue due to the lack of compaction - I see a single delta with
>> 255 buckets and no base_* files at all.
>>
>> Cheers,
>> Gopal
>>
>>
>>
>>
>>
>>
>>
>>
>>
>

Mime
View raw message