hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravi teja <raviort...@gmail.com>
Subject Re: Repair table doesnt update the transient_lastDdlTime of updated partitions.
Date Tue, 25 Aug 2015 18:06:46 GMT
Thanks a lot Noam, you are a saviour!

Ravi

On Tue, Aug 25, 2015 at 10:03 PM, Noam Hasson <noam.hasson@kenshoo.com>
wrote:

> Hi,
>
> Check if this helps you:
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionTouch
>
> Noam.
>
> On Tue, Aug 25, 2015 at 6:43 PM, ravi teja <raviorteja@gmail.com> wrote:
>
>> Sorry For the incomplete mail, sent bymistake
>>
>> I am working towards a incremental solution on hive based on the
>> transient_lastDdlTime of the partitions.
>> We mostly deal with hive external tables.
>>
>> The transient_lastDdlTime of a partition gets updated when the insertion
>> to the table happens via the insert query route, we are good there.
>>
>> But the issue is, if the file level updation happens in the partition
>> folder, then hive doesnt update transient_lastDdlTime for that partition
>>  and we are not able to get the changed partitions list because of this.
>>
>>
>> Unfortunately we cant change the way the hive table is being updated, its
>> based on the file based update to the underlying location.
>> When we do a file based ingestion, then we have the complete list of
>> partitions updated.
>> But this cannot be passed to the incremental system, hence our source of
>> truth is hive metastore's a and its transient_lastDdlTime.
>>
>> Is there  a way where I can update the transient_lastDdlTime in the
>> metastore , for the partitions changed by adding files?
>> I have tried to re-add the changed partition to the table, for updated
>> ones so that the transient_lastDdlTime will change, but its not possible
>> as it throws an already exists exception.
>>
>> Is there any other way?
>> Thanks in advance.
>>
>> Thanks,
>> Ravi
>>
>> On Tue, Aug 25, 2015 at 9:02 PM, ravi teja <raviorteja@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am working towards a incremental solution on hive based on the
>>> transient_lastDdlTime of the partitions.
>>> If the we in
>>>
>>> Thanks,
>>> Ravi
>>>
>>
>>
>
> This e-mail, as well as any attached document, may contain material which
> is confidential and privileged and may include trademark, copyright and
> other intellectual property rights that are proprietary to Kenshoo Ltd,
>  its subsidiaries or affiliates ("Kenshoo"). This e-mail and its
> attachments may be read, copied and used only by the addressee for the
> purpose(s) for which it was disclosed herein. If you have received it in
> error, please destroy the message and any attachment, and contact us
> immediately. If you are not the intended recipient, be aware that any
> review, reliance, disclosure, copying, distribution or use of the contents
> of this message without Kenshoo's express permission is strictly prohibited.

Mime
View raw message