hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Szita <sz...@cloudera.com>
Subject Re: Review Request 56118: DROP TABLE in hive doesn't Throw Error
Date Mon, 06 Feb 2017 11:10:45 GMT


> On Feb. 3, 2017, 2:42 p.m., Aihua Xu wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, line 1679
> > <https://reviews.apache.org/r/56118/diff/2/?file=1623348#file1623348line1679>
> >
> >     With this approach, I feel it will place Hive in worse situation since now it
could fail during the data deletion and it failed, metastore is rolled back but part of the
data is removed. The table is in unusable conditition.
> >     
> >     Without this change, we roll back when the metadata drop has issues. After roll
back, the table is still consistent.
> >     
> >     If the metadata drop succeeds, then we continue deleting the data. It's not
that good that the data could not be purged, but it won't cause any inconsistency since metadata
is gone and the data is not visiable to the caller.

Okay, I see, let's stick to removing the metadata as well then.
Although I still suggest we _do not return success_ to the client - if some file/dir couldn't
have been deleted then the drop_table command clearly didn't succeeded entirely and the user
should know about this.

So I think the exception throwing changes should be kept (along with the configurability of
this strict mode). Also, it should be logged additionally what path could not have been deleted,
and what (partition) paths remained on disk because of this.
This all will ultimately give the user: (1) notification that due to an error some data is
remained; (2) what paths have to be considered for retrying to delete them manually (can't
retry from Hive itself, since we drop the metadata regardless).

So - having much more experience in Hive, what do you think about this approach?


- Adam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56118/#review164121
-----------------------------------------------------------


On Feb. 3, 2017, 2:22 p.m., Adam Szita wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56118/
> -----------------------------------------------------------
> 
> (Updated Feb. 3, 2017, 2:22 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Peter Vary, and Sergio Pena.
> 
> 
> Bugs: HIVE-14181
>     https://issues.apache.org/jira/browse/HIVE-14181
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Failure during table drop doesn't throw errors and results in success - some times data
resides in warehouse, but table (meta data) is removed from metastore resulting in incosistency
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 53b9b0c6962c9b1cd2eef1cb71687ec0245cfac3

>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
af125c38236582ba532f5e3de3d2ba724f38b101 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java f8c3c4e48db0df9d6c18801bcd61f9e5dc6eb7c2

> 
> Diff: https://reviews.apache.org/r/56118/diff/
> 
> 
> Testing
> -------
> 
> -Added test case
> -Tested on cluster
> 
> 
> Thanks,
> 
> Adam Szita
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message