impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailesh Mukil (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3459: Add test for DROP TABLE PURGE for S3
Date Thu, 12 May 2016 01:31:57 GMT
Sailesh Mukil has uploaded a new patch set (#2).

Change subject: IMPALA-3459: Add test for DROP TABLE PURGE for S3
......................................................................

IMPALA-3459: Add test for DROP TABLE PURGE for S3

It was previously thought that PURGE had no effect on S3. However,
the Hive Metastore actually created a .Trash directory and copied the
files there when a DROP TABLE was conducted from Impala.

This patch just enables the existing PURGE tests for S3. There were a
few reasons this wasn't working before. The paths given to the S3
client (boto3) should not have a leading "/". This has been fixed as
it doesn't make a difference for HDFS if that exists or not.

Also, PURGE is a pure delete whereas a regular DROP is a copy. A copy
is consistent whereas a delete is only eventually consistent, so when
we PURGE a table or partition, the files will still be visible for
sometime after the query has completed. The tests have been modified
to accomodate for this case as well.

Change-Id: I52d2451e090b00ae2fd9a879c28defa6c940047c
---
M tests/common/skip.py
M tests/metadata/test_ddl.py
2 files changed, 52 insertions(+), 44 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/36/3036/2
-- 
To view, visit http://gerrit.cloudera.org:8080/3036
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I52d2451e090b00ae2fd9a879c28defa6c940047c
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Sailesh Mukil <sailesh@cloudera.com>

Mime
View raw message