impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amos Bird (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-1654: Support general predicates in most partition DDL operations.
Date Fri, 01 Jul 2016 10:59:47 GMT
Amos Bird has uploaded a new patch set (#6).

Change subject: IMPALA-1654: Support general predicates in most partition DDL operations.
......................................................................

IMPALA-1654: Support general predicates in most partition DDL operations.

This commit handles partition DDL in a more general way. We can now use
compound predicates to specify a list of partitions in statement like
ALTER TABLE DROP PARTITION and COMPUTE INCREMENTAL STATS, etc. It will
also make sure some statements only accept one partition at a time, such
as PARTITION SET LOCATION and LOAD DATA. ALTER TABLE ADD PARTITION
remains using the old PartitionKeyValue's logic.

The changed partition related DDLs are as follows,

Table: p (i int) partitioned by (j int, k int)
Partitions:
+-------+---+-------+--------+------+--------------+-------------------+
| j     | k | #Rows | #Files | Size | Bytes Cached | Cache Replication |
+-------+---+-------+--------+------+--------------+-------------------+
| 1     | 1 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| 1     | 2 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| 1     | 3 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| 2     | 1 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| 2     | 2 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| 2     | 3 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| Total |   | -1    | 0      | 0B   | 0B           |                   |
+-------+---+-------+--------+------+--------------+-------------------+

1. show files in p partition (j<2, k=3);
2. alter table p partition (j<2, k>2) set cached in 'testPool';

// j can appear more than once,
3.1. alter table p partition (j<2, j>0, k<>4) set uncached;
// it is the same as
3.2. alter table p partition (j<2 and j>0, k<>4) set uncached;

// missing 'k' matches all values of k
4. alter table p partition (j<2) set fileformat textfile;
5. alter table p partition (j is null) set serdeproperties ("k"="v");
6. alter table p partition (j is not null) set tblproperties ("k"="v");
7. alter table p drop partition (j<2);
8. compute incremental stats p partition(j<2);

The remaining old partition related DDLs are as follows,

1. load data inpath '/path/from' into table p partition (j=2, k=3);
2. alter table p add partition (j=2, k=5);
3. alter table p partition (j=2, k=3) set location '/path/to';
4. insert into p partition (j=2, k=3) values (1), (2), (3);

Examples:

[localhost.localdomain:21000] >
alter table p drop partition (j=2, k=3);
Query: alter table p drop partition (j=2, k=3)
+-------------------------+
| summary                 |
+-------------------------+
| Dropped 1 partition(s). |
+-------------------------+
Fetched 1 row(s) in 0.78s
[localhost.localdomain:21000] >
alter table p drop partition (j=2, k<3);
Query: alter table p drop partition (j=2, k<3)
+-------------------------+
| summary                 |
+-------------------------+
| Dropped 2 partition(s). |
+-------------------------+
Fetched 1 row(s) in 0.41s
[localhost.localdomain:21000] >
alter table p drop partition (k=1);
Query: alter table p drop partition (k=1)
+-------------------------+
| summary                 |
+-------------------------+
| Dropped 1 partition(s). |
+-------------------------+
Fetched 1 row(s) in 0.25s
[localhost.localdomain:21000] > show partitions p;
Query: show partitions p
+-------+---+-------+--------+------+--------------+-------------------+
| j     | k | #Rows | #Files | Size | Bytes Cached | Cache Replication |
+-------+---+-------+--------+------+--------------+-------------------+
| 1     | 2 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| 1     | 3 | -1    | 0      | 0B   | NOT CACHED   | NOT CACHED        |
| Total |   | -1    | 0      | 0B   | 0B           |                   |
+-------+---+-------+--------+------+--------------+-------------------+
Fetched 3 row(s) in 0.01s

Change-Id: I2c9162fcf9d227b8daf4c2e761d57bab4e26408f
---
M be/src/service/query-exec-state.cc
M common/thrift/CatalogService.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableAddPartitionStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableDropPartitionStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableSetCachedStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableSetFileFormatStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableSetLocationStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableSetStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/com/cloudera/impala/analysis/AlterTableStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/DropStatsStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/LoadDataStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/PartitionSpec.java
M fe/src/main/java/com/cloudera/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
M fe/src/main/java/com/cloudera/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/com/cloudera/impala/analysis/ParserTest.java
M shell/impala_client.py
M testdata/workloads/functional-query/queries/QueryTest/alter-table.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats-incremental.test
M testdata/workloads/functional-query/queries/QueryTest/compute-stats.test
A testdata/workloads/functional-query/queries/QueryTest/partition-ddl-predicates.test
M testdata/workloads/functional-query/queries/QueryTest/show.test
M tests/metadata/test_ddl.py
35 files changed, 991 insertions(+), 490 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/63/1563/6
-- 
To view, visit http://gerrit.cloudera.org:8080/1563
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2c9162fcf9d227b8daf4c2e761d57bab4e26408f
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Amos Bird <amosbird@gmail.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Amos Bird <amosbird@gmail.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Jim Apple <jbapple@cloudera.com>

Mime
View raw message