Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Message-Id: <201607051920.u65JK3D0005944@ip-10-146-233-104.ec2.internal>
Date: Tue, 5 Jul 2016 19:19:54 +0000
From: "Bikramjeet Vig (Code Review)" <gerrit@cloudera.org>
To: Casey Ching <casey@cloudera.com>, impala-cr@cloudera.com,
        dev@impala.incubator.apache.org
CC: Marcel Kornacker <marcel@cloudera.com>,
        Taras Bobrovytsky <tbobrovytsky@cloudera.com>,
        Matthew Jacobs <mj@cloudera.com>,
        Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Reply-To: bikramjeet.vig@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-3721=3A_Simplify_creating_external_Kudu_tables=0A?=
In-Reply-To: <gerrit.1458773009000.Ic141102818b6dad3016181b179a14024d0ff709d@gerrit.cloudera.org>
References: <gerrit.1458773009000.Ic141102818b6dad3016181b179a14024d0ff709d@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Tue, 05 Jul 2016 19:20:10 -0000

Bikramjeet Vig has uploaded a new patch set (#24).

Change subject: IMPALA-3721: Simplify creating external Kudu tables
......................................................................

IMPALA-3721: Simplify creating external Kudu tables

Creating an external Kudu table was a lot harder than it needed to be
because the columns needed to be specified twice, once in Kudu and once
again in Impala. Also table properties needed to be specified and that
is more tedious than it needs to be.

Changes:
  1) Read table schema from Kudu. When attempting to create an external
     table "foo" in database "bar", Impala will search for a Kudu table
     name "foo.bar" and "bar" (Kudu doesn't have database name spaces
     yet.)
  2) The Kudu table is now required to exist at the time of creation in
     Impala.
  3) Disallow table properties that could conflict with an existing
     table. Ex: key_columns cannot be specified.
  4) Add KUDU as a file format.
  5) Add a startup flag to catalogd to specify the default Kudu master
     addresses. The flag is used as the default value for the table
     property kudu_master_addresses.
  6) Fix a post merge issue (IMPALA-3178) where DROP DATABASE CASCADE
     wasn't implemented for Kudu tables and silently ignored. The Kudu
     tables wouldn't be removed in Kudu.
  7) Moved the kudu_master_addrs flag from catalogd to impalad. The flag
     was originally in catalogd so there would only be one definition of
     the value. Unfortunately the way CTAS is implemented, the complete
     to-be-created table definition needs to be available during analysis.
     The most practical thing to do is to move the flag to the impalad so
     it is easily accessible.
  8) Remove DDL delegates. There was only one functional delegate (for
     Kudu) the existence of the other delegate and the use of delegates in
     general has led to confusion. The Kudu delegate only exists to provide
     functionality missing from Hive. Eventually Hive should have the needed
     functionality and the Kudu delegate (renamed in this patch to KuduCatalogOpExecutor)
     can be removed.

Now an external Kudu table can be created with "CREATE EXTERNAL TABLE t
STORED AS KUDU" assuming table "t" exists in Kudu. Users can still
override table properties such as the Kudu table name or master
addresses in the usual way.

Change-Id: Ic141102818b6dad3016181b179a14024d0ff709d
---
M be/src/catalog/catalog.cc
M be/src/service/frontend.cc
M bin/start-catalogd.sh
M bin/start-impala-cluster.py
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/com/cloudera/impala/analysis/AnalysisUtils.java
M fe/src/main/java/com/cloudera/impala/analysis/ColumnDef.java
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableDataSrcStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/CreateTableStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/ToSqlUtils.java
M fe/src/main/java/com/cloudera/impala/catalog/Db.java
M fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/com/cloudera/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java
M fe/src/main/java/com/cloudera/impala/catalog/Table.java
D fe/src/main/java/com/cloudera/impala/catalog/delegates/DdlDelegate.java
D fe/src/main/java/com/cloudera/impala/catalog/delegates/KuduDdlDelegate.java
D fe/src/main/java/com/cloudera/impala/catalog/delegates/UnsupportedOpDelegate.java
M fe/src/main/java/com/cloudera/impala/service/CatalogOpExecutor.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/main/java/com/cloudera/impala/service/JniCatalog.java
M fe/src/main/java/com/cloudera/impala/service/JniFrontend.java
A fe/src/main/java/com/cloudera/impala/service/KuduCatalogOpExecutor.java
A fe/src/main/java/com/cloudera/impala/util/KuduClient.java
M fe/src/main/java/com/cloudera/impala/util/KuduUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzerTest.java
M fe/src/test/java/com/cloudera/impala/analysis/ParserTest.java
M fe/src/test/java/com/cloudera/impala/testutil/CatalogServiceTestCatalog.java
M testdata/bin/generate-schema-statements.py
M testdata/datasets/functional/functional_schema_template.sql
D testdata/workloads/functional-query/queries/QueryTest/create_kudu.test
M testdata/workloads/functional-query/queries/QueryTest/kudu-scan-node.test
M testdata/workloads/functional-query/queries/QueryTest/kudu-show-create.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_crud.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_partition_ddl.test
M testdata/workloads/functional-query/queries/QueryTest/kudu_stats.test
A tests/common/kudu_test_suite.py
A tests/custom_cluster/test_kudu.py
M tests/query_test/test_kudu.py
44 files changed, 1,355 insertions(+), 801 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/17/2617/24
-- 
To view, visit http://gerrit.cloudera.org:8080/2617
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic141102818b6dad3016181b179a14024d0ff709d
Gerrit-PatchSet: 24
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Casey Ching <casey@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet.vig@cloudera.com>
Gerrit-Reviewer: Casey Ching <casey@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <marcel@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tbobrovytsky@cloudera.com>