From issues-return-18810-archive-asf-public=cust-asf.ponee.io@hawq.incubator.apache.org Wed Aug 8 03:56:07 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 64AD0180657 for ; Wed, 8 Aug 2018 03:56:06 +0200 (CEST) Received: (qmail 43891 invoked by uid 500); 8 Aug 2018 01:56:05 -0000 Mailing-List: contact issues-help@hawq.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hawq.incubator.apache.org Delivered-To: mailing list issues@hawq.incubator.apache.org Received: (qmail 43882 invoked by uid 99); 8 Aug 2018 01:56:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Aug 2018 01:56:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2AA25CB9C4 for ; Wed, 8 Aug 2018 01:56:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4 X-Spam-Level: X-Spam-Status: No, score=-4 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id g2mpOwOC0V6a for ; Wed, 8 Aug 2018 01:56:04 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 8713C5F42E for ; Wed, 8 Aug 2018 01:56:03 +0000 (UTC) Received: (qmail 43861 invoked by uid 99); 8 Aug 2018 01:56:02 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Aug 2018 01:56:02 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id A48CBDFCB9; Wed, 8 Aug 2018 01:56:02 +0000 (UTC) From: huor To: issues@hawq.incubator.apache.org Reply-To: issues@hawq.incubator.apache.org References: In-Reply-To: Subject: [GitHub] incubator-hawq pull request #1384: HAWQ-1628. Add HDFS protocol for external... Content-Type: text/plain Message-Id: <20180808015602.A48CBDFCB9@git1-us-west.apache.org> Date: Wed, 8 Aug 2018 01:56:02 +0000 (UTC) Github user huor commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1384#discussion_r208435664 --- Diff: src/backend/commands/tablecmds.c --- @@ -939,38 +983,217 @@ DefineExternalRelation(CreateExternalStmt *createExtStmt) char* commandString = NULL; char rejectlimittype = '\0'; char formattype; + char* formattername = NULL; int rejectlimit = -1; int encoding = -1; int preferred_segment_num = -1; bool issreh = false; /* is single row error handling requested? */ + bool isexternal = createExtStmt->isexternal; bool iswritable = createExtStmt->iswritable; bool isweb = createExtStmt->isweb; + bool forceCreateDir = createExtStmt->forceCreateDir; + + bool isExternalHdfs = false; + bool isExternalMagma = false; + bool isExternalHive = false; + char* location = NULL; + int location_len = NULL; /* * now set the parameters for keys/inheritance etc. Most of these are * uninteresting for external relations... */ - createStmt->relation = createExtStmt->relation; - createStmt->tableElts = createExtStmt->tableElts; - createStmt->inhRelations = NIL; - createStmt->constraints = NIL; - createStmt->options = NIL; - createStmt->oncommit = ONCOMMIT_NOOP; - createStmt->tablespacename = NULL; + createStmt->base = createExtStmt->base; + // external table options is not compatible with internal table + // set NIL here + createStmt->base.options = NIL; createStmt->policy = createExtStmt->policy; /* policy was set in transform */ - + + /* + * Recognize formatter option if there are some tokens found in parser. + * This design is to give CREATE EXTERNAL TABLE DDL the flexiblity to + * support user defined external formatter options. + */ + recognizeExternalRelationFormatterOptions(createExtStmt); + + /* + * Get tablespace, database, schema for the relation + */ + RangeVar *rel = createExtStmt->base.relation; + // get tablespace name for the relation + Oid tablespace_id = (gp_upgrade_mode) ? DEFAULTTABLESPACE_OID : GetDefaultTablespace(); + if (!OidIsValid(tablespace_id)) + { + tablespace_id = get_database_dts(MyDatabaseId); + } + char *tablespace_name = get_tablespace_name(tablespace_id); + + // get database name for the relation + char *database_name = rel->catalogname ? rel->catalogname : get_database_name(MyDatabaseId); + + // get schema name for the relation + char *schema_name = get_namespace_name(RangeVarGetCreationNamespace(rel)); + + // get table name for the relation + char *table_name = rel->relname; + + /* + * Do some special logic when we use custom + */ + if (exttypeDesc->exttabletype == EXTTBL_TYPE_LOCATION) + { + if (exttypeDesc->location_list == NIL) + { + if (dfs_url == NULL) + { + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("Cannot create table on HDFS when the service is not available"), + errhint("Check HDFS service and hawq_dfs_url configuration"), + errOmitLocation(true))); + } + + location_len = strlen(PROTOCOL_HDFS) + /* hdfs:// */ + strlen(dfs_url) + /* hawq_dfs_url */ + // 1 + strlen(filespace_name) + /* '/' + filespace name */ + 1 + strlen(tablespace_name) + /* '/' + tablespace name */ + 1 + strlen(database_name) + /* '/' + database name */ + 1 + strlen(schema_name) + /* '/' + schema name */ + 1 + strlen(table_name) + 1; /* '/' + table name + '\0' */ + + char *path; + + if (createExtStmt->parentPath == NULL) { + path = (char *)palloc(sizeof(char) * location_len); + sprintf(path, "%s%s/%s/%s/%s/%s", + PROTOCOL_HDFS, + dfs_url, /* filespace_name, */ tablespace_name, + database_name, schema_name, table_name); + } + else { + path = (char *)palloc(sizeof(char) * + (location_len + strlen(createExtStmt->parentPath)+1)); + sprintf(path, "%s%s/%s/%s/%s/%s/%s", + PROTOCOL_HDFS, + dfs_url, /* filespace_name, */ tablespace_name, + database_name, schema_name, + createExtStmt->parentPath, table_name); + } + + exttypeDesc->location_list = list_make1((Node *) makeString(path)); + } + + /* Check the location to extract protocol */ + ListCell *first_uri = list_head(exttypeDesc->location_list); + Value *v = lfirst(first_uri); + char *uri_str = pstrdup(v->val.str); + Uri *uri = ParseExternalTableUri(uri_str); + bool isHdfs = is_hdfs_protocol(uri); + bool isHive = is_hive_protocol(uri); + + pfree(uri_str); + FreeExternalTableUri(uri); + + if (isHdfs) + { + /* We have an HDFS external protocol */ + isExternalHdfs = true; + } + + if (isHive) + { + /* We have an HIVE external protocol */ + isExternalHive = true; + } + } + + if (isExternalHdfs) + { + if (list_length(exttypeDesc->location_list)!= 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("Only support 1 external HDFS location. " + "Now input %d location(s)", + list_length(exttypeDesc->location_list)))); + } + + if (isExternalHive) + { + if (list_length(exttypeDesc->location_list)!= 1) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("Only support 1 external HIVE location. " + "Now input %d location(s)", + list_length(exttypeDesc->location_list)))); + } + + if (isExternalMagma) --- End diff -- No magma ---