From dev-return-35461-apmail-sqoop-dev-archive=sqoop.apache.org@sqoop.apache.org Fri Jan 5 10:05:12 2018 Return-Path: X-Original-To: apmail-sqoop-dev-archive@www.apache.org Delivered-To: apmail-sqoop-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A9123174CD for ; Fri, 5 Jan 2018 10:05:12 +0000 (UTC) Received: (qmail 66533 invoked by uid 500); 5 Jan 2018 10:05:12 -0000 Delivered-To: apmail-sqoop-dev-archive@sqoop.apache.org Received: (qmail 66499 invoked by uid 500); 5 Jan 2018 10:05:12 -0000 Mailing-List: contact dev-help@sqoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@sqoop.apache.org Delivered-To: mailing list dev@sqoop.apache.org Received: (qmail 66488 invoked by uid 500); 5 Jan 2018 10:05:12 -0000 Delivered-To: apmail-incubator-sqoop-dev@incubator.apache.org Received: (qmail 66485 invoked by uid 99); 5 Jan 2018 10:05:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jan 2018 10:05:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C8363C0A33; Fri, 5 Jan 2018 10:05:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.44 X-Spam-Level: **** X-Spam-Status: No, score=4.44 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_NONE=-0.0001, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id HssRyRNtA3vL; Fri, 5 Jan 2018 10:05:10 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 717CE5F3B5; Fri, 5 Jan 2018 10:05:10 +0000 (UTC) Received: from reviews.apache.org (unknown [10.41.0.12]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 0C522E095C; Fri, 5 Jan 2018 10:05:10 +0000 (UTC) Received: from reviews-vm2.apache.org (localhost [IPv6:::1]) by reviews.apache.org (ASF Mail Server at reviews-vm2.apache.org) with ESMTP id F1C1DC4004D; Fri, 5 Jan 2018 10:05:07 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============0249639946027729240==" MIME-Version: 1.0 Subject: Re: Review Request 63920: Sqoop ignores metastore properties defined in sqoop-site.xml From: Szabolcs Vasas To: Szabolcs Vasas , Boglarka Egyed , Sqoop Date: Fri, 05 Jan 2018 10:05:05 -0000 Message-ID: <20180105100505.59325.10993@reviews-vm2.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Szabolcs Vasas X-ReviewGroup: Sqoop X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/63920/ X-Sender: Szabolcs Vasas References: <20180105094626.59087.79634@reviews-vm2.apache.org> In-Reply-To: <20180105094626.59087.79634@reviews-vm2.apache.org> X-ReviewBoard-Diff-For: src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java X-ReviewBoard-Diff-For: src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java X-ReviewBoard-Diff-For: src/test/org/apache/sqoop/testutil/Argument.java X-ReviewBoard-Diff-For: src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java X-ReviewBoard-Diff-For: src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java X-ReviewBoard-Diff-For: src/test/org/apache/sqoop/testutil/ArgumentUtils.java X-ReviewBoard-Diff-For: src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java Reply-To: Szabolcs Vasas X-ReviewRequest-Repository: sqoop-trunk --===============0249639946027729240== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63920/ ----------------------------------------------------------- (Updated Jan. 5, 2018, 10:05 a.m.) Review request for Sqoop. Bugs: SQOOP-3255 https://issues.apache.org/jira/browse/SQOOP-3255 Repository: sqoop-trunk Description ------- This patch restores the following metastore behavior: If --meta-connect parameter is present then Sqoop will try to connect to the metastore database specified in this parameter value. It will use the username and password specified in --meta-username and --meta-password parameters if they are not present Sqoop will use empty username/password. If the database in the connection string is not supported then Sqoop will throw an exception. If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect configuartion parameter is set to false. If the parameter is false then Sqoop throws an error since there are no applicable metastore implementations. The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter. In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB, MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will create an embedded HSQLDB metastore in the home directory of the OS user which executes the sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username (default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported then Sqoop will throw an exception. SqoopOptions: - It turned out that the default value initialisation did not work, because the configuration parameters are not present in system properties just in the configuration object - getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr AutoGenericJobStorage: - The class is restored version of AutoHsqldbStorage. - it extends GenericJobStorage GenericJobStorage: - the auto metastore parameters are moved to AutoGenericJobStorage - driverClass field is removed since it can be derived from the metastoreConnectStr - canAccept is modified JobTool: - SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages so the supported databases depend on the job storage implementation. - applyMetastoreOptions now copies the parameter values from the CommandLine object to the SqoopObject instance. - the storage descriptor map initialized in initializeStorageDescriptor now Diffs (updated) ----- src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14 src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210 src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518 src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00 src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51 src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8 src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION src/test/org/apache/sqoop/testutil/Argument.java PRE-CREATION src/test/org/apache/sqoop/testutil/ArgumentUtils.java PRE-CREATION Diff: https://reviews.apache.org/r/63920/diff/6/ Changes: https://reviews.apache.org/r/63920/diff/5-6/ Testing ------- Executed unit and third party test suite, created new test cases. Thanks, Szabolcs Vasas --===============0249639946027729240==--