sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Szabolcs Vasas <vasas.szabo...@gmail.com>
Subject Re: Review Request 63920: Sqoop ignores metastore properties defined in sqoop-site.xml
Date Fri, 05 Jan 2018 10:05:05 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Jan. 5, 2018, 10:05 a.m.)

Review request for Sqoop.

Bugs: SQOOP-3255

Repository: sqoop-trunk


This patch restores the following metastore behavior:
If --meta-connect parameter is present then Sqoop will try to connect to the metastore database
specified in this parameter value. It will use the username and password specified in --meta-username
and --meta-password parameters if they are not present Sqoop will use empty username/password.
If the database in the connection string is not supported then Sqoop will throw an exception.

If --meta-connect parameter is not present Sqoop checks if sqoop.metastore.client.enable.autoconnect
configuartion parameter is set to false. If the parameter is false then Sqoop throws an error
since there are no applicable metastore implementations.
The next step is to check sqoop.metastore.client.autoconnect.url configuration parameter.
In this parameter users can specify a JDBC connection string to a metastore database (HSQLDB,
MySQL, Oracle, PostgreSQL, DB2 and SQL Server are supported) if it is not present Sqoop will
create an embedded HSQLDB metastore in the home directory of the OS user which executes the
sqoop command. The username and password can be specified in sqoop.metastore.client.autoconnect.username
(default value: SA) and sqoop.metastore.client.autoconnect.password (default value is the
empty string). If the connection string in sqoop.metastore.client.autoconnect.url is not supported
then Sqoop will throw an exception.

 - It turned out that the default value initialisation did not work, because the configuration
parameters are not present in system properties just in the configuration object
 - getLocalAutoConnectString is moved to AutoGenericJobStorage.getHomeDirFileConnectStr

 - The class is restored version of AutoHsqldbStorage.
 - it extends GenericJobStorage

 - the auto metastore parameters are moved to AutoGenericJobStorage
 - driverClass field is removed since it can be derived from the metastoreConnectStr
 - canAccept is modified

 - SUPPORTED_DRIVERS is moved to GenericJobStorage since JobTool just delegates to job storages
so the supported databases depend on the job storage implementation.
 - applyMetastoreOptions now copies the parameter values from the CommandLine object to the
SqoopObject instance.
 - the storage descriptor map initialized in initializeStorageDescriptor now

Diffs (updated)

  src/java/org/apache/sqoop/SqoopOptions.java d5fdfba14 
  src/java/org/apache/sqoop/metastore/AutoGenericJobStorage.java PRE-CREATION 
  src/java/org/apache/sqoop/metastore/GenericJobStorage.java e4ffde210 
  src/java/org/apache/sqoop/metastore/JobStorageFactory.java 9a348d518 
  src/java/org/apache/sqoop/tool/JobTool.java 5b95c7d00 
  src/test/com/cloudera/sqoop/TestIncrementalImport.java 166792b51 
  src/test/com/cloudera/sqoop/metastore/SavedJobsTestBase.java 81789e76f 
  src/test/com/cloudera/sqoop/metastore/TestMetastoreConfigurationParameters.java PRE-CREATION

  src/test/com/cloudera/sqoop/testutil/HsqldbTestServer.java 8d0a30db8 
  src/test/org/apache/sqoop/metastore/TestAutoGenericJobStorage.java PRE-CREATION 
  src/test/org/apache/sqoop/metastore/TestGenericJobStorage.java PRE-CREATION 
  src/test/org/apache/sqoop/metastore/TestGenericJobStorageValidate.java PRE-CREATION 
  src/test/org/apache/sqoop/testutil/Argument.java PRE-CREATION 
  src/test/org/apache/sqoop/testutil/ArgumentUtils.java PRE-CREATION 

Diff: https://reviews.apache.org/r/63920/diff/6/

Changes: https://reviews.apache.org/r/63920/diff/5-6/


Executed unit and third party test suite, created new test cases.


Szabolcs Vasas

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message