hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Istvan Fajth (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-23726) Create table may throw MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from a null string)
Date Fri, 19 Jun 2020 10:59:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Istvan Fajth updated HIVE-23726:
--------------------------------
    Description: 
- Given:
 metastore.warehouse.tenant.colocation is set to true
 a test database was created as {{create database test location '/data'}}
 - When:
 I try to create a table as {{create table t1 (a int) location '/data/t1'}}
 - Then:
The create table fails with the following exception:
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException:
Can not create a Path from a null string)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1138)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1143)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:148)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98)
	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: java.lang.IllegalArgumentException:
Can not create a Path from a null string
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63325)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63293)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1780)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1767)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3518)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1052)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1037)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1127)
	... 28 more
{code}

Addtional info:
- The exception does not occur if I don't specify the location in the create table statement
- The exception does not occur if I specify the managedlocation as well in the create database
statement.
- If I do not specify any location for the database then I can not point the external table
to an arbitrary directory and can not use external files to load data into a managed database
via Hive easily this is especially a problem if I have an external data source on something
else then HDFS, and want to pull the data into Hive.
The exception thrown when I try to create the external table specifying an arbitrary location
in a database created as {{create database test2}}:
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:An external table's
location should not be located outside the location specified on its database, table:t1,location:/data_temp/test2/t1,Database
location for external tables:/warehouse/tablespace/external/hive/test2.db)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1138)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1143)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:148)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98)
	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225) 
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) 
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: An external table's location
should not be located outside the location specified on its database, table:t1,location:/data_temp/test2/t1,Database
location for external tables:/warehouse/tablespace/external/hive/test2.db
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63325)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63293)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1780)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1767)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3518)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1052)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1037)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1127)
{code}
- The describe database shows the following output for the test database:
{code}
+----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
| db_name  | comment  |                     location                     | managedlocation
 | owner_name  | owner_type  | parameters  |
+----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
| test     |          | hdfs://vb0206.halxg.cloudera.com:8020/data_temp  |               
  | hive        | USER        |             |
+----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
{code}
Notice that managed location for the database is empty.
- The code where the exception occurs is [the following line|https://github.com/apache/hive/blob/7cdf362f9f1a53025b371a8d5122336ded71d692/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L802]:
{code}
dbLocation = Path.getPathWithoutSchemeAndAuthority(new Path(db.getManagedLocationUri()));
{code}
it seems that the db.getManagedLocationUri() gives back a null reference for the test table.
Metastore debug log does not contain the managed location, so I can not confirm this based
on the logs, it would be nice as part of the fix to add the managedlocation used for the path
validation of the table as well to the [debug log here|https://github.com/apache/hive/blob/7cdf362f9f1a53025b371a8d5122336ded71d692/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L792]
also.


  was:
- Given:
 metastore.warehouse.tenant.colocation is set to true
 a test database was created as {{create database test location '/data'}}
 - When:
 I try to create a table as {{create table t1 (a int) location '/data/t1'}}
 - Then:
The create table fails with the following exception:
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException:
Can not create a Path from a null string)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1138)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1143)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:148)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98)
	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: java.lang.IllegalArgumentException:
Can not create a Path from a null string
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63325)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63293)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1780)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1767)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3518)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1052)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1037)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1127)
	... 28 more
{code}

Addtional info:
- The exception does not occur if I don't specify the location in the create table statement
- If I do not specify a location for the database then I can not point the external table
to an arbitrary directory and can not use external files to load data into a managed database
via Hive easily this is especially a problem if I have an external data source on something
else then HDFS, and want to pull the data into Hive.
The exception thrown when I try to create the external table specifying an arbitrary location
in a database created as {{create database test2}}:
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:An external table's
location should not be located outside the location specified on its database, table:t1,location:/data_temp/test2/t1,Database
location for external tables:/warehouse/tablespace/external/hive/test2.db)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1138)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1143)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:148)
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98)
	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225) 
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) 
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: An external table's location
should not be located outside the location specified on its database, table:t1,location:/data_temp/test2/t1,Database
location for external tables:/warehouse/tablespace/external/hive/test2.db
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63325)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63293)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1780)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1767)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3518)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1052)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1037)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1127)
{code}
- The describe database shows the following output for the test database:
{code}
+----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
| db_name  | comment  |                     location                     | managedlocation
 | owner_name  | owner_type  | parameters  |
+----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
| test     |          | hdfs://vb0206.halxg.cloudera.com:8020/data_temp  |               
  | hive        | USER        |             |
+----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
{code}
Notice that managed location for the database is empty.
- The code where the exception occurs is [the following line|https://github.com/apache/hive/blob/7cdf362f9f1a53025b371a8d5122336ded71d692/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L802]:
{code}
dbLocation = Path.getPathWithoutSchemeAndAuthority(new Path(db.getManagedLocationUri()));
{code}
it seems that the db.getManagedLocationUri() gives back a null reference for the test table.
Metastore debug log does not contain the managed location, so I can not confirm this based
on the logs, it would be nice as part of the fix to add the managedlocation used for the path
validation of the table as well to the [debug log here|https://github.com/apache/hive/blob/7cdf362f9f1a53025b371a8d5122336ded71d692/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L792]
also.



> Create table may throw MetaException(message:java.lang.IllegalArgumentException: Can
not create a Path from a null string)
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-23726
>                 URL: https://issues.apache.org/jira/browse/HIVE-23726
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Istvan Fajth
>            Priority: Major
>
> - Given:
>  metastore.warehouse.tenant.colocation is set to true
>  a test database was created as {{create database test location '/data'}}
>  - When:
>  I try to create a table as {{create table t1 (a int) location '/data/t1'}}
>  - Then:
> The create table fails with the following exception:
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException:
Can not create a Path from a null string)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1138)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1143)
> 	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:148)
> 	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98)
> 	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
> 	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> 	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> 	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> 	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> 	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> 	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: java.lang.IllegalArgumentException:
Can not create a Path from a null string
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63325)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63293)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63219)
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1780)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1767)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3518)
> 	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1052)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1037)
> 	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> 	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> 	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
> 	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1127)
> 	... 28 more
> {code}
> Addtional info:
> - The exception does not occur if I don't specify the location in the create table statement
> - The exception does not occur if I specify the managedlocation as well in the create
database statement.
> - If I do not specify any location for the database then I can not point the external
table to an arbitrary directory and can not use external files to load data into a managed
database via Hive easily this is especially a problem if I have an external data source on
something else then HDFS, and want to pull the data into Hive.
> The exception thrown when I try to create the external table specifying an arbitrary
location in a database created as {{create database test2}}:
> {code}
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:An external table's
location should not be located outside the location specified on its database, table:t1,location:/data_temp/test2/t1,Database
location for external tables:/warehouse/tablespace/external/hive/test2.db)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1138)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1143)
> 	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:148)
> 	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98)
> 	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:80)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
> 	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> 	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> 	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> 	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> 	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)

> 	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)

> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: An external table's location
should not be located outside the location specified on its database, table:t1,location:/data_temp/test2/t1,Database
location for external tables:/warehouse/tablespace/external/hive/test2.db
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63325)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63293)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63219)
> 	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1780)
> 	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1767)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3518)
> 	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1052)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1037)
> 	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> 	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> 	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3411)
> 	at com.sun.proxy.$Proxy35.createTable(Unknown Source)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1127)
> {code}
> - The describe database shows the following output for the test database:
> {code}
> +----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
> | db_name  | comment  |                     location                     | managedlocation
 | owner_name  | owner_type  | parameters  |
> +----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
> | test     |          | hdfs://vb0206.halxg.cloudera.com:8020/data_temp  |          
       | hive        | USER        |             |
> +----------+----------+--------------------------------------------------+------------------+-------------+-------------+-------------+
> {code}
> Notice that managed location for the database is empty.
> - The code where the exception occurs is [the following line|https://github.com/apache/hive/blob/7cdf362f9f1a53025b371a8d5122336ded71d692/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L802]:
> {code}
> dbLocation = Path.getPathWithoutSchemeAndAuthority(new Path(db.getManagedLocationUri()));
> {code}
> it seems that the db.getManagedLocationUri() gives back a null reference for the test
table. Metastore debug log does not contain the managed location, so I can not confirm this
based on the logs, it would be nice as part of the fix to add the managedlocation used for
the path validation of the table as well to the [debug log here|https://github.com/apache/hive/blob/7cdf362f9f1a53025b371a8d5122336ded71d692/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java#L792]
also.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message