hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16277) Exchange Partition between filesystems throws "IllegalArgumentException Wrong FS"
Date Sat, 01 Apr 2017 18:23:41 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952341#comment-15952341
] 

Sahil Takiar commented on HIVE-16277:
-------------------------------------

[~vihangk1], [~spena], [~mohitsabharwal] anyone chance someone could take a look at this?
Right now, I'm looking for some feedback on the approach. I still have some code cleanup +
unit testing to do.

I'm trying to add support for exchanging of partitions across filesystems. I introduced a
new HiveMetaStore method call {{exchange_partitions_metadata}} which only exchanges the partition
metadata in HMS, but doesn't actually move the data. This is different from the existing {{exchange_partitions}}
method in HMS which both renames the partition on the physical filesystem and switches the
metadata.

I wanted to move the actual renaming of directories to {{Hive.java}} - in the case where a
folder needs to be moved cross-filesystem, the partition data needs to be copied. Doing this
in HMS doesn't sound like the right approach, as copying the data could take hours, depending
on the size of the partition.

> Exchange Partition between filesystems throws "IllegalArgumentException Wrong FS"
> ---------------------------------------------------------------------------------
>
>                 Key: HIVE-16277
>                 URL: https://issues.apache.org/jira/browse/HIVE-16277
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-16277.1.patch, HIVE-16277.2.patch, HIVE-16277.3.patch, HIVE-16277.4.patch
>
>
> The following query: {{alter table s3_tbl exchange partition (country='USA') with table
hdfs_tbl}} fails with the following exception:
> {code}
> Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got
exception: java.lang.IllegalArgumentException Wrong FS: s3a://[bucket]/table/country=USA,
expected: file:///)
> 	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:379)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
> 	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:347)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:361)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
exception: java.lang.IllegalArgumentException Wrong FS: s3a://[bucket]/table/country=USA,
expected: file:///)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.exchangeTablePartitions(Hive.java:3553)
> 	at org.apache.hadoop.hive.ql.exec.DDLTask.exchangeTablePartition(DDLTask.java:4691)
> 	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:570)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2182)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1838)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1525)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1231)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> 	... 11 more
> Caused by: MetaException(message:Got exception: java.lang.IllegalArgumentException Wrong
FS: s3a://[bucket]/table/country=USA, expected: file:///)
> 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1387)
> 	at org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:208)
> 	at org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:200)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.exchange_partitions(HiveMetaStore.java:2967)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> 	at com.sun.proxy.$Proxy28.exchange_partitions(Unknown Source)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.exchange_partitions(HiveMetaStoreClient.java:690)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> 	at com.sun.proxy.$Proxy29.exchange_partitions(Unknown Source)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.exchangeTablePartitions(Hive.java:3546)
> 	... 21 more (state=08S01,code=1)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message