Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Sat, 1 Apr 2017 18:23:41 +0000 (UTC)
From: "Sahil Takiar (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13058060.1490144743000.184586.1491071021564@Atlassian.JIRA>
In-Reply-To: <JIRA.13058060.1490144743000@Atlassian.JIRA>
References: <JIRA.13058060.1490144743000@Atlassian.JIRA> <JIRA.13058060.1490144743302@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HIVE-16277) Exchange Partition between
 filesystems throws "IllegalArgumentException Wrong FS"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 01 Apr 2017 18:23:46 -0000


    [ https://issues.apache.org/jira/browse/HIVE-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952341#comment-15952341 ] 

Sahil Takiar commented on HIVE-16277:
-------------------------------------

[~vihangk1], [~spena], [~mohitsabharwal] anyone chance someone could take a look at this? Right now, I'm looking for some feedback on the approach. I still have some code cleanup + unit testing to do.

I'm trying to add support for exchanging of partitions across filesystems. I introduced a new HiveMetaStore method call {{exchange_partitions_metadata}} which only exchanges the partition metadata in HMS, but doesn't actually move the data. This is different from the existing {{exchange_partitions}} method in HMS which both renames the partition on the physical filesystem and switches the metadata.

I wanted to move the actual renaming of directories to {{Hive.java}} - in the case where a folder needs to be moved cross-filesystem, the partition data needs to be copied. Doing this in HMS doesn't sound like the right approach, as copying the data could take hours, depending on the size of the partition.

> Exchange Partition between filesystems throws "IllegalArgumentException Wrong FS"
> ---------------------------------------------------------------------------------
>
>                 Key: HIVE-16277
>                 URL: https://issues.apache.org/jira/browse/HIVE-16277
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-16277.1.patch, HIVE-16277.2.patch, HIVE-16277.3.patch, HIVE-16277.4.patch
>
>
> The following query: {{alter table s3_tbl exchange partition (country='USA') with table hdfs_tbl}} fails with the following exception:
> {code}
> Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.lang.IllegalArgumentException Wrong FS: s3a://[bucket]/table/country=USA, expected: file:///)
> 	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:379)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
> 	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:347)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:361)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.lang.IllegalArgumentException Wrong FS: s3a://[bucket]/table/country=USA, expected: file:///)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.exchangeTablePartitions(Hive.java:3553)
> 	at org.apache.hadoop.hive.ql.exec.DDLTask.exchangeTablePartition(DDLTask.java:4691)
> 	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:570)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2182)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1838)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1525)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1231)
> 	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> 	... 11 more
> Caused by: MetaException(message:Got exception: java.lang.IllegalArgumentException Wrong FS: s3a://[bucket]/table/country=USA, expected: file:///)
> 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1387)
> 	at org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:208)
> 	at org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:200)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.exchange_partitions(HiveMetaStore.java:2967)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> 	at com.sun.proxy.$Proxy28.exchange_partitions(Unknown Source)
> 	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.exchange_partitions(HiveMetaStoreClient.java:690)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> 	at com.sun.proxy.$Proxy29.exchange_partitions(Unknown Source)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.exchangeTablePartitions(Hive.java:3546)
> 	... 21 more (state=08S01,code=1)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)