hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Moore, Douglas" <Douglas.Mo...@thinkbiganalytics.com>
Subject Re: Unable to move files on Hive/Hdfs
Date Mon, 04 May 2015 23:12:24 GMT
Yep, permission problem. Weird though it seems to be moving a file within the same dir.
Thanks for the update!
- Douglas
From: amit kumar <ak3716@gmail.com<mailto:ak3716@gmail.com>>
Reply-To: <user@hive.apache.org<mailto:user@hive.apache.org>>
Date: Tue, 5 May 2015 04:40:18 +0530
To: <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Re: Unable to move files on Hive/Hdfs

Hi Doug,

I have use CDH 5.2.1

I performed the below task, and getting the error, but after rolling back the below changes
issue has been resolved itself.

Disable ACLs on Name Nodes
Set Enable Access Control Lists = False
Save Changes
Restart Hadoop Cluster


Thanks,


On Tue, May 5, 2015 at 4:36 AM, amit kumar <ak3716@gmail.com<mailto:ak3716@gmail.com>>
wrote:
Hi Doug,

I have use CDH 5.2.1

Disable ACLs on Name Nodes

Set Enable Access Control Lists = False
Save Changes
Restart Hadoop Cluster


Stack trace:
2015-05-04 10:38:18,820 INFO  [main]: exec.Task (SessionState.java:printInfo(537)) - Moving
data to: hdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-37-43_010_529731467724830376-1/-ext-10000
from hdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-37-43_010_529731467724830376-1/-ext-10002
2015-05-04 10:38:18,857 ERROR [main]: exec.Task (SessionState.java:printError(546)) - Failed
with exception Unable to move sourcehdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-37-43_010_529731467724830376-1/-ext-10002
to destination hdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-37-43_010_529731467724830376-1/-ext-10000
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move sourcehdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-37-43_010_529731467724830376-1/-ext-10002
to destination hdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-37-43_010_529731467724830376-1/-ext-10000
        at org.apache.hadoop.hive.ql.metadata.Hive.renameFile(Hive.java:2269)
        at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:89)
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:200)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1516)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1283)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:924)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:914)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:269)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:221)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:431)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:367)
        at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:464)
        at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:474)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.hdfs.protocol.AclException: The ACL operation has been rejected.
 Support for ACLs has been disabled by setting dfs.namenode.acls.enabled to false.
        at org.apache.hadoop.hdfs.server.namenode.NNConf.checkAclsConfigFlag(NNConf.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAclStatus(FSNamesystem.java:8553)


After rolling those same changes out, the problem resolved itself.


On Tue, May 5, 2015 at 4:28 AM, Moore, Douglas <Douglas.Moore@thinkbiganalytics.com<mailto:Douglas.Moore@thinkbiganalytics.com>>
wrote:
Hi Amit,

We've seen the same error on MoveTask with Hive 0.14 / HDP 2.2 release. There are lots of
reasons for this though.
Can you provide more details about the stack trace and version so we can compare?

For our problem we've seen some relief with SET hive.metastore.client.socket.timeout=60s
but the problem still happens from time to time.

Thanks,
Douglas

From: amit kumar <ak3716@gmail.com<mailto:ak3716@gmail.com>>
Reply-To: <user@hive.apache.org<mailto:user@hive.apache.org>>
Date: Tue, 5 May 2015 03:12:15 +0530
To: <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Unable to move files on Hive/Hdfs


While moving the data from hive/hdfs we get below error,

Please suggest on this.

Moving data to: hdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-02-39_841_5305383954203911235-1/-ext-10000
Failed with exception Unable to move sourcehdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-02-39_841_5305383954203911\
235-1/-ext-10002 to destination hdfs://nameservice1/tmp/hive-srv-hdp-edh-d/hive_2015-05-04_10-02-39_841_5305383954203911235-1/-ext-\
10000
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 5.83 sec HDFS Read: 553081 HDFS Write: 489704 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 830 msec
Error (1). Execution Failed.
2015-05-04 10:03:13 ERROR (1) in run_hive

Thanks,



Mime
View raw message