hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Udit Mehta <ume...@groupon.com>
Subject Issue in Insert Overwrite directory operation
Date Mon, 13 Jun 2016 23:54:57 GMT
Hi All,

I see a weird issue when trying to do a "INSERT OVERWRITE DIRECTORY"
operation. The query seems to work when I limit the data set but fails with
the following exception if the data set is larger:

Failed with exception Unable to move source
hdfs://namenode/user/grp_admin/external_test1/output/.hive-staging_hive_2016-06-13_21-34-36_449_7074605
to destination /user/grp_admin/external_test1/output

I ensured that the directory has enough space so there is no disk quota
issues here.
Does anyone know what is happening here?

Running Hive on Tez. Hive version is 1.2.1. Fails even with Hive on MR.

Run 1 with smaller data set:

    > insert overwrite directory
'/user/grp_admin/external_test1/output' row format delimited fields
terminated by '\t'

    > select * from test_table limit 1000;

Query ID = hive_20160613213624_d9d54ef0-0b28-4e98-b49e-197043f67c43

Total jobs = 3

Launching Job 1 out of 3





Status: Running (Executing on YARN cluster with App id
application_1464825277140_26149)



--------------------------------------------------------------------------------

        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

--------------------------------------------------------------------------------

Map 1 ..........   SUCCEEDED     12         12        0        0       0       0

Reducer 2 ......   SUCCEEDED      1          1        0        0       0       0

--------------------------------------------------------------------------------

VERTICES: 02/02  [==========================>>] 100%  ELAPSED TIME: 21.03 s

--------------------------------------------------------------------------------

Stage-4 is selected by condition resolver.

Stage-3 is filtered out by condition resolver.

Stage-5 is filtered out by condition resolver.

Moving data to:
hdfs://namenode/user/grp_admin/external_test1/output/.hive-staging_hive_2016-06-13_21-36-24_620_4270199609063911787-1/-ext-10000

Moving data to: /user/grp_admin/external_test1/output

OK

Time taken: 21.501 seconds

Run 2 with larger data set:

    > insert overwrite directory
'/user/grp_admin/external_test1/output' row format delimited fields
terminated by '\t'

    > select * from test_table;

Query ID = hive_20160613213436_a1b0087a-84ff-48a0-ac76-25811aaafe28

Total jobs = 3

Launching Job 1 out of 3

Tez session was closed. Reopening...

Session re-established.





Status: Running (Executing on YARN cluster with App id
application_1464825277140_26149)



--------------------------------------------------------------------------------

        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

--------------------------------------------------------------------------------

Map 1 ..........   SUCCEEDED     12         12        0        0       0       0

--------------------------------------------------------------------------------

VERTICES: 01/01  [==========================>>] 100%  ELAPSED TIME: 72.69 s

--------------------------------------------------------------------------------

Stage-4 is selected by condition resolver.

Stage-3 is filtered out by condition resolver.

Stage-5 is filtered out by condition resolver.

Moving data to:
hdfs://namenode/user/grp_admin/external_test1/output/.hive-staging_hive_2016-06-13_21-34-36_449_7074605303086037347-1/-ext-10000

Moving data to: /user/grp_admin/external_test1/output

Failed with exception Unable to move source
hdfs://namenode/user/grp_admin/external_test1/output/.hive-staging_hive_2016-06-13_21-34-36_449_7074605303086037347-1/-ext-10000/000000_0
to destination /user/grp_admin/external_test1/output

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask

Mime
View raw message