hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-3477) Duplicate data possible with speculative execution for dynamic partitions
Date Tue, 18 Sep 2012 11:24:08 GMT
Namit Jain created HIVE-3477:
--------------------------------

             Summary: Duplicate data possible with speculative execution for dynamic partitions
                 Key: HIVE-3477
                 URL: https://issues.apache.org/jira/browse/HIVE-3477
             Project: Hive
          Issue Type: Bug
          Components: Query Processor
            Reporter: Namit Jain
            Assignee: Namit Jain


Consider a query like:

insert overwrite T partition (ds)
select * from
(mapreduce-subq1
  union all
mapreduce-subq2)x;

Once, mapreduce-subq1 and mapreduce-subq2 are done, the task for the union
is invoked. At the end of the union task, jobClose is invoked.

Note that there are 2 tablescan operators. The tree is something like:


TABLESCAN1  --
              \
               UNION -- SELECT -- FILESINK
              /
TABLESCAN2  --


In the current setup, jobClose will be invoked twice for FileSink.
In case of speculative execution, it is possible that data is still is
being written to tmp Dir. after jobClose is finished once. 

The correct fix would be to make sure that jobClose is only invoked once.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message