hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Wilfong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-3106) Add option to make multi inserts more atomic
Date Mon, 11 Jun 2012 18:20:42 GMT

    [ https://issues.apache.org/jira/browse/HIVE-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292940#comment-13292940
] 

Kevin Wilfong commented on HIVE-3106:
-------------------------------------

Spoke with njain offline. He suggested adding a dummy task which depends on the tasks each
move task would depend on, and which has move tasks as its children. This will reduce the
number of dependency edges in the dependency graph. This dummy task (DependencyCollectionTask)
will only be added if this option is turned on.
                
> Add option to make multi inserts more atomic
> --------------------------------------------
>
>                 Key: HIVE-3106
>                 URL: https://issues.apache.org/jira/browse/HIVE-3106
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: Kevin Wilfong
>            Assignee: Kevin Wilfong
>         Attachments: HIVE-3106.1.patch.txt
>
>
> Currently, with multi-insert queries as soon the output of one of the inserts is ready
the move task associated with that insert is run, creating the table/partition.  However,
if concurrency is enabled the lock on this table/partition is not released until the entire
query finishes, which can be much later.
> This causes issues if, for example, a user is waiting for an output of the multi-insert
query which is created long before the other outputs, and checking for it's existence using
the metastore's Thrift methods (get_table/get_partition).  In which case, the user will run
their query which uses the output, and it will experience a timeout trying to acquire the
lock on the table/partition.
> If all the move tasks depend on the parent's of all other move tasks, the output creation
will be much closer to atomic relieving this problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message