hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] Created: (HIVE-1557) increase concurrency
Date Wed, 18 Aug 2010 17:16:36 GMT
increase concurrency
--------------------

                 Key: HIVE-1557
                 URL: https://issues.apache.org/jira/browse/HIVE-1557
             Project: Hadoop Hive
          Issue Type: Improvement
          Components: Query Processor
            Reporter: Namit Jain


Copying Joy's comment from https://issues.apache.org/jira/browse/HIVE-1293

a little bummed that locks need to be held for entire query execution. that could mean a writer
blocking readers for hours.
hive's query plans seem to be of two distinct stages:
1. read a bunch of stuff, compute intermediate/final data
2. move final data into output locations

ie. - a single query never reads what it writes (into a final output location). even if #1
and #2 are mingled today - they can easily be put in order.

in that sense - we only need to get shared locks for all read entities involved in #1 to begin
with. once phase #1 is done, we can drop all the read locks and get the exclusive locks for
all the write entities in #2, perform #2 and quit. that way exclusive locks are held for a
very short duration. i think this scheme is similarly deadlock free (now there are two independent
lock acquire/release phases - and each of them can lock stuff in lex. order).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message