hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Sichi (JIRA)" <>
Subject [jira] Commented: (HIVE-1293) Concurreny Model for Hive
Date Wed, 05 May 2010 19:17:11 GMT


John Sichi commented on HIVE-1293:

Regarding the implementation of lock acquisition/release in terms of zookeeper, can you elaborate
on how you are proposing to map the lock hierarchy into the znode hierarchy?

It sounds like the znode paths will correspond to database.table.partition.subpartition...
paths, which is good.

However, does the lock acquisition recipe actually need to reflect the hierarchy, which I
think is what you are proposing?  In other words, can't we just come up with a flat list of
object locks to take (including both table-level and partition-level), sort them, and then
acquire them each independently using the non-hierarchical recipe (except as you mention with
failfast instead of wait)?  If any fail, then delete them all before re-entering the retry

Assuming the sorting matches the compound naming scheme, this should guarantee hierarchical
lock acquisition order within each table.

Also, I do not understand the part below.

"The 'X' lock for table T is specified as follows:
# For all parent znodes of T, call getChildren() without setting the watch flag."

Do you mean "for the parent znode of T" rather than "all parent znodes of T", and this is
supposed to apply for case where T is actually a partition?

> Concurreny Model for Hive
> -------------------------
>                 Key: HIVE-1293
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Namit Jain
> Concurrency model for Hive:
> Currently, hive does not provide a good concurrency model. The only guanrantee provided
in case of concurrent readers and writers is that
> reader will not see partial data from the old version (before the write) and partial
data from the new version (after the write).
> This has come across as a big problem, specially for background processes performing
maintenance operations.
> The following possible solutions come to mind.
> 1. Locks: Acquire read/write locks - they can be acquired at the beginning of the query
or the write locks can be delayed till move
> task (when the directory is actually moved). Care needs to be taken for deadlocks.
> 2. Versioning: The writer can create a new version if the current version is being read.
Note that, it is not equivalent to snapshots,
> the old version can only be accessed by the current readers, and will be deleted when
all of them have finished.
> Comments.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message