hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward Capriolo (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1293) Concurreny Model for Hive
Date Wed, 05 May 2010 19:06:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864450#action_12864450
] 

Edward Capriolo commented on HIVE-1293:
---------------------------------------

I see you have made mention to zookeeper. Do we really need zookeeper to implement locking
when we already have a transactional MetaStore via JPOX (Derby/MySQL)? From my prospecting
on the management side and from a development side having to integrate and understand the
Zookeeper API for simple low performance locking seems overkill.  

For example, I understand that zookeeper has facilities to auto-expire old locks and other
features that make it seem ideal in a use case, but I think the visibility into it is rather
low. If our locks were implemented in the meta-store, clearing them would be easy to do thought
hive commands or with JDBC. What tools would we have to investigate Hive-Zookeeper-Locking?


I say this because of experience with hbase-zookeeper. Other the 'hbase status' zookeeper
is a total black box and makes troubleshooting that component of hbase difficult. I would
not want to see hive go down the same road. 

No matter which implementation we go with we should have  DDL commands like 'show locks' 'clear
locks'. 

> Concurreny Model for Hive
> -------------------------
>
>                 Key: HIVE-1293
>                 URL: https://issues.apache.org/jira/browse/HIVE-1293
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Namit Jain
>
> Concurrency model for Hive:
> Currently, hive does not provide a good concurrency model. The only guanrantee provided
in case of concurrent readers and writers is that
> reader will not see partial data from the old version (before the write) and partial
data from the new version (after the write).
> This has come across as a big problem, specially for background processes performing
maintenance operations.
> The following possible solutions come to mind.
> 1. Locks: Acquire read/write locks - they can be acquired at the beginning of the query
or the write locks can be delayed till move
> task (when the directory is actually moved). Care needs to be taken for deadlocks.
> 2. Versioning: The writer can create a new version if the current version is being read.
Note that, it is not equivalent to snapshots,
> the old version can only be accessed by the current readers, and will be deleted when
all of them have finished.
> Comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message