hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <>
Subject Re: MoveTasks releasing locks that don't belong to it?
Date Wed, 23 Jul 2014 20:47:33 GMT
For what it is worth I run with locks off. I played with it in versions
0.8x -> 0.10. I found them problematic particularly from hive-server. We
ended up doing our own locking application side. I am very suprised that
some vendors "distributions" suggest that this is better/safer "on" when I
have found the opposite to be true.

Do not get me wrong the feature works and we used to for a while, but
periodically things would pop up around this feature that would
log-jam/break some etl process and the reward was not worth the drawbacks.

On Wed, Jul 23, 2014 at 4:27 PM, Darren Yin <> wrote:

> In releaseLocks in MoveTask
> <>,
> it looks like the lockMgr.getLocks line actually grabs all locks associated
> with whatever lock objects (e.g., a partition) the MoveTask is concerned
> with.
> My theory for what can happen: A MoveTask starts writing to a partition,
> taking some of its own locks. While it's running, other hive jobs put locks
> on the same partition or table. Then, when the MoveTask ends, it finds all
> locks associated with that partition and releases them all, and then the
> other hive jobs who originally put those locks there run into NoNode
> KeeperExceptions (using Zookeeper). I'm actually on a modified build of
> Hive 0.11, but from reading the code, it appears that the issue still
> exists in trunk as well.
> Has anyone else run into this sort of issue?
> Thanks!
> --Darren

View raw message