hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darren Yin <darren....@gmail.com>
Subject MoveTasks releasing locks that don't belong to it?
Date Wed, 23 Jul 2014 20:27:13 GMT
In releaseLocks in MoveTask
<https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java>,
it looks like the lockMgr.getLocks line actually grabs all locks associated
with whatever lock objects (e.g., a partition) the MoveTask is concerned
with.

My theory for what can happen: A MoveTask starts writing to a partition,
taking some of its own locks. While it's running, other hive jobs put locks
on the same partition or table. Then, when the MoveTask ends, it finds all
locks associated with that partition and releases them all, and then the
other hive jobs who originally put those locks there run into NoNode
KeeperExceptions (using Zookeeper). I'm actually on a modified build of
Hive 0.11, but from reading the code, it appears that the issue still
exists in trunk as well.

Has anyone else run into this sort of issue?

Thanks!
--Darren

Mime
View raw message