hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject RE: locks are held on tables even when no job running
Date Fri, 23 Oct 2015 08:18:23 GMT
Hi Furcy,

 

Thanks for the info.

 

I ran the same job twice, killing it first time and starting again. Actually your point about
5 min duration seems to be correct. my process basically creates a new hive table with two
additional columns and populate it from an existing table hence the locks

 

Even if the job is killed “WAITING EXCLUSIVE” locks are still held on the new table (see
below) and that is the cause of the issue

 

Lock ID Database        Table   Partition       State   Type    Transaction ID  Last Hearbeat
  Acquired At     User    Hostname

14031   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL    1445586539232   NULL
   hduser  rhes564

14029   asehadoop       t_staging       NULL    ACQUIRED        SHARED_READ     NULL    1445586247044
  1445585940653   hduser  rhes564

14029   asehadoop       t       NULL    ACQUIRED        SHARED_READ     NULL    1445586247044
  1445585940654   hduser  rhes564

14030   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL    1445586471827   NULL
   hduser  rhes564

 

I am not sure this is behaving like classic RDBMS like Sybase or MSSQL where the rollback
is happening after KILL command and the locks are held until rollback is complete. Killing
a process itself will not release the locks!

 

Regards,

 

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.


co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This message is for the
designated recipient only, if you are not the intended recipient, you should destroy it immediately.
Any information in this message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility
of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd,
its subsidiaries nor their employees accept any responsibility.

 

From: Furcy Pin [mailto:furcy.pin@flaminem.com] 
Sent: 23 October 2015 09:08
To: user@hive.apache.org
Subject: Re: locks are held on tables even when no job running

 

Hi Mich,

 

I believe the duration of locks is defined by hive.txn.timeout, which is 5 min by default.

https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

 

Retry your SHOW LOCKS command and check that the Last HeartBeat is not changing.

If it is, it means your query is still active somehow. If it isn't, the lock should disappear
by itself after the timeout.

 

Also, I don't know if this is a known issue in Hive's Jira, but we noticed that trying to
read and write data

into the same table within a single query creates a deadlock.

 

If that is what you were trying to do, you should rather write your data in a temporary file
and then move it back into the table.

 

 

 

 

 

 

 

 

On Fri, Oct 23, 2015 at 9:56 AM, Mich Talebzadeh <mich@peridale.co.uk <mailto:mich@peridale.co.uk>
> wrote:

Hi,

 

 

What is the duration of locks held in Hive?

 

I have got the following locks in Hive, although I have already killed the jobs!

 

 

Lock ID Database        Table   Partition       State   Type    Transaction ID  Last Hearbeat
  Acquired At     User    Hostname

14031   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL    1445586539232   NULL
   hduser  rhes564

14029   asehadoop       t_staging       NULL    ACQUIRED        SHARED_READ     NULL    1445586247044
  1445585940653   hduser  rhes564

14029   asehadoop       t       NULL    ACQUIRED        SHARED_READ     NULL    1445586247044
  1445585940654   hduser  rhes564

14030   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL    1445586471827   NULL
   hduser  rhes564

 

 

mapred job -list

Total jobs:0

                  JobId      State           StartTime      UserName           Queue     
Priority       UsedContainers  RsvdContainers  UsedMem         RsvdMem         NeededMem 
       AM info

 

No locks are held in metastore (Oracle in my case) as well.

 

Thanks

 

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7.


co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This message is for the
designated recipient only, if you are not the intended recipient, you should destroy it immediately.
Any information in this message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility
of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd,
its subsidiaries nor their employees accept any responsibility.

 

 


Mime
View raw message