hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject RE: locks are held on tables even when no job running
Date Wed, 28 Oct 2015 10:18:54 GMT
An interesting one I noted is that even after killing the process locks are
still held for a long time

 

First kill the job

 

/home/hduser/hadoop-2.6.0/bin/hadoop job  -kill job_1445590859106_0003

 

INFO impl.YarnClientImpl: Killed application application_1445590859106_0003

 

Locks are still held

 

Lock ID Database        Table   Partition       State   Type    Transaction
ID  Last Hearbeat   Acquired At     User    Hostname

14048   asehadoop       t_staging       NULL    ACQUIRED        SHARED_READ
NULL    1446024486900   1446020114548   hduser  rhes564

14048   asehadoop       t       NULL    ACQUIRED        SHARED_READ     NULL
1446024486900   1446020114549   hduser  rhes564

 

And If I look into table HIVE_LOCKS in my Oracle metadata, I see that they
are being held for a long time after killing the process

 

1* select to_char(sysdate,'HH:MI:SS') AS "Timing", HL_TABLE, HL_LOCK_STATE,
HL_LOCK_TYPE, HL_LAST_HEARTBEAT, HL_ACQUIRED_AT from hiveuser.HIVE_LOCKS;

 

Timing   HL_TABLE        HL_LOCK_STATE   HL_LOCK_TYPE    HL_LAST_HEARTBEAT
HL_ACQUIRED_AT

-------- --------------- --------------- --------------- -----------------
--------------

09:53:26 t_staging       a               r                      1.4460E+12
1.4460E+12

09:53:26 t               a               r                      1.4460E+12
1.4460E+12

 

Timing   HL_TABLE        HL_LOCK_STATE   HL_LOCK_TYPE    HL_LAST_HEARTBEAT
HL_ACQUIRED_AT

-------- --------------- --------------- --------------- -----------------
--------------

10:09:53 t_staging       a               r                      1.4460E+12
1.4460E+12

10:09:53 t               a               r                      1.4460E+12
1.4460E+12

 

 

Which is more than 16 minutes

 

The only way I can get rid of the locks is to drop table t (see email below
for the code)

 

drop table if exists t;

OK

Time taken: 0.119 seconds

hive>  desc t;

FAILED: SemanticException [Error 10001]: Table not found t

 

 

select to_char(sysdate,'HH:MI:SS') AS "Timing", HL_TABLE, HL_LOCK_STATE,
HL_LOCK_TYPE, HL_LAST_HEARTBEAT, HL_ACQUIRED_AT from hiveuser.HIVE_LOCKS;

 

no rows selected

 

 

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf

Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.

 

From: Mich Talebzadeh [mailto:mich@peridale.co.uk] 
Sent: 23 October 2015 22:37
To: user@hive.apache.org
Subject: RE: locks are held on tables even when no job running

 

Hi Eugene,

 

The code drops the table if exists and that is the exclusive lock. Once
created it is populated from another

 

use asehadoop;

drop table if exists t;

create table t (

owner                   varchar(30)

,object_name             varchar(30)

,subobject_name          varchar(30)

,object_id               bigint

,data_object_id          bigint

,object_type             varchar(19)

,created                 timestamp

,last_ddl_time           timestamp

,timestamp               varchar(19)

,status                  varchar(7)

,temporary2              varchar(1)

,generated               varchar(1)

,secondary               varchar(1)

,namespace               bigint

,edition_name            varchar(30)

,padding1                varchar(4000)

,padding2                varchar(3500)

,attribute               varchar(32)

,op_type                 int

,op_time                 timestamp

)

clustered by (object_id) into 256 buckets

;

INSERT INTO TABLE t

SELECT

          owner

        , object_name

        , subobject_name

        , object_id

        , data_object_id

        , object_type

        , cast(created AS timestamp)

        , cast(last_ddl_time AS timestamp)

        , timestamp

        , status

        , temporary2

        , generated

        , secondary

        , namespace

        , edition_name

        , padding1

        , padding2

        , attribute

        , 1

        , cast(from_unixtime(unix_timestamp()) AS timestamp)

FROM t_staging

;

 

I killed the program using Ctrl C and then used mapred job as below to kill

 

mapred job  -kill job_1445590859106_0002

Killed job job_1445590859106_0002

 

 

Still locks were held

 

HTH

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf

Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.

 

From: Eugene Koifman [mailto:ekoifman@hortonworks.com] 
Sent: 23 October 2015 17:13
To: user@hive.apache.org <mailto:user@hive.apache.org> 
Subject: Re: locks are held on tables even when no job running

 

Mich,

how were you running/killing the job?  was it ^C of CLI or something else? 

(The only time you'd get Exclusive lock is to drop an object.  (With
DbTxnManager which looks like what you are using))

The locks will timeout but https://issues.apache.org/jira/browse/HIVE-11317
may be relevant.

 

Furcy,

could you file a Jira with a repro for the deadlock you are describing?

 

Thanks,

Eugene

 

 

 

From: Mich Talebzadeh <mich@peridale.co.uk <mailto:mich@peridale.co.uk> >
Reply-To: "user@hive.apache.org <mailto:user@hive.apache.org> "
<user@hive.apache.org <mailto:user@hive.apache.org> >
Date: Friday, October 23, 2015 at 1:18 AM
To: "user@hive.apache.org <mailto:user@hive.apache.org> "
<user@hive.apache.org <mailto:user@hive.apache.org> >
Subject: RE: locks are held on tables even when no job running

 

Hi Furcy,

 

Thanks for the info.

 

I ran the same job twice, killing it first time and starting again. Actually
your point about 5 min duration seems to be correct. my process basically
creates a new hive table with two additional columns and populate it from an
existing table hence the locks

 

Even if the job is killed "WAITING EXCLUSIVE" locks are still held on the
new table (see below) and that is the cause of the issue

 

Lock ID Database        Table   Partition       State   Type    Transaction
ID  Last Hearbeat   Acquired At     User    Hostname

14031   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL
1445586539232   NULL    hduser  rhes564

14029   asehadoop       t_staging       NULL    ACQUIRED        SHARED_READ
NULL    1445586247044   1445585940653   hduser  rhes564

14029   asehadoop       t       NULL    ACQUIRED        SHARED_READ     NULL
1445586247044   1445585940654   hduser  rhes564

14030   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL
1445586471827   NULL    hduser  rhes564

 

I am not sure this is behaving like classic RDBMS like Sybase or MSSQL where
the rollback is happening after KILL command and the locks are held until
rollback is complete. Killing a process itself will not release the locks!

 

Regards,

 

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf

Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.

 

From: Furcy Pin [mailto:furcy.pin@flaminem.com] 
Sent: 23 October 2015 09:08
To: user@hive.apache.org <mailto:user@hive.apache.org> 
Subject: Re: locks are held on tables even when no job running

 

Hi Mich,

 

I believe the duration of locks is defined by hive.txn.timeout, which is 5
min by default.

https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

 

Retry your SHOW LOCKS command and check that the Last HeartBeat is not
changing.

If it is, it means your query is still active somehow. If it isn't, the lock
should disappear by itself after the timeout.

 

Also, I don't know if this is a known issue in Hive's Jira, but we noticed
that trying to read and write data

into the same table within a single query creates a deadlock.

 

If that is what you were trying to do, you should rather write your data in
a temporary file and then move it back into the table.

 

 

 

 

 

 

 

 

On Fri, Oct 23, 2015 at 9:56 AM, Mich Talebzadeh <mich@peridale.co.uk
<mailto:mich@peridale.co.uk> > wrote:

Hi,

 

 

What is the duration of locks held in Hive?

 

I have got the following locks in Hive, although I have already killed the
jobs!

 

 

Lock ID Database        Table   Partition       State   Type    Transaction
ID  Last Hearbeat   Acquired At     User    Hostname

14031   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL
1445586539232   NULL    hduser  rhes564

14029   asehadoop       t_staging       NULL    ACQUIRED        SHARED_READ
NULL    1445586247044   1445585940653   hduser  rhes564

14029   asehadoop       t       NULL    ACQUIRED        SHARED_READ     NULL
1445586247044   1445585940654   hduser  rhes564

14030   asehadoop       t       NULL    WAITING EXCLUSIVE       NULL
1445586471827   NULL    hduser  rhes564

 

 

mapred job -list

Total jobs:0

                  JobId      State           StartTime      UserName
Queue      Priority       UsedContainers  RsvdContainers  UsedMem
RsvdMem         NeededMem         AM info

 

No locks are held in metastore (Oracle in my case) as well.

 

Thanks

 

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf

Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly

 

http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> 

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.

 

 


Mime
View raw message