hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nickSoul (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-19950) Hive ACID NOT LOCK LockComponent Correctly
Date Wed, 20 Jun 2018 17:53:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

nickSoul updated HIVE-19950:
----------------------------
    Description: 
Hi,

    When using Streaming Mutation recently, I found LockComponents were not locked correctly
by current transaction.  Below is my test case:

Step1: Begin a transaction with transactionId 126, and the transaction locks a table. Then
hangs the transaction. The lock information were correctly restored in mariaDB
{code:java}
MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS;
+----------------+----------------+----------+-----------+-----------+----
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID |
+----------------+----------------+----------+-----------+-----------+----
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL |
+----------------+----------------+----------+-----------+-----------+---- {code}
 

Step2: Begin the other transaction with a transactionId 127 before previous transaction 126
finished. Transaction 127 tries to lock the same table too, but failed at first attempt. The
lock information were correctly restored in mariaDB, Lock 385 was blocked by Lock 384. 
{code:java}
MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS;
+----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID |
+----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL |
| 385 | 1 | 127 | test_acid | acid_test | NULL | w | w | NULL | 384 | 1 |
+----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+{code}
 

Step3: Then transaction 127 tries to lock the table for a second retry after 30s with another
lockId: 386, this time it successfully locked the table,  whereas transaction 126 is still
holding the lock. Lock informations in MetaStore DB:
{code:java}
MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS; +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID | +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL | | 385
| 1 | 127 | test_acid | acid_test | NULL | w | w | NULL | 384 | 1 | | 386 | 1 | 127 | test_acid
| acid_test | NULL | a | w | 1529513069000 | NULL | NULL | +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
{code}
{code:java}
 {code}
    After going through the code.  I found it dosen't care about other transaction's lock
on the LockComponents in second retry.

    I wonder if i use it in a wrong way, or misunderstand sth about ACID in hive.

                                                           
                                                           
           Thanks

  was:
Hi,

When using Streaming Mutation recently, I found LockComponents were not locked correctly
by current transaction.  Below is my test case:

Step1: Begin a transaction with transactionId 126, and the transaction locks a table. Then
hangs the transaction. The lock information were correctly restored in mariaDB
{code:java}
MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS;
+----------------+----------------+----------+-----------+-----------+----
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID |
+----------------+----------------+----------+-----------+-----------+----
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL |
+----------------+----------------+----------+-----------+-----------+---- {code}
 

Step2: Begin the other transaction with a transactionId 127 before previous transaction 126
finished. Transaction 127 tries to lock the same table too, but failed at first attempt. The
lock information were correctly restored in mariaDB, Lock 385 was blocked by Lock 384. 
{code:java}
MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS;
+----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID |
+----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL |
| 385 | 1 | 127 | test_acid | acid_test | NULL | w | w | NULL | 384 | 1 |
+----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+{code}
 

Step3: Then transaction 127 tries to lock the table for a second retry after 30s with another
lockId: 386, this time it successfully locked the table,  whereas transaction 126 is still
holding the lock. Lock informations in MetaStore DB:
{code:java}
MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS; +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID | +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL | | 385
| 1 | 127 | test_acid | acid_test | NULL | w | w | NULL | 384 | 1 | | 386 | 1 | 127 | test_acid
| acid_test | NULL | a | w | 1529513069000 | NULL | NULL | +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
{code}
{code:java}
 {code}
    After going through the code.  I found it dosen't care about other transaction's lock
on the LockComponents in second retry.

    I wonder if i use it in a wrong way, or misunderstand sth about ACID in hive.

                                                           
                                                           
           Thanks


> Hive ACID NOT LOCK LockComponent Correctly
> ------------------------------------------
>
>                 Key: HIVE-19950
>                 URL: https://issues.apache.org/jira/browse/HIVE-19950
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 2.3.2
>            Reporter: nickSoul
>            Priority: Major
>
> Hi,
>     When using Streaming Mutation recently, I found LockComponents were not locked
correctly by current transaction.  Below is my test case:
> Step1: Begin a transaction with transactionId 126, and the transaction locks a table.
Then hangs the transaction. The lock information were correctly restored in mariaDB
> {code:java}
> MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS;
> +----------------+----------------+----------+-----------+-----------+----
> | HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID |
> +----------------+----------------+----------+-----------+-----------+----
> | 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL
|
> +----------------+----------------+----------+-----------+-----------+---- {code}
>  
> Step2: Begin the other transaction with a transactionId 127 before previous transaction
126 finished. Transaction 127 tries to lock the same table too, but failed at first attempt. The
lock information were correctly restored in mariaDB, Lock 385 was blocked by Lock 384. 
> {code:java}
> MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS;
> +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
> | HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID |
> +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
> | 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL
|
> | 385 | 1 | 127 | test_acid | acid_test | NULL | w | w | NULL | 384 | 1 |
> +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+{code}
>  
> Step3: Then transaction 127 tries to lock the table for a second retry after 30s with
another lockId: 386, this time it successfully locked the table,  whereas transaction 126
is still holding the lock. Lock informations in MetaStore DB:
> {code:java}
> MariaDB [hive]> select HL_LOCK_EXT_ID,HL_LOCK_INT_ID,HL_TXNID,HL_DB,HL_TABLE,HL_PARTITION,HL_LOCK_STATE,HL_LOCK_TYPE,HL_ACQUIRED_AT,HL_BLOCKEDBY_EXT_ID,HL_BLOCKEDBY_INT_ID
from HIVE_LOCKS; +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| HL_LOCK_EXT_ID | HL_LOCK_INT_ID | HL_TXNID | HL_DB | HL_TABLE | HL_PARTITION | HL_LOCK_STATE
| HL_LOCK_TYPE | HL_ACQUIRED_AT | HL_BLOCKEDBY_EXT_ID | HL_BLOCKEDBY_INT_ID | +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
| 384 | 1 | 126 | test_acid | acid_test | NULL | a | w | 1529512857000 | NULL | NULL | | 385
| 1 | 127 | test_acid | acid_test | NULL | w | w | NULL | 384 | 1 | | 386 | 1 | 127 | test_acid
| acid_test | NULL | a | w | 1529513069000 | NULL | NULL | +----------------+----------------+----------+-----------+-----------+--------------+---------------+--------------+----------------+---------------------+---------------------+
> {code}
> {code:java}
>  {code}
>     After going through the code.  I found it dosen't care about other transaction's
lock on the LockComponents in second retry.
>     I wonder if i use it in a wrong way, or misunderstand sth about ACID in hive.
>                                                          
                                                           
             Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message