hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11388) there should only be 1 Initiator for compactions per Hive installation
Date Mon, 21 Dec 2015 18:43:46 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066851#comment-15066851
] 

Alan Gates commented on HIVE-11388:
-----------------------------------

I would suggest adding hostname to the MUTEX_TABLE so that admins can easily figure out which
instance of metastore is running a given thread.

But what happens when that metastore dies?  How does one of the existing metastores know to
start an initiator?  The table will also need a 'last_hearbeat' column that is updated each
time the initiator runs.  Other initiator threads will need to spin checking this every minute
or so to see if the previous initiator has died, and if so take over.  If the first metastore
then heartbeats again it would have to discover it no longer holds the lock and kill its initiator
thread.

> there should only be 1 Initiator for compactions per Hive installation
> ----------------------------------------------------------------------
>
>                 Key: HIVE-11388
>                 URL: https://issues.apache.org/jira/browse/HIVE-11388
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>            Priority: Critical
>
> org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs inside the metastore
service to manage compactions of ACID tables.  There should be exactly 1 instance of this
thread (even with multiple Thrift services).
> This is documented in https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
but not enforced.
> Should add enforcement, since more than 1 Initiator could cause concurrent attempts to
compact the same table/partition - which will not work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message