hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11388) there should only be 1 Initiator for compactions per Hive installation
Date Wed, 11 Nov 2015 20:05:11 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000995#comment-15000995
] 

Eugene Koifman commented on HIVE-11388:
---------------------------------------

A simpler way to deal with Cleaner: entries in COMPACTION_QUEUE have WORKER_ID which includes
hostname of the worker.  If cleaner runs on each metastore, we can make each Cleaner only
handle entries from Worker(s) from the same host.  This will ensure Cleaner don't step on
each other.

Side note, when metastores get restarted (perhaps even on different host), we have logic to
removeTimedoutWokers() so we can piggyback on this to ensure that Cleaner doesn't miss any
compactions (in the long run at least).  But also see HIVE-11685.

We also want to make sure AcidHouseKeeper instances don't step on each other.



> there should only be 1 Initiator for compactions per Hive installation
> ----------------------------------------------------------------------
>
>                 Key: HIVE-11388
>                 URL: https://issues.apache.org/jira/browse/HIVE-11388
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>
> org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs inside the metastore
service to manage compactions of ACID tables.  There should be exactly 1 instance of this
thread (even with multiple Thrift services).
> This is documented in https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration
but not enforced.
> Should add enforcement, since more than 1 Initiator could cause concurrent attempts to
compact the same table/partition - which will not work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message