hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <>
Subject [jira] [Commented] (HIVE-12353) When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.
Date Thu, 21 Jan 2016 04:12:39 GMT


Eugene Koifman commented on HIVE-12353:

bq. info.highestTxnId = rs.wasNull() ? null : highestTxnId;
ResultSet.getLong() returns 0 when value was actually NULL so I thought it would be more consistent
with other places in the code that use 0 as invalid txn id

bq. PreparedStatement
cc_meta_info is a binary field - I don't think there is way to set it w/o prepared statement.
 (Even though it isn't set yet but will be soon)

bq. A concern on the history stuff
my thought here was that if I'm looking at 10 failed compactions, it would be useful for me
to know when the last successful run was.  Allowing retention value to be set per type allows

bq. checkFailedCompactions
The idea in this loop is to look at the last "failedThreshold" elements.  If they are all
"failed" - don't start another one.  If not all "failed" - that's the intervening success.

bq. It also seems users could unwittingly mess things
Yes, that is why there is CompactionTxnHandler.getFailedCompactionRetention() to make sure
this doesn't happen

> When Compactor fails it calls CompactionTxnHandler.markedCleaned().  it should not.
> -----------------------------------------------------------------------------------
>                 Key: HIVE-12353
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>            Priority: Blocker
>         Attachments: HIVE-12353.2.patch, HIVE-12353.3.patch, HIVE-12353.4.patch, HIVE-12353.6.patch,
HIVE-12353.7.patch, HIVE-12353.patch
> One of the things that this method does is delete entries from TXN_COMPONENTS for partition
that it was trying to compact.
> This causes Aborted transactions in TXNS to become empty according to
> CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be deleted.  
> Once they are deleted, data that belongs to these txns is deemed committed...
> We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) states.  We
should also not delete then entry from markedCleaned()
> We'll have separate process that cleans 'f' and 's' records after X minutes (or after
> N records for a given partition exist).
> This allows SHOW COMPACTIONS to show some history info and how many times compaction
failed on a given partition (subject to retention interval) so that we don't have to call
markCleaned() on Compactor failures at the same time preventing Compactor to constantly getting
stuck on the same bad partition/table.
> Ideally we'd want to include END_TIME field.

This message was sent by Atlassian JIRA

View raw message