hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <>
Subject [jira] [Work logged] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
Date Mon, 05 Oct 2020 09:09:00 GMT


ASF GitHub Bot logged work on HIVE-21052:

                Author: ASF GitHub Bot
            Created on: 05/Oct/20 09:08
            Start Date: 05/Oct/20 09:08
    Worklog Time Spent: 10m 
      Work Description: pvargacl commented on a change in pull request #1548:

File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/
@@ -97,9 +100,9 @@ public void run() {
           long minOpenTxnId = txnHandler.findMinOpenTxnIdForCleaner();
 "Cleaning based on min open txn id: " + minOpenTxnId);
           List<CompletableFuture> cleanerList = new ArrayList<>();
-          for(CompactionInfo compactionInfo : txnHandler.findReadyToClean()) {
+          for (CompactionInfo compactionInfo : txnHandler.findReadyToClean()) {
-                    clean(compactionInfo, minOpenTxnId)), cleanerExecutor));
+                  clean(compactionInfo, minOpenTxnId)), cleanerExecutor));

Review comment:
       Two questions here:
   1. In the original Jira there was discussion about not allowing concurrent cleanings of
the same stuff (partition / table). Should we worry about this?
   2. The slow cleanAborted will clog the executor service, we should do something about this,
either in this patch, or follow up something like
immediately after this.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 495221)
    Time Spent: 2h 20m  (was: 2h 10m)

> Make sure transactions get cleaned if they are aborted before addPartitions is called
> -------------------------------------------------------------------------------------
>                 Key: HIVE-21052
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 3.0.0, 3.1.1
>            Reporter: Jaume M
>            Assignee: Jaume M
>            Priority: Critical
>              Labels: pull-request-available
>         Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, HIVE-21052.10.patch,
HIVE-21052.11.patch, HIVE-21052.12.patch, HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, HIVE-21052.8.patch, HIVE-21052.9.patch
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
> If the transaction is aborted between openTxn and addPartitions and data has been written
on the table the transaction manager will think it's an empty transaction and no cleaning
will be done.
> This is currently an issue in the streaming API and in micromanaged tables. As proposed
by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and when addPartitions
is called remove this entry from TXN_COMPONENTS and add the corresponding partition entry
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that specifies
that a transaction was opened and it was aborted it must generate jobs for the worker for
every possible partition available.
> cc [~ewohlstadter]

This message was sent by Atlassian Jira

View raw message