hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
Date Mon, 13 Jul 2020 14:24:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23832?focusedWorklogId=458080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-458080
]

ASF GitHub Bot logged work on HIVE-23832:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 13/Jul/20 14:23
            Start Date: 13/Jul/20 14:23
    Worklog Time Spent: 10m 
      Work Description: deniskuzZ commented on a change in pull request #1243:
URL: https://github.com/apache/hive/pull/1243#discussion_r453686774



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/compact/AlterTableCompactOperation.java
##########
@@ -99,7 +99,7 @@ private void waitForCompactionToFinish(CompactionResponse resp) throws HiveExcep
     wait: while (true) {
       //double wait time until 5min
       waitTimeMs = waitTimeMs*2;
-      waitTimeMs = Math.max(waitTimeMs, waitTimeOut);
+      waitTimeMs = Math.min(waitTimeMs, waitTimeOut);

Review comment:
       default is 300000ms (5 min) that's max, min = waitTimeMs = 1000 X 2 = 2000ms;




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 458080)
    Time Spent: 40m  (was: 0.5h)

> Compaction cleaner fails to clean up deltas when using blocking compaction
> --------------------------------------------------------------------------
>
>                 Key: HIVE-23832
>                 URL: https://issues.apache.org/jira/browse/HIVE-23832
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Denys Kuzmenko
>            Assignee: Denys Kuzmenko
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> {code}
> CREATE TABLE default.compcleanup (
>    cda_id             int,
>    cda_run_id         varchar(255),
>    cda_load_ts        timestamp,
>    global_party_id    string,
>    group_id           string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>    cda_date           int,
>    cda_job_name       varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601, cda_job_name='core_base')
COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the presence of open txn
(by `ALTER TABLE`) below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([]) pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_0000002_v0000035
> {code}
> AcidUtils.processBaseDir
> {code}
> if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted, validTxnList)) {
>    return;
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message