hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Denys Kuzmenko (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-23832) Compaction cleaner fails to clean up deltas when using blocking compaction
Date Fri, 10 Jul 2020 07:50:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Denys Kuzmenko updated HIVE-23832:
----------------------------------
    Description: 
{code}
CREATE TABLE default.compcleanup (
   cda_id             int,
   cda_run_id         varchar(255),
   cda_load_ts        timestamp,
   global_party_id    string,
   group_id           string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
   cda_date           int,
   cda_job_name       varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601, cda_job_name='core_base') COMPACT
'MAJOR' AND WAIT;
{code}

When using blocking compaction Cleaner skips processing due to the open txn by `ALTER TABLE`
below Compactor's one.

{code}
AcidUtils - getChildState() ignoring([]) pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_0000002_v0000035
{code}

{code}
    if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted, validTxnList)) {
      return;
    }
{code}

  was:
{code}
CREATE TABLE default.compcleanup (
   cda_id             int,
   cda_run_id         varchar(255),
   cda_load_ts        timestamp,
   global_party_id    string,
   group_id           string)
COMMENT 'gp_2_gr'
PARTITIONED BY (
   cda_date           int,
   cda_job_name       varchar(12))
STORED AS ORC;
-- cda_date=20200601/cda_job_name=core_base
INSERT INTO default.compcleanup VALUES (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
ALTER TABLE default.compcleanup PARTITION (cda_date=20200601, cda_job_name='core_base') COMPACT
'MAJOR' AND WAIT;
{code}

When using blocking compaction Cleaner skips processing due to the open txn (by ALTER TABLE)
below Compactor's one.

{code}
AcidUtils - getChildState() ignoring([]) pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_0000002_v0000035
{code}


> Compaction cleaner fails to clean up deltas when using blocking compaction
> --------------------------------------------------------------------------
>
>                 Key: HIVE-23832
>                 URL: https://issues.apache.org/jira/browse/HIVE-23832
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Denys Kuzmenko
>            Assignee: Denys Kuzmenko
>            Priority: Major
>
> {code}
> CREATE TABLE default.compcleanup (
>    cda_id             int,
>    cda_run_id         varchar(255),
>    cda_load_ts        timestamp,
>    global_party_id    string,
>    group_id           string)
> COMMENT 'gp_2_gr'
> PARTITIONED BY (
>    cda_date           int,
>    cda_job_name       varchar(12))
> STORED AS ORC;
> -- cda_date=20200601/cda_job_name=core_base
> INSERT INTO default.compcleanup VALUES (1,'cda_run_id',NULL,'global_party_id','group_id',20200601,'core_base');
> SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
> UPDATE default.compcleanup SET cda_id = 2 WHERE cda_id = 1;
> SELECT * FROM default.compcleanup where cda_date = 20200601  and cda_job_name = 'core_base';
> ALTER TABLE default.compcleanup PARTITION (cda_date=20200601, cda_job_name='core_base')
COMPACT 'MAJOR' AND WAIT;
> {code}
> When using blocking compaction Cleaner skips processing due to the open txn by `ALTER
TABLE` below Compactor's one.
> {code}
> AcidUtils - getChildState() ignoring([]) pfile:/Users/denyskuzmenko/data/cdh/hive/warehouse/compcleanup5/cda_date=110601/cda_job_name=core_base/base_0000002_v0000035
> {code}
> {code}
>     if (!isDirUsable(baseDir, parsedBase.getVisibilityTxnId(), aborted, validTxnList))
{
>       return;
>     }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message