hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Csaba Ringhofer (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
Date Mon, 20 Jan 2020 18:16:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019669#comment-17019669
] 

Csaba Ringhofer commented on HIVE-21931:
----------------------------------------

Thanks, I am ok with HIVE-22554  as solution. I am closing this now, will reopen if it doesn't
work for me.

> Slow compaction for tiny tables
> -------------------------------
>
>                 Key: HIVE-21931
>                 URL: https://issues.apache.org/jira/browse/HIVE-21931
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 3.1.0
>            Reporter: Csaba Ringhofer
>            Priority: Major
>              Labels: compaction
>
> I observed the issue in Impala development environment when (major) compacting insert_only
transactional tables in Hive. The compaction could take ~10 minutes even when it only had
to merge 2 rows from 2 inserts. The actual work was done much earlier, the new base file was
correctly written to HDFS, and Hive seemed to wait without doing any work.
> The compactions are started manually, hive.compactor.initiator.on=false to avoid "surprise
compaction" during tests.
> {code}
> hive.compactor.abortedtxn.threshold=1000
> hive.compactor.check.interval=300s
> hive.compactor.cleaner.run.interval=5000ms
> hive.compactor.compact.insert.only=true
> hive.compactor.crud.query.based=false
> hive.compactor.delta.num.threshold=10
> hive.compactor.delta.pct.threshold=0.1
> hive.compactor.history.reaper.interval=2m
> hive.compactor.history.retention.attempted=2
> hive.compactor.history.retention.failed=3
> hive.compactor.history.retention.succeeded=3
> hive.compactor.initiator.failed.compacts.threshold=2
> hive.compactor.initiator.on=false
> hive.compactor.max.num.delta=500
> hive.compactor.worker.threads=4
> hive.compactor.worker.timeout=86400s
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message