hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.
Date Thu, 25 Sep 2014 20:53:34 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alan Gates updated HIVE-8258:
-----------------------------
    Attachment: HIVE-8258.patch

This patch changes the cleaner to snapshot all of the locks held on a table or partition when
the request comes in.  It then watches for those locks to expire and then does the cleaning.
 This keeps the cleaner from removing files that are still being read but at the same time
allows the cleaner to ignore any subsequent locks.  This works since any process that obtained
a lock after the cleaner will be reading the newer versions of the file.

> Compactor cleaners can be starved on a busy table or partition.
> ---------------------------------------------------------------
>
>                 Key: HIVE-8258
>                 URL: https://issues.apache.org/jira/browse/HIVE-8258
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 0.13.1
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Critical
>         Attachments: HIVE-8258.patch
>
>
> Currently the cleaning thread in the compactor does not run on a table or partition while
any locks are held on this partition.  This leaves it open to starvation in the case of a
busy table or partition.  It only needs to wait until all locks on the table/partition at
the time of the compaction have expired.  Any jobs initiated after that (and thus any locks
obtained) will be for the new versions of the files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message