hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <>
Subject [jira] [Commented] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.
Date Tue, 30 Sep 2014 20:17:35 GMT


Eugene Koifman commented on HIVE-8258:

Review Comments:
1. TestCleaner has 3 unused imports
2. Cleaner: comment at lines 74-78:  I think it would be good to elaborate on why this works.
 Something like "any readers that acquired new locks on the same partition will not read files
we are trying to delete since they will have been merged into other deltas/base by compaction
and AcidUtils.getAcidState() has the logic to do that"
3. Cleaner line 86:  {noformat}if (!compactId2LockMap.containsKey( {{noformat} - I
don't think this is the right map to use here
4. The cleaner may be in removeFiles() doing fs.delete() while some reader may be calling
AcidUtils.getAcidState() at exactly the same.  Is this a race condition that can cause problems?
 You get a list of files in getAcidState() but by the time you query the metadata about one
such file it's deleted by the cleaner.  Is the FS flexible enough for this?

> Compactor cleaners can be starved on a busy table or partition.
> ---------------------------------------------------------------
>                 Key: HIVE-8258
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 0.13.1
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Critical
>         Attachments: HIVE-8258.patch
> Currently the cleaning thread in the compactor does not run on a table or partition while
any locks are held on this partition.  This leaves it open to starvation in the case of a
busy table or partition.  It only needs to wait until all locks on the table/partition at
the time of the compaction have expired.  Any jobs initiated after that (and thus any locks
obtained) will be for the new versions of the files.

This message was sent by Atlassian JIRA

View raw message