accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-420) Allow per compaction iterator settings
Date Tue, 22 May 2012 21:01:41 GMT


Keith Turner commented on ACCUMULO-420:

This feature needs to work correctly in the case of concurrent compaction request.  If two
user request compaction of a table at around the same time and one passes iterators and one
does not, then iterators should be applied to all relevant tablets for the user that passed
iterators.   Also, tablet servers can decide to compact all files in a tablet at any time.
 This another form of concurrency that must be considered.

The current design is that there is a compaction counter in zookeeper.  A compaction operation
will increment this.  As each tablet is compacted it records what the compaction counter was
when it started.  Compaction operations wait for the compaction counters for each relevant
tablet in the metadata table to reach at least its count.

The following is a possible new design.

compact FATE op
 # set compaction iterators in zookeeper
 # increment compaction counter in zookeeper
 # wait until all tablets in compaction range have at least this operations compaction counter
 # remove compaction iterators in zookeeper

TServer compact all operation
 # get compaction iterators from zookeeper
 # get compaction counter in zookeeper
 # compact with current compaction iterators
 # write compaction counter to metadata table

 * setting compaction iterators in zookeeper must accomodate concurrent request
 * the iterators are intentionally read from zookeeper before the counter.  The iterators
are written before the counter.  Does zookeeper guarantee that iterators written before the
counter will always be seen?  If not there is a race condition. 
 * If two users submit compaction request with iterators, this design could run compactions
with both users iterators settings.  Does anyone see an issue w/ this?
 * Iterators from one user compaction request could be applied to multiple compactions of
a single tablet.
> Allow per compaction iterator settings
> --------------------------------------
>                 Key: ACCUMULO-420
>                 URL:
>             Project: Accumulo
>          Issue Type: New Feature
>            Reporter: Keith Turner
>            Assignee: Sapan Shah
>             Fix For: 1.5.0
> It may be useful to allow the compact command to specify an iterator to be used for that
compaction.  For example if someone wanted to apply a filter once to a table, they could force
a compaction with that filter.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message