accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dylan Hutchison <>
Subject Re: Scans during Compaction
Date Mon, 23 Feb 2015 18:42:50 GMT
Thanks Adam and Keith.

I see the following as a potential solution that achieves (1) low latency
for clients that want to see entries after an iterator and (2) the entries
from that iterator persisting in the Accumulo table.

   1. Start a major compaction in thread T1 of a client with the iterator
   set, blocking until the compaction completes.
   2. Start scanning in thread T2 of the client with the same iterator now
   set at scan-time scope. Use an isolated scanner to make sure we do not read
   the results of the major compaction committing, though this is not
   full-proof due to timing and that the isolated scanner is row-wise.
   3. Eventually, T1 unblocks and signals that the compaction completes.
   T1 interrupts T2.
   4. Thread T2 stops scanning, removes the scan-time iterator, and starts
   scanning again at the point it last left off, now seeing the results of the
   major compaction which already passed through the iterator.

The whole scheme is only necessary if the client wants results faster than
the major compaction completes.  A disadvantage is duplicated work -- the
iterator runs at scan-time and at compaction-time until the compaction
finishes.  This may strain server resources.

Will think about other schemes.  If only we could attach an apply-once
scan-time iterator, that also persists its results to an Accumulo table in
a streaming fashion.  Or on the flip side, a one-time compaction iterator
that streams results, such that we could scan from them right away instead
of needing to wait for the entire compaction to complete.

Dylan Hutchison

On Mon, Feb 23, 2015 at 12:48 PM, Adam Fuchs <> wrote:

> Dylan,
> The effect of a major compaction is never seen in queries before the major
> compaction completes. At the end of the major compaction there is a
> multi-phase commit which eventually replaces all of the old files with the
> new file. At that point the major compaction will have completely processed
> the given tablet's data (although other tablets may not be synchronized).
> For long-running non-isolated queries (more than a second or so) the
> iterator tree is occasionally rebuilt and re-seeked. When it is rebuilt it
> will use whatever is the latest file set, which will include the results of
> a completed major compaction.
> In your case #1 that's a tricky guarantee to make across a whole tablet,
> but it can be made one row at a time by using an isolated iterator.
> To make your case #2 work, you probably will have to implement some
> higher-level logic to only start your query after the major compaction has
> completed, using an external mechanism to track the completion of your
> transformation.
> Adam
> On Mon, Feb 23, 2015 at 12:35 PM, Dylan Hutchison <>
> wrote:
>> Hello all,
>> When I initiate a full major compaction (with flushing turned on)
>> manually via the Accumulo API
>> <,,,%20java.util.List,%20boolean,%20boolean)>,
>> how does the table appear to
>>    1. clients that started scanning the table before the major
>>    compaction began;
>>    2. clients that start scanning during the major compaction?
>> I'm interested in the case where there is an iterator attached to the
>> full major compaction that modifies entries (respecting sorted order of
>> entries).
>> The best possible answer for my use case, with case #2 more important
>> than case #1 and *low latency* more important than high throughput, is
>> that
>>    1. clients that started scanning before the compaction began would
>>    not see entries altered by the compaction-time iterator;
>>    2. clients that start scanning during the major compaction stream
>>    back entries as they finish processing from the major compaction, such that
>>    the clients *only* see entries that have passed through the
>>    compaction-time iterator.
>> How accurate are these descriptions?  If #2 really were as I would like
>> it to be, then a scan on the range (-inf,+inf) started after compaction
>> would "monitor compaction progress," such that the first entry batch
>> transmits to the scanner as soon as it is available from the major
>> compaction, and the scanner finishes (receives all entries) exactly when
>> the compaction finishes.  If this is not possible, I may make something to
>> that effect by calling the blocking version of compact().
>> Bonus: how does cancelCompaction()
>> <>
>> affect clients scanning in case #1 and case #2?
>> Regards,
>> Dylan Hutchison


View raw message