accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-3423) speed up WAL roll-overs
Date Sat, 09 May 2015 01:28:00 GMT


Josh Elser commented on ACCUMULO-3423:

I just want to state: these changes are making me really nervous for inclusion in 1.7.0. I
don't believe enough rigor has gone into actually flushing out bugs. I'll be continuing to
work on this over the weekend, but I want to make everyone else aware of my considerations
about reverting these changes for 1.7.0.

> speed up WAL roll-overs
> -----------------------
>                 Key: ACCUMULO-3423
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: master, tserver
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Blocker
>             Fix For: 1.7.0
>         Attachments: WAL-slowdown-graphs.jpg
>          Time Spent: 40m
>  Remaining Estimate: 0h
> After reading the proposal on HBASE-10278, I realized there are many ways to make the
Accumulo WAL roll-over faster.
> # Open two WALogs, but use only one until it reaches the WALog roll-over size
> # Rollover consists only of swapping the writers
> # WALog roll consists of the final close, which can happen in parallel
> # Don't mark the tablets with log entries: they are already marked with the tserver
> # The tserver can make notes about the logs-in-use in the metadata table(s) as part of
opening the log.
> # The master can copy the log entries to tablets while unassigning them,  piggybacking
on the unassigment mutation.
> # Tablet servers can remove their current log entries from the metadata tables when they
have no tablets using them.
> There are two issues: 
> # tablets will have an empty file in recovery, nearly all the time, but the recovery
code already handles that case.  
> # presently, a tablet doesn't have a marker for a log it did not use. Many more tablets
will attempt to recover when it is unnecessary.
> This would also address ACCUMULO-2889.

This message was sent by Atlassian JIRA

View raw message