accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-3310) Garbage collection might remove files with references in the replication table
Date Sat, 08 Nov 2014 05:50:34 GMT


Josh Elser commented on ACCUMULO-3310:

Nice summarization. I also agree that #2 would be the best way to go.

Because tabletservers have to write "up" instead of "across" (a write to another non-metadata/root
table might be blocked by a minor compaction), tabletservers must write to metadata. This
is an invariant right now. Periodically, the master reads these records and propagates them
to the replication table. Eventually, there will be no additional records written to metadata
for a given file, as no tservers will be using that WAL anymore.

Presently, when the Master observes this case, it deletes the records from metadata after
ensuring the final record is propagated to the replication table. As Christopher points out,
there is a small possibility for a race condition to occur here. It should be easily mitigated
by moving the clean up of the metadata table the same time when the replication table is also
cleaned up. The biggest difficulty here will be ensuring that the records in both tables are
always cleaned up and orphans are not left in the metadata table. For example, if we delete
the records from the metadata for a WAL that is completely replicated and then update the
metadata table, we have the possibility to crash and never know that we didn't get to clean
up the metadata table. Not a huge issue, but something to be cognizant of in the implementation.

> Garbage collection might remove files with references in the replication table
> ------------------------------------------------------------------------------
>                 Key: ACCUMULO-3310
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: replication
>            Reporter: Christopher Tubbs
>            Assignee: Josh Elser
>             Fix For: 1.7.0
> There's a possible race condition deleting files from HDFS still needed for replication.
The Accumulo garbage collector will not check for references to files in use if it thinks
that the replication table does not exist (or is offline, as the case will be after ACCUMULO-3147
is completed). The problem with this is that the table may actually exist, and may be the
only place where the in-use file is still referenced, but the garbage collector may not yet
know the table exists.
> This probably has a very low probability of happening, and only really affects the case
when replication is first started. However, if it did happen, it would mean an error replicating
> Some solutions discussed with [~elserj]:
> # Always scan the table. _This requires the table to always exist and be online and available
for scanning._
> # Keep the status messages around until after they are removed from the replication table.
This requires moving the deletes from {{StatusMaker}} to {{RemoveCompleteReplicationRecords}}.
_There may be some implications for delaying removal of these entries, but probably not too
many, and none we can't work through._
> # Move the contents of the replication table to the metadata table and don't have a separate
replication table. _This would require a big refactoring, and might have performance implications
to the metadata table._

This message was sent by Atlassian JIRA

View raw message