accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1044) bulk imported files showing up in metadata after bulk import fails
Date Mon, 24 Jun 2013 16:40:20 GMT


Eric Newton commented on ACCUMULO-1044:

So, in 1.5.0, the situation is much better, but there's still a race condition:

 * master delegates loading to a tserver worker
 * worker tserver assigns the files to the appropriate serving tservers
 * the serving tserver fails to load the file due to contention, full gc, and other distributed
computing problems; worker tserver times out
 * working tserver gives up, reports failure to the master
 * a thread on the serving server wakes up and starts the tablet load
 * master talks to all the worker servers, and finds them not working on the transaction
 * master looks for references, finds none, and moves the file to failures
 * the server tserver awakes up and finishes the file load and updates the !METADATA table
 * the master wipes out any loaded markers

The master needs to ask the serving tservers if they are done with the transaction; and since
the master doesn't know which server got the files, it needs to ask all of them.  The serving
tservers will then be up-to-date with zookeeper, and will never load the file afterwards.
 If the master asks the serving tserver to remove the loaded flags, then we do not need to
be concerned about splits.
> bulk imported files showing up in metadata after bulk import fails
> ------------------------------------------------------------------
>                 Key: ACCUMULO-1044
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: master, tserver
>    Affects Versions: 1.4.2
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Critical
>              Labels: 15_qa_bug
>             Fix For: 1.6.0
> Bulk import fails.  The file is moved to the failures directory.
> But references in the !METADATA table remain.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message