accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2819) Provide WorkAssigner which is order-aware
Date Wed, 21 May 2014 01:59:38 GMT


ASF subversion and git services commented on ACCUMULO-2819:

Commit 50117b35535f7701ebde58f089489dd8dbcd432d in accumulo's branch refs/heads/ACCUMULO-378
from [~elserj]
[;h=50117b3 ]

ACCUMULO-2819 Add a test to ensure that an inprogress file that's fully replicated is removed.

This is mainly an optimization as it would be cleaned up outside of the
work assignment loop, but it should help to make a more responsive work assignment loop.

> Provide WorkAssigner which is order-aware
> -----------------------------------------
>                 Key: ACCUMULO-2819
>                 URL:
>             Project: Accumulo
>          Issue Type: Sub-task
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.7.0
> The current WorkAssigner implementation, which uses the DistributedWorkQueue, is great
because it allows the Master to be unaware of what tservers are available, and to allow any
tserver to perform the replication.
> The downside of this is that it is possible to replicate data that was ingested later
before the earlier ingested data. For example, say {{table1}} uses {{wal1}} to ingest some
data. We record that {{wal1}} has some replication to do, but, for whatever reason, we don't
get to it. More data is ingested into {{table1}}, and it starts using {{wal2}} after enough
data was ingested. Now, we have {{wal1}} and {{wal2}} which both have data to be replicated
for {{table1}}.
> Using the DistributedWorkQueue, we have no guarantee that {{wal1}} will be replicated
before {{wal2}}, which means we might replay a column update for the same row in the wrong
order (update from {{wal2}} and then update from {{wal1}}).
> While the DistributedWorkQueue is nice for the mentioned reason, in addition to the higher
throughput, it has obvious deficiencies depending on the workload and table schema. We need
to create a WorkAssigner that is order aware (what was the order in which the WALs for a table
were minor compacted, and ensure that replication occurs in that same order.

This message was sent by Atlassian JIRA

View raw message