Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 147B511B07 for ; Fri, 16 May 2014 22:18:30 +0000 (UTC) Received: (qmail 33945 invoked by uid 500); 16 May 2014 11:44:30 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 91440 invoked by uid 500); 16 May 2014 11:21:40 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 37785 invoked by uid 99); 16 May 2014 11:11:36 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 May 2014 11:11:36 +0000 Date: Fri, 16 May 2014 11:11:35 +0000 (UTC) From: "Josh Elser (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (ACCUMULO-2819) Provide WorkAssigner which is order-aware MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Josh Elser created ACCUMULO-2819: ------------------------------------ Summary: Provide WorkAssigner which is order-aware Key: ACCUMULO-2819 URL: https://issues.apache.org/jira/browse/ACCUMULO-2819 Project: Accumulo Issue Type: Sub-task Reporter: Josh Elser Assignee: Josh Elser Fix For: 1.7.0 The current WorkAssigner implementation, which uses the DistributedWorkQueue, is great because it allows the Master to be unaware of what tservers are available, and to allow any tserver to perform the replication. The downside of this is that it is possible to replicate data that was ingested later before the earlier ingested data. For example, say {{table1}} uses {{wal1}} to ingest some data. We record that {{wal1}} has some replication to do, but, for whatever reason, we don't get to it. More data is ingested into {{table1}}, and it starts using {{wal2}} after enough data was ingested. Now, we have {{wal1}} and {{wal2}} which both have data to be replicated for {{table1}}. Using the DistributedWorkQueue, we have no guarantee that {{wal1}} will be replicated before {{wal2}}, which means we might replay a column update for the same row in the wrong order (update from {{wal2}} and then update from {{wal1}}). While the DistributedWorkQueue is nice for the mentioned reason, in addition to the higher throughput, it has obvious deficiencies depending on the workload and table schema. We need to create a WorkAssigner that is order aware (what was the order in which the WALs for a table were minor compacted, and ensure that replication occurs in that same order. -- This message was sent by Atlassian JIRA (v6.2#6252)