Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of josh.elser@gmail.com designates
 209.85.216.176 as permitted sender)
Message-ID: <53ACDB90.3080907@gmail.com>
Date: Thu, 26 Jun 2014 22:48:48 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: dev@accumulo.apache.org
Subject: Re: Accumulo 1.7 and Data Center Replication
References: 
 <CAA7ooCAAaEN_chghJue12p7Rpf3epken3v7AcKdWMOPJLsXkwQ@mail.gmail.com>
In-Reply-To: 
 <CAA7ooCAAaEN_chghJue12p7Rpf3epken3v7AcKdWMOPJLsXkwQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Joe,

I'm the guy to ask if you'd like more information about the replication 
feature. You already found the parent ticket, so that has a bunch of 
technical "what's been done".

At a high level, replication was implemented as a framework in Accumulo 
to copy data that was written to a table to another "location". The 
provided initial implementation is to replicate the data as-is to 
another Accumulo table (usually some other Accumulo instance). You'll 
also find a new page in the monitor some basic administration tools in 
the code via Instance#replicationOperations.

I've published a recent version of the user manual[1] which goes into 
some more detail on the feature, as well as how to configure it.

You can also check the replication component on JIRA [2] to see what I 
have lined up. Automatically replicating bulk-loaded files will be a bit 
of work. There are some other minor things that could be improved. We 
can delve into the more technical implementation difficulties if you'd like.

I've written a basic test to evaluate equivalence by generating a Merkle 
tree for two tables. This has been promising so far, but it currently is 
living in my Github[3]. I need to figure out where/how best to include 
it in Apache.

Finally, having resources to do a larger-scale test would be great, and 
testing failure conditions over multiple nodes is probably the biggest 
area that needs to be tested more. I can simulate this on a small scale, 
but I don't have the resources to do an appropriate larger test with 
injected failure.

If you have something specific you'd like to help out with, I'd be happy 
to work with you.

<employer-hat>This feature will also be included in the next version of 
Accumulo shipped in HDP</employer-hat>

- Josh

[1] http://people.apache.org/~elserj/accumulo_user_manual.html#_replication
[2] 
https://issues.apache.org/jira/issues/?jql=project%20%3D%20ACCUMULO%20AND%20component%20%3D%20replication%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20key%20DESC
[3] https://github.com/joshelser/merkle

On 6/26/14, 8:50 PM, Joe Stein wrote:
> Hi, I was hoping to get some more info around the 1.7 release and what are
> the to-be-dos and plans around it?
>
> Is there any help that is needed from a contribution perspective in
> anyways? Testing? Documentation? Pending coding or such?
>
> We are going to be rolling trunk into two of our lab environments
> specifically for https://issues.apache.org/jira/browse/ACCUMULO-378 as it
> is a requirement for one of my projects at Bloomberg for Accumulo to have
> data center replication before we go live.   This works is going to be over
> the next month(s) with lots of cycles dedicated to Accumulo 1.7 in the next
> few sprints.
>
> Also, I wanted to reach out if folks are looking for full time, contract or
> even side work with Accumulo. We have projects right now going on and are
> looking for more hands on keyboards.
>
> Anyways, thanks for all the great work!!!! I am looking forward to more
> continued success with the system, more integrations and to be able to
> become more active in the community.
>
> /*******************************************
>   Joe Stein
>   Founder, Principal Consultant
>   Big Data Open Source Security LLC
>   http://www.stealth.ly
>   Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>