accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser" <josh.el...@gmail.com>
Subject Review Request 21752: Initial implementation of multi-instance replication
Date Wed, 21 May 2014 05:02:07 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21752/
-----------------------------------------------------------

Review request for accumulo.


Bugs: ACCUMULO-378
    https://issues.apache.org/jira/browse/ACCUMULO-378


Repository: accumulo


Description
-------

Initial implementation that covers (high-level) replication of "live" data using WAls (no
support for bulk-imports yet), master->slave and master->master configurations, replication
implementations to another Accumulo instance, and multiple replication "assignment" implementations
(in what order do we replicate WALs, replicating WALs concurrently, etc). Normal code path
(no tables configured for replication) should not have been altered. Some effort was put into
code cleanup (compiler warnings, unused imports, formatting issues), but I'd ask that reviewers
focus on the substance rather than the nitty-gritty at the moment. I think this is at a fairly
stable point right now, so I wanted to make it available to those who plan to look at it.

"Big" things that aren't yet addressed: insight via monitor, bulk-import replication, authentication/authorization
to the remote Accumulo cluster and encryption of data to remote Accumulo cluster. All of these
(maybe not bulk-import replication) should be done before this gets merged in upstream.

The ITs are a great place to start looking to see how things work.


Diffs
-----

  core/pom.xml 2c3f648 
  core/src/main/java/org/apache/accumulo/core/Constants.java 7d602bb 
  core/src/main/java/org/apache/accumulo/core/client/Connector.java 4a2acff 
  core/src/main/java/org/apache/accumulo/core/client/admin/ReplicationOperations.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java 0df35f6 
  core/src/main/java/org/apache/accumulo/core/client/impl/MasterClient.java 1fa8f12 
  core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationClient.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/client/impl/ReplicationOperationsImpl.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/client/mock/MockAccumulo.java 2c26ecc 
  core/src/main/java/org/apache/accumulo/core/client/mock/MockConnector.java 996198c 
  core/src/main/java/org/apache/accumulo/core/client/mock/MockTable.java cb50761 
  core/src/main/java/org/apache/accumulo/core/client/mock/MockTableOperations.java a44a027

  core/src/main/java/org/apache/accumulo/core/client/replication/PeerExistsException.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/client/replication/PeerNotFoundException.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/client/replication/ReplicaSystem.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/client/replication/ReplicaSystemFactory.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/client/replication/ReplicationTable.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/conf/Property.java 62b0a33 
  core/src/main/java/org/apache/accumulo/core/data/ConditionalMutation.java e43968c 
  core/src/main/java/org/apache/accumulo/core/data/Mutation.java 42fa143 
  core/src/main/java/org/apache/accumulo/core/data/thrift/MultiScanResult.java f0b8a87 
  core/src/main/java/org/apache/accumulo/core/data/thrift/ScanResult.java 6472587 
  core/src/main/java/org/apache/accumulo/core/data/thrift/TConditionalMutation.java 3f9b3f7

  core/src/main/java/org/apache/accumulo/core/data/thrift/TMutation.java f698892 
  core/src/main/java/org/apache/accumulo/core/data/thrift/UpdateErrors.java 59ce5cb 
  core/src/main/java/org/apache/accumulo/core/iterators/Combiner.java ceb4411 
  core/src/main/java/org/apache/accumulo/core/metadata/schema/MetadataSchema.java 06eae23

  core/src/main/java/org/apache/accumulo/core/protobuf/ProtobufUtil.java PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/AccumuloReplicationReplayer.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/PrintReplicationRecords.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/replication/RemoteReplicationErrorCode.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/ReplicationConfigurationUtil.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/ReplicationCoordinatorErrorCode.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/ReplicationSchema.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/replication/ReplicationTarget.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/replication/StatusUtil.java PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/proto/Replication.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/replication/thrift/KeyValues.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/replication/thrift/RemoteCoordinationException.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/thrift/RemoteReplicationException.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/thrift/Replication.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/replication/thrift/ReplicationCoordinator.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/thrift/ReplicationServicer.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/replication/thrift/WalEdits.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/schema/Section.java PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/tabletserver/thrift/ReplicationFailedException.java
PRE-CREATION 
  core/src/main/protobuf/replication.proto PRE-CREATION 
  core/src/main/scripts/generate-protobuf.sh PRE-CREATION 
  core/src/main/scripts/generate-thrift.sh 5a5d69f 
  core/src/main/thrift/data.thrift ae6f439 
  core/src/main/thrift/replication.thrift PRE-CREATION 
  core/src/test/java/org/apache/accumulo/core/replication/ReplicationConfigurationUtilTest.java
PRE-CREATION 
  core/src/test/java/org/apache/accumulo/core/replication/ReplicationOperationsImplTest.java
PRE-CREATION 
  core/src/test/java/org/apache/accumulo/core/replication/ReplicationSchemaTest.java PRE-CREATION

  core/src/test/java/org/apache/accumulo/core/replication/ReplicationTargetTest.java PRE-CREATION

  core/src/test/java/org/apache/accumulo/core/replication/StatusUtilTest.java PRE-CREATION

  core/src/test/java/org/apache/accumulo/core/replication/proto/StatusTest.java PRE-CREATION

  docs/src/main/resources/design/ACCUMULO-378-design.mdtext PRE-CREATION 
  docs/src/main/resources/state/replicationstatus.gv PRE-CREATION 
  docs/src/main/resources/state/replicationstatus.png PRE-CREATION 
  minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloConfigImpl.java
3258991 
  server/base/pom.xml f698621 
  server/base/src/main/java/org/apache/accumulo/server/conf/TableConfigurationCopy.java PRE-CREATION

  server/base/src/main/java/org/apache/accumulo/server/constraints/MetadataConstraints.java
ecb0db8 
  server/base/src/main/java/org/apache/accumulo/server/fs/VolumeUtil.java 436667c 
  server/base/src/main/java/org/apache/accumulo/server/init/Initialize.java 10cdce6 
  server/base/src/main/java/org/apache/accumulo/server/replication/AbstractWorkAssigner.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/replication/ReplicationTable.java PRE-CREATION

  server/base/src/main/java/org/apache/accumulo/server/replication/StatusCombiner.java PRE-CREATION

  server/base/src/main/java/org/apache/accumulo/server/replication/WorkAssigner.java PRE-CREATION

  server/base/src/main/java/org/apache/accumulo/server/util/MasterMetadataUtil.java e936b97

  server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java 341f39a

  server/base/src/main/java/org/apache/accumulo/server/util/ReplicationTableUtil.java PRE-CREATION

  server/base/src/main/java/org/apache/accumulo/server/util/SystemPropUtil.java 1d93f90 
  server/base/src/main/java/org/apache/accumulo/server/zookeeper/DistributedWorkQueue.java
4738c2b 
  server/base/src/test/java/org/apache/accumulo/server/master/balancer/TableLoadBalancerTest.java
82e5885 
  server/base/src/test/java/org/apache/accumulo/server/replication/ReplicationTableTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/replication/StatusCombinerTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/util/ReplicationTableUtilTest.java
PRE-CREATION 
  server/gc/pom.xml 5c501de 
  server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogs.java ae850af

  server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectionAlgorithm.java 40fb847 
  server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectionEnvironment.java 3e36617

  server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java 7ebfff0 
  server/gc/src/main/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferences.java
PRE-CREATION 
  server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java f90b965

  server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectionTest.java 537fc46 
  server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
PRE-CREATION 
  server/gc/src/test/resources/log4j.properties e74337f 
  server/master/pom.xml fd53a8c 
  server/master/src/main/java/org/apache/accumulo/master/Master.java 526ca8a 
  server/master/src/main/java/org/apache/accumulo/master/replication/DistributedWorkQueueWorkAssigner.java
PRE-CREATION 
  server/master/src/main/java/org/apache/accumulo/master/replication/FinishedWorkUpdater.java
PRE-CREATION 
  server/master/src/main/java/org/apache/accumulo/master/replication/MasterReplicationCoordinator.java
PRE-CREATION 
  server/master/src/main/java/org/apache/accumulo/master/replication/RemoveCompleteReplicationRecords.java
PRE-CREATION 
  server/master/src/main/java/org/apache/accumulo/master/replication/ReplicationDriver.java
PRE-CREATION 
  server/master/src/main/java/org/apache/accumulo/master/replication/SequentialWorkAssigner.java
PRE-CREATION 
  server/master/src/main/java/org/apache/accumulo/master/replication/StatusMaker.java PRE-CREATION

  server/master/src/main/java/org/apache/accumulo/master/replication/WorkDriver.java PRE-CREATION

  server/master/src/main/java/org/apache/accumulo/master/replication/WorkMaker.java PRE-CREATION

  server/master/src/main/java/org/apache/accumulo/master/tableOps/CreateTable.java 5b64053

  server/master/src/test/java/org/apache/accumulo/master/replication/AbstractWorkAssignerTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/DistributedWorkQueueWorkAssignerTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/FinishedWorkUpdaterTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/MasterReplicationCoordinatorTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/RemoveCompleteReplicationRecordsTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/SequentialWorkAssignerTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/StatusMakerTest.java
PRE-CREATION 
  server/master/src/test/java/org/apache/accumulo/master/replication/WorkMakerTest.java PRE-CREATION

  server/master/src/test/resources/log4j.properties e74337f 
  server/tserver/pom.xml 100bc2d 
  server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java fadf4ed 
  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java 144d59b 
  server/tserver/src/main/java/org/apache/accumulo/tserver/log/TabletServerLogger.java 871f4ae

  server/tserver/src/main/java/org/apache/accumulo/tserver/replication/AccumuloReplicaSystem.java
PRE-CREATION 
  server/tserver/src/main/java/org/apache/accumulo/tserver/replication/BatchWriterReplicationReplayer.java
PRE-CREATION 
  server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationProcessor.java
PRE-CREATION 
  server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationServicerHandler.java
PRE-CREATION 
  server/tserver/src/main/java/org/apache/accumulo/tserver/replication/ReplicationWorker.java
PRE-CREATION 
  server/tserver/src/test/java/org/apache/accumulo/tserver/replication/AccumuloReplicaSystemTest.java
PRE-CREATION 
  server/tserver/src/test/java/org/apache/accumulo/tserver/replication/ReplicationProcessorTest.java
PRE-CREATION 
  server/tserver/src/test/resources/log4j.properties e74337f 
  test/src/main/java/org/apache/accumulo/test/replication/MockReplicaSystem.java PRE-CREATION

  test/src/main/java/org/apache/accumulo/test/replication/ReplicationTablesPrinterThread.java
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/test/replication/CyclicReplicationIT.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationDeadlockTest.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationFilesClosedAfterUnusedTest.java
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/test/replication/ReplicationIT.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationPortAdvertisementIT.java
PRE-CREATION 
  test/src/test/java/org/apache/accumulo/test/replication/ReplicationSequentialIT.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationSourceOnlyIT.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationTablesMacTest.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationTest.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationWithGCIT.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/ReplicationWithMakerTest.java PRE-CREATION

  test/src/test/java/org/apache/accumulo/test/replication/StatusCombinerMacTest.java PRE-CREATION

  test/src/test/resources/log4j.properties 3fa1983 

Diff: https://reviews.apache.org/r/21752/diff/


Testing
-------

All existing unit and integration tests pass. Expected as normal code path hasn't changed.

Loads of new tests included, many ITs which ensure that replication "bookkeeping" is cleaned
up when appropriate, all data is replicated to a peer (and not over-replicated or re-replicated).


Thanks,

Josh Elser


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message