hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17732) Coprocessor Design Improvements
Date Wed, 27 Sep 2017 23:58:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183473#comment-16183473
] 

Hudson commented on HBASE-17732:
--------------------------------

FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3789 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3789/])
HBASE-17732 Coprocessor Design Improvements (appy: rev 97513466c05f5eaadb94425c98098063ac374098)
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
* (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpointNullResponse.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverForAddingMutationsFromCoprocessors.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckMOB.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEnvironment.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorConfiguration.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerRetriableFailure.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
* (edit) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/BulkDeleteEndpoint.java
* (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
* (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncAdminBuilder.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSettingTimeoutOnBlockingPoint.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* (add) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALCoprocessor.java
* (edit) hbase-endpoint/src/main/java/org/apache/hadoop/hbase/coprocessor/AggregateImplementation.java
* (add) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorServiceBackwardCompatibility.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsReplication.java
* (edit) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/ExampleMasterObserverWithMetrics.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mob/compactions/TestMobCompactor.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityReplication.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMobCloneSnapshotFromClient.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionLocatorTimeout.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestResultFromCoprocessor.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestFSWAL.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestWithDisabledAuthorization.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientOperationInterrupt.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionServerObserver.java
* (edit) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RowCountEndpoint.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterSpaceQuotaObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/CoprocessorWhitelistMasterObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverBypass.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionServerCoprocessor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALCoprocessorEnvironment.java
* (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorService.java
* (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestProtobufLog.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/ObserverContext.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicaWithCluster.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckTwoRS.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController3.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/SingletonCoprocessorService.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/ErrorThrowingGetObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java
* (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsOfflineMode.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionServerCoprocessorEnvironment.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorMetrics.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALCoprocessor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java
* (edit) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorServiceBackwardCompatiblity.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/ProtobufCoprocessorService.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbort.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController2.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorEndpoint.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/WriteSinkCoprocessor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/NoOpScanPolicyObserver.java
* (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithOperationAttributes.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestScanEarlyTermination.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCoprocessorWhitelistMasterObserver.java
* (edit) hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupDeleteWithFailures.java
* (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/EndpointObserver.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckReplicas.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* (edit) hbase-endpoint/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableBatch.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncNonMetaRegionLocatorConcurrenyLimit.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicasClient.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BulkLoadObserver.java
* (edit) hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/BackupObserver.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterCoprocessor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestOpenTableInCoprocessor.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/namespace/TestNamespaceAuditor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionCoprocessorEnvironment.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerWithCorruptHFile.java
* (edit) hbase-endpoint/src/main/java/org/apache/hadoop/hbase/coprocessor/Export.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
* (edit) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/ExampleRegionObserverWithMetrics.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLs.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelReplicationWithExpAsString.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestAsyncCoprocessorEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckOneRS.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotClientRetries.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/package-info.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWithTags.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRowProcessorEndpoint.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* (edit) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/ZooKeeperScanPolicyObserver.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestEnableTable.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAvoidCellReferencesIntoShippedBlocks.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
* (edit) hbase-endpoint/src/test/java/org/apache/hadoop/hbase/coprocessor/ColumnAggregationEndpointWithErrors.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/Coprocessor.java
* (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionCoprocessor.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorHost.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterCoprocessorEnvironment.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java


> Coprocessor Design Improvements
> -------------------------------
>
>                 Key: HBASE-17732
>                 URL: https://issues.apache.org/jira/browse/HBASE-17732
>             Project: HBase
>          Issue Type: Improvement
>          Components: Coprocessors
>            Reporter: Appy
>            Assignee: Appy
>            Priority: Critical
>              Labels: incompatible
>             Fix For: 2.0.0-alpha-4
>
>         Attachments: HBASE-17732.master.001.patch, HBASE-17732.master.002.patch, HBASE-17732.master.003.patch,
HBASE-17732.master.004.patch, HBASE-17732.master.005.patch, HBASE-17732.master.006.patch,
HBASE-17732.master.007.patch, HBASE-17732.master.008.patch, HBASE-17732.master.009.patch,
HBASE-17732.master.010.patch, HBASE-17732.master.011.patch, HBASE-17732.master.012.patch,
HBASE-17732.master.013.patch, HBASE-17732.master.014.patch
>
>
> The two main changes are:
> * *Adding template for coprocessor type to CoprocessorEnvironment i.e. {{interface CoprocessorEnvironment<C
extends Coprocessor>}}*
>   ** Enables us to load only relevant coprocessors in hosts. Right now each type of host
loads all types of coprocs and it's only during execOperation that it checks if the coproc
is of correct type i.e. XCoprocessorHost will load XObserver, YObserver, and all others, and
will check in execOperation if {{coproc instanceOf XObserver}} and ignore the rest.
>   ** Allow sharing of a bunch functions/classes which are currently duplicated in each
host. For eg. CoprocessorOperations, CoprocessorOperationWithResult, execOperations().
> * *Introduce 4 coprocessor classes and use composition between these new classes and
and old observers*
>   ** The real gold here is, moving forward, we'll be able to break down giant everything-in-one
observers (masterobserver has 100+ functions) into smaller, more focused observers. These
smaller observer can then have different compat guarantees!!
> Here's a more detailed design doc: https://docs.google.com/document/d/1mPkM1CRRvBMZL4dBQzrus8obyvNnHhR5it2yyhiFXTg/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message