Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4F4EC200CE0 for ; Thu, 27 Jul 2017 03:58:35 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 4DABC169FAF; Thu, 27 Jul 2017 01:58:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1DDFE169FAE for ; Thu, 27 Jul 2017 03:58:33 +0200 (CEST) Received: (qmail 98879 invoked by uid 500); 27 Jul 2017 01:58:32 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 98869 invoked by uid 99); 27 Jul 2017 01:58:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jul 2017 01:58:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 6AC22180312 for ; Thu, 27 Jul 2017 01:58:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.919 X-Spam-Level: X-Spam-Status: No, score=0.919 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 2VUDcPui14Ly for ; Thu, 27 Jul 2017 01:58:29 +0000 (UTC) Received: from unhygienix.ics.uci.edu (unhygienix.ics.uci.edu [128.195.14.130]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 9DF055F572 for ; Thu, 27 Jul 2017 01:58:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by unhygienix.ics.uci.edu (Postfix) with ESMTP id 452A4241ECC; Wed, 26 Jul 2017 18:58:28 -0700 (PDT) Date: Wed, 26 Jul 2017 18:58:27 -0700 From: "abdullah alamoudi (Code Review)" To: Jenkins CC: Xikui Wang , Yingyi Bu , Murtadha Hubail , Till Westmann Reply-To: bamousaa@gmail.com X-Gerrit-MessageType: newpatchset Subject: Change in asterixdb[master]: TERIXDB-1992][ING] Suspend/Resume active entities X-Gerrit-Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40 X-Gerrit-ChangeURL: X-Gerrit-Commit: fea02f4db96af88ea41aa3148b92665d0ea167dc In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 Message-Id: <20170727015828.452A4241ECC@unhygienix.ics.uci.edu> archived-at: Thu, 27 Jul 2017 01:58:35 -0000 Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/1875 to look at the new patch set (#24). Change subject: TERIXDB-1992][ING] Suspend/Resume active entities ...................................................................... TERIXDB-1992][ING] Suspend/Resume active entities - user model changes: wait for completion only returns when the entity becomes inactive. - storage format changes: no - interface changes: 01) Introduce: IMetadataLockManager for entity locking. IActiveNotificationHandler for handling active events. IRetryPolicy for recovery of failed active jobs. IActiveEntityController for controlling active entities. 02) IJobLifecycleListener.notifyJobFinish now passes the JobStatus and Exceptions. 03) IActiveEntityEventsListener.isActive() returns true, if entity is active, false otherwise. 04) IActiveEntityEventsListener.unregister() removes the listener upon entity deletion. 05) IActiveEntityEventsListener.getJobFailure() returns the job failure if the entity is in a failed state. 06) IStatementExecutor.getComponentProvider() returns the storage component provider. 07) IStatementExecutor.getApplicationContext() returns the application context. 08) IMetadataManager.upsertEntity to perfrom metadata entities' upsert operations. 09) IMetadataNode.upsertEntity to perfrom metadata entities' upsert operations 10) ICcApplicationContext.getMetadataLockManager() returns the lock manger. 11) ICcApplicationContext.getClusterStateManager() returns the cluster state manager. details: - Starting and stopping of active entities now go through their listeners rather than having the listener indicating active entity and not having it indicating inactive entity. This facilitates suspend/resume operations of long running active jobs for the sake of DDL operations or topology changes. - Unit tests for the vast majority of code paths and different possible scenarios have been added. Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40 --- A CMakeLists.txt M asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveEvent.java D asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJobNotificationHandler.java D asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveLifecycleListener.java M asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveRuntimeManager.java M asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActivityState.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/CountRetryPolicy.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/CountRetryPolicyFactory.java R asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEntityEventSubscriber.java M asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEntityEventsListener.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveNotificationHandler.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IRetryPolicy.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IRetryPolicyFactory.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/InfiniteRetryPolicy.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/InfiniteRetryPolicyFactory.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/NoRetryPolicyFactory.java A asterixdb/asterix-active/src/main/java/org/apache/asterix/active/SingleThreadEventProcessor.java M asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActivePartitionMessage.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/InvertedIndexPOperator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AbstractLangTranslator.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ActiveStatsApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ConnectorApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RebalanceApiServlet.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/active/ActiveEntityEventsListener.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/active/ActiveNotificationHandler.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/active/FeedEventsListener.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/DefaultStatementExecutorFactory.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/GlobalRecoveryManager.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/ConnectorApiServletTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/Action.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/ActiveEventsListenerTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/ActiveStatsTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/Actor.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/DummyFeedEventsListener.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/TestClusterControllerActor.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/TestEventsListener.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/TestNodeControllerActor.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/active/TestUserActor.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IClusterEventsSubscriber.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IMetadataLockManager.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/cluster/IGlobalRecoveryManager.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/ICcApplicationContext.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/AsterixException.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/CompilationException.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/MetadataException.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/RuntimeDataException.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/metadata/IDataset.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/metadata/IMetadataLock.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/metadata/LockList.java M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IDataSourceAdapter.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/AbstractFeedDataFlowController.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedRecordDataFlowController.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedTupleForwarder.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/FeedAdapter.java D asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/ActiveEntityEventsListener.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/AdapterExecutor.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/watch/AbstractSubscriber.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/watch/NoOpSubscriber.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/watch/StatsSubscriber.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/watch/WaitForStateSubscriber.java M asterixdb/asterix-lang-aql/src/main/java/org/apache/asterix/lang/aql/statement/SubscribeFeedStatement.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/ConnectFeedStatement.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataCache.java D asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataException.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataManager.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IActiveEntityController.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IMetadataEntityTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IMetadataManager.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IMetadataNode.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IValueExtractor.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataManagerUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/BuiltinTypeMap.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/FeedConnection.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Index.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/CompactionPolicyTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/DatasetTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/DatasourceAdapterTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/DatatypeTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/DataverseTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/ExternalFileTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FeedConnectionTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FeedPolicyTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FeedTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FunctionTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/LibraryTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/NodeGroupTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/NodeTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/BuiltinFeedPolicies.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/DatasetLock.java D asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/IMetadataLock.java D asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/LockList.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/MetadataLock.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/MetadataLockManager.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/MetadataLockUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/MetadataUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SplitsAndConstraintsUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/valueextractors/MetadataEntityValueExtractor.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/valueextractors/NestedDatatypeNameValueExtractor.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/valueextractors/TupleCopyValueExtractor.java M asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/DatasetTupleTranslatorTest.java M asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslatorTest.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/utils/CcApplicationContext.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/utils/ClusterStateManager.java M asterixdb/asterix-tools/src/test/java/org/apache/asterix/tools/datagen/AdmDataGen.java M asterixdb/asterix-tools/src/test/java/org/apache/asterix/tools/translator/ADGenDmlTranslator.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/opcallbacks/UpsertOperationCallback.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/dataset/IDatasetPartitionManager.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/HyracksDataException.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/job/IJobLifecycleListener.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/application/CCServiceContext.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/cluster/NodeManager.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/dataset/DatasetDirectoryService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/job/JobManager.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/RegisterNodeWork.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/cluster/NodeManagerTest.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/utils/ExceptionUtils.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/NodeControllerService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/dataset/DatasetPartitionManager.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/partitions/PipelinedPartition.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/work/AbortAllTasksWork.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/work/AbortTasksWork.java M hyracks-fullstack/hyracks/hyracks-dataflow-common/src/main/java/org/apache/hyracks/dataflow/common/utils/FrameDebugUtils.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/collectors/NonDeterministicChannelReader.java 141 files changed, 6,171 insertions(+), 1,845 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/75/1875/24 -- To view, visit https://asterix-gerrit.ics.uci.edu/1875 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifeac8c73e6bad39a13663b84a52121356e3c6b40 Gerrit-PatchSet: 24 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi