Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 49EB1200BAE for ; Fri, 28 Oct 2016 13:16:56 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 487F1160AE3; Fri, 28 Oct 2016 11:16:56 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3E8A7160ADD for ; Fri, 28 Oct 2016 13:16:55 +0200 (CEST) Received: (qmail 55596 invoked by uid 500); 28 Oct 2016 11:16:54 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 55583 invoked by uid 99); 28 Oct 2016 11:16:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Oct 2016 11:16:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 01E04189B5F for ; Fri, 28 Oct 2016 11:16:54 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id kEYlA1x1LxPP for ; Fri, 28 Oct 2016 11:16:50 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id 87D095FC52 for ; Fri, 28 Oct 2016 11:16:49 +0000 (UTC) Received: (qmail 45097 invoked by uid 99); 28 Oct 2016 11:09:55 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Oct 2016 11:09:55 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 1F9B9E0FA0; Fri, 28 Oct 2016 11:09:55 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: ralphsu@apache.org To: commits@eagle.incubator.apache.org Message-Id: <8dca676ab5c845978d50fb7139a13b41@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: incubator-eagle git commit: EAGLE-683: Improve metadata store performance Date: Fri, 28 Oct 2016 11:09:55 +0000 (UTC) archived-at: Fri, 28 Oct 2016 11:16:56 -0000 Repository: incubator-eagle Updated Branches: refs/heads/master 38d46c8bb -> e3f358c84 EAGLE-683: Improve metadata store performance Currently, we enable periodically schedule in coordinator service itself. If this is enabled, the schedule spec will finally used up metadata storage. So we need to improve metadata store performance by: 1. Disable periodically schedule by default (storage like mysql doesn't have capped feature). 2. For mongodb storage, use capped collection for schedule_specs, policy_snapshots, streamSnapshots, groupSpecs, alertSpecs, monitoredStreams, spoutSpecs (all schedule spec related collection). Author: Zeng, Bryant Reviewer: ralphsu This closes #566 Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/e3f358c8 Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/e3f358c8 Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/e3f358c8 Branch: refs/heads/master Commit: e3f358c8462af497fdc9f4498dbf374599af57e6 Parents: 38d46c8 Author: mizeng Authored: Wed Oct 26 14:22:33 2016 +0800 Committer: Ralph, Su Committed: Fri Oct 28 19:09:59 2016 +0800 ---------------------------------------------------------------------- .../eagle/alert/coordinator/Coordinator.java | 6 ++- .../metadata/impl/MongoMetadataDaoImpl.java | 53 ++++++++++++++++---- .../alert/resource/impl/MongoImplTest.java | 3 -- .../src/test/resources/application-mongo.conf | 3 +- 4 files changed, 50 insertions(+), 15 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e3f358c8/eagle-core/eagle-alert-parent/eagle-alert/alert-coordinator/src/main/java/org/apache/eagle/alert/coordinator/Coordinator.java ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-coordinator/src/main/java/org/apache/eagle/alert/coordinator/Coordinator.java b/eagle-core/eagle-alert-parent/eagle-alert/alert-coordinator/src/main/java/org/apache/eagle/alert/coordinator/Coordinator.java index e0bd5c3..deeeec9 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-coordinator/src/main/java/org/apache/eagle/alert/coordinator/Coordinator.java +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-coordinator/src/main/java/org/apache/eagle/alert/coordinator/Coordinator.java @@ -280,10 +280,12 @@ public class Coordinator { }); scheduleSrv.scheduleAtFixedRate(loader, initDelayMillis, delayMillis, TimeUnit.MILLISECONDS); - // + // disable periodically schedule by default due for the sake of Metadata store performance + /*** scheduleSrv.scheduleAtFixedRate(new CoordinatorTrigger(config, client), CoordinatorTrigger.INIT_PERIODICALLY_TRIGGER_DELAY, CoordinatorTrigger.INIT_PERIODICALLY_TRIGGER_INTERVAL, TimeUnit.MILLISECONDS); - + ***/ + Runtime.getRuntime().addShutdownHook(new Thread(new CoordinatorShutdownHook(scheduleSrv))); LOG.info("Eagle Coordinator started ..."); http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e3f358c8/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/main/java/org/apache/eagle/alert/metadata/impl/MongoMetadataDaoImpl.java ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/main/java/org/apache/eagle/alert/metadata/impl/MongoMetadataDaoImpl.java b/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/main/java/org/apache/eagle/alert/metadata/impl/MongoMetadataDaoImpl.java index f8186ac..088896c 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/main/java/org/apache/eagle/alert/metadata/impl/MongoMetadataDaoImpl.java +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/main/java/org/apache/eagle/alert/metadata/impl/MongoMetadataDaoImpl.java @@ -25,6 +25,8 @@ import com.mongodb.MongoClient; import com.mongodb.MongoClientURI; import com.mongodb.client.MongoCollection; import com.mongodb.client.MongoDatabase; +import com.mongodb.client.MongoIterable; +import com.mongodb.client.model.CreateCollectionOptions; import com.mongodb.client.model.IndexOptions; import com.mongodb.client.model.UpdateOptions; import com.mongodb.client.result.DeleteResult; @@ -59,6 +61,8 @@ public class MongoMetadataDaoImpl implements IMetadataDao { private static final String DB_NAME = "ump_alert_metadata"; private static final Logger LOG = LoggerFactory.getLogger(MongoMetadataDaoImpl.class); private static final ObjectMapper mapper = new ObjectMapper(); + private static final int DEFAULT_CAPPED_MAX_SIZE = 500 * 1024 * 1024; + private static final int DEFAULT_CAPPED_MAX_DOCUMENTS = 20000; static { mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); @@ -66,6 +70,8 @@ public class MongoMetadataDaoImpl implements IMetadataDao { private final String connection; private final MongoClient client; + private final int cappedMaxSize; + private final int cappedMaxDocuments; private MongoDatabase db; private MongoCollection cluster; @@ -91,10 +97,39 @@ public class MongoMetadataDaoImpl implements IMetadataDao { @Inject public MongoMetadataDaoImpl(Config config) { this.connection = config.getString("connection"); + this.cappedMaxSize = config.hasPath("cappedMaxSize") ? config.getInt("cappedMaxSize") : DEFAULT_CAPPED_MAX_SIZE; + this.cappedMaxDocuments = config.hasPath("cappedMaxDocuments") ? config.getInt("cappedMaxDocuments") : DEFAULT_CAPPED_MAX_DOCUMENTS; this.client = new MongoClient(new MongoClientURI(this.connection)); init(); } + private boolean isCollectionExists(String collectionName) { + boolean result = false; + MongoIterable allCollections = db.listCollectionNames(); + for ( String collection : allCollections ) { + if (collection.equals(collectionName)) { + result = true; + break; + } + } + + return result; + } + + private MongoCollection getCollection(String collectionName) { + // first check if collection exists, if not then create a new collection with cappedSize + if (!isCollectionExists(collectionName)) { + CreateCollectionOptions option = new CreateCollectionOptions(); + option.capped(true); + option.maxDocuments(cappedMaxDocuments); + option.sizeInBytes(cappedMaxSize); + db.createCollection(collectionName, option); + } + + return db.getCollection(collectionName); + + } + private void init() { db = client.getDatabase(DB_NAME); IndexOptions io = new IndexOptions().background(true).name("nameIndex"); @@ -138,10 +173,10 @@ public class MongoMetadataDaoImpl implements IMetadataDao { BsonDocument doc1 = new BsonDocument(); IndexOptions io1 = new IndexOptions().background(true).name("versionIndex"); doc1.append("version", new BsonInt32(1)); - scheduleStates = db.getCollection("schedule_specs"); + scheduleStates = getCollection("schedule_specs"); scheduleStates.createIndex(doc1, io1); - spoutSpecs = db.getCollection("spoutSpecs"); + spoutSpecs = getCollection("spoutSpecs"); { IndexOptions ioInternal = new IndexOptions().background(true).name("topologyIdIndex"); BsonDocument docInternal = new BsonDocument(); @@ -149,7 +184,7 @@ public class MongoMetadataDaoImpl implements IMetadataDao { spoutSpecs.createIndex(docInternal, ioInternal); } - alertSpecs = db.getCollection("alertSpecs"); + alertSpecs = getCollection("alertSpecs"); { IndexOptions ioInternal = new IndexOptions().background(true).name("topologyNameIndex"); BsonDocument docInternal = new BsonDocument(); @@ -157,22 +192,22 @@ public class MongoMetadataDaoImpl implements IMetadataDao { alertSpecs.createIndex(docInternal, ioInternal); } - groupSpecs = db.getCollection("groupSpecs"); + groupSpecs = getCollection("groupSpecs"); groupSpecs.createIndex(doc1, io1); - publishSpecs = db.getCollection("publishSpecs"); + publishSpecs = getCollection("publishSpecs"); publishSpecs.createIndex(doc1, io1); - policySnapshots = db.getCollection("policySnapshots"); + policySnapshots = getCollection("policySnapshots"); policySnapshots.createIndex(doc1, io); - streamSnapshots = db.getCollection("streamSnapshots"); + streamSnapshots = getCollection("streamSnapshots"); streamSnapshots.createIndex(doc1, io); - monitoredStreams = db.getCollection("monitoredStreams"); + monitoredStreams = getCollection("monitoredStreams"); monitoredStreams.createIndex(doc1, io); - assignments = db.getCollection("assignments"); + assignments = getCollection("assignments"); assignments.createIndex(doc1, io1); } http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e3f358c8/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/java/org/apache/eagle/service/alert/resource/impl/MongoImplTest.java ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/java/org/apache/eagle/service/alert/resource/impl/MongoImplTest.java b/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/java/org/apache/eagle/service/alert/resource/impl/MongoImplTest.java index ff83b80..213943d 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/java/org/apache/eagle/service/alert/resource/impl/MongoImplTest.java +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/java/org/apache/eagle/service/alert/resource/impl/MongoImplTest.java @@ -203,9 +203,6 @@ public class MongoImplTest { Assert.assertEquals(200, result.code); List assigns = dao.listStreams(); Assert.assertEquals(1, assigns.size()); - dao.removeStream("stream"); - assigns = dao.listStreams(); - Assert.assertEquals(0, assigns.size()); } // alert { http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e3f358c8/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/resources/application-mongo.conf ---------------------------------------------------------------------- diff --git a/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/resources/application-mongo.conf b/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/resources/application-mongo.conf index fbb99de..bdc632c 100644 --- a/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/resources/application-mongo.conf +++ b/eagle-core/eagle-alert-parent/eagle-alert/alert-metadata-parent/alert-metadata/src/test/resources/application-mongo.conf @@ -16,6 +16,7 @@ { "datastore": { "metadataDao": "org.apache.eagle.alert.metadata.impl.InMemMetadataDaoImpl", - "connection": "mongodb://localhost:27017" + "connection": "mongodb://localhost:27017", + "cappedSize": 20000 } }