bookkeeper-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [bookkeeper] branch master updated: BP-29: Metadata API module
Date Tue, 06 Feb 2018 06:12:54 GMT
This is an automated email from the ASF dual-hosted git repository.

sijie pushed a commit to branch master
in repository

The following commit(s) were added to refs/heads/master by this push:
     new 011bb65  BP-29: Metadata API module
011bb65 is described below

commit 011bb652363b471e52e082cd9a13ff6a31375525
Author: Sijie Guo <>
AuthorDate: Mon Feb 5 22:12:46 2018 -0800

    BP-29: Metadata API module
    Descriptions of the changes in this PR:
    Related to BP-28 (#1113), this proposal is to propose how we want to organize the metadata
modules, to support different metadata storage implementation.
    Master Ticket: #1123
    Author: Sijie Guo <>
    Reviewers: Jia Zhai <None>, Sijie Guo <>, Venkateswararao
Jujjuri (JV) <None>
    This closes #1117 from sijie/BP-29-metadata-store-api
 site/bps/ | 87 +++++++++++++++++++++++++++++
 site/community/      |  3 +-
 2 files changed, 89 insertions(+), 1 deletion(-)

diff --git a/site/bps/ b/site/bps/
new file mode 100644
index 0000000..9a9b887
--- /dev/null
+++ b/site/bps/
@@ -0,0 +1,87 @@
+title: "BP-29: Metadata API module"
+state: 'Accepted'
+release: "N/A"
+### Motivation
+We have already abstracted all the metadata operations into interfaces. And all the bookkeeper
implementations only reply on metadata interfaces,
+rather than depending on zookeeper. This proposal is to organize the metadata interfaces
and its implementations in a separate module and make
+bookkeeper implementation only depends on metadata interfaces, not depends on zookeeper.
This would a few benefits:
+- It allows supporting different metadata storages, without bringing in dependencies of metadata
store implementation directly into
+  bookkeeper-server module. The development of different metadata storage can be done without
interleaving with each other.
+- It would define a clean module dependency between bookkeeper implementation and metadata
api, and how bookkeeper load a different metadata
+  implementation.
+### Public Interfaces
+A more generic setting `metadataServiceUri` is introduced for replacing implementation specific
settings `zkServers` and `zkLedgersRootPath`.
+A metadata service uri defines the location of a metadata storage. In zookeeper based implementation,
the metadata service url will be
+This new setting in bookie configuration will be like as below:
+If we eventually support Etcd as one of the metadata storages. Then the setting in bookie
configuration to use Etcd will be:
+### Proposed Changes
+#### Configuration
+This BP proposes introducing a more generic metadata setting `metadataServiceUri` to replace
implementation specific settings
+`zkServers` and `zkLedgersRootPath`. All implementation specific settings should be considered
moving to implementation itself.
+The `metadataServiceUri` can also be used for replacing the need of configuring `ledgerManagerFactoryClass`,
`registrationClientClass` and
+`registrationManagerClass`. It is unnecessarily complicated to configure multiple settings
to load a specific metadata implementation.
+We can just use the `scheme` field in `metadataServiceUri` to resolve which metadata implementation
to use. Using uri to resolve
+different driver or implementation is commonly seen at java world, for example, jdbc to support
different database drivers. Also, distributedlog
+uses this pattern to load different metadata driver.
+So in zookeeper based metadata implementation, the metadata service uri can be:
+- `zk+flat://`: the scheme is "zk+flat". it means a zookeeper base metadata
implementation and it uses flat ledger manager.
+- `zk+hierarchical://`: the scheme is "zk+hierarchical". it means a zookeeper
base metadata implementation and it
+  uses hierarchical ledger manager.
+- `zk+longhierarchical://`: the scheme is "zk+longhierarchical". it means
a zookeeper base metadata implementation and it
+  uses long hierarchical ledger manager.
+#### Metadata Stores
+Introduce a new directory called `metadata-stores` for storing all the metadata related modules.
Under this directory, it will have following modules:
+- `api`: it is the metadata api module `metadata-store-api`. It contains all the files defining
the metadata interfaces.
+- `zookeeper`: it is the zookeeper implementation module `metadata-store-zookeeper`. It contains
all the files that implementing the metadata interfaces
+  using zookeeper.
+If a new metadata implementation is added, a new directory will be created under `metadata-stores`
to contain the implementation. For example, if we
+are adding `Etcd` as the metadata store. A `Etcd` directory will be created under `metadata-stores/etcd`
to store the files that implement metadata
+interfaces using etcd client. And its module is named as `metadata-store-etcd`.
+We then change bookkeeper-server to depend on `metadata-store-api` only.
+This approach is same as other pluggable modules `stats-api` and `http-server`.
+### Compatibility, Deprecation, and Migration Plan
+No compatibility concern at this moment. New setting is introduced and old settings will
still continue to work.
+No immediate deprecation.
+No migration is needed.
+### Test Plan
+This proposal is mostly around refactor. So existing test cases would cover this.
+### Rejected Alternatives
diff --git a/site/community/ b/site/community/
index 754436e..25c586c 100644
--- a/site/community/
+++ b/site/community/
@@ -85,7 +85,7 @@ using Google Doc.
 This section lists all the _bookkeeper proposals_ made to BookKeeper.
-*Next Proposal Number: 28*
+*Next Proposal Number: 30*
 ### Inprogress
@@ -99,6 +99,7 @@ Proposal | State
 [BP-18: LedgerType, Flags and StorageHints](
| Accepted
 [BP-26: Move distributedlog library as part of bookkeeper](../../bps/BP-26-move-distributedlog-core-library)
| Accepted
 [BP-27: New BookKeeper CLI](../../bps/BP-27-new-bookkeeper-cli) | Draft
+[BP-29: Metadata API module](../../bps/BP-29-metadata-store-api-module) | Draft
 ### Adopted

To stop receiving notification emails like this one, please contact

View raw message