From commits-return-19374-archive-asf-public=cust-asf.ponee.io@jackrabbit.apache.org Sat Jan 13 00:05:59 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 01F10180621 for ; Sat, 13 Jan 2018 00:05:59 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E5E8B160C42; Fri, 12 Jan 2018 23:05:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 10AE9160C20 for ; Sat, 13 Jan 2018 00:05:57 +0100 (CET) Received: (qmail 86368 invoked by uid 500); 12 Jan 2018 23:05:57 -0000 Mailing-List: contact commits-help@jackrabbit.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@jackrabbit.apache.org Delivered-To: mailing list commits@jackrabbit.apache.org Received: (qmail 86359 invoked by uid 99); 12 Jan 2018 23:05:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Jan 2018 23:05:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C45B11808DB; Fri, 12 Jan 2018 23:05:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -8.711 X-Spam-Level: X-Spam-Status: No, score=-8.711 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id fnMZS_Uv1arS; Fri, 12 Jan 2018 23:05:54 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id B05B55F297; Fri, 12 Jan 2018 23:05:53 +0000 (UTC) Received: from moin-vm.apache.org (moin-vm.apache.org [163.172.69.106]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id C8718E0352; Fri, 12 Jan 2018 23:05:50 +0000 (UTC) Received: from moin-vm.apache.org (localhost [IPv6:::1]) by moin-vm.apache.org (ASF Mail Server at moin-vm.apache.org) with ESMTP id 50E8B80220; Sat, 13 Jan 2018 00:05:48 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Apache Wiki To: Apache Wiki Date: Fri, 12 Jan 2018 23:05:47 -0000 Message-ID: <151579834737.29809.5101987273869482231@moin-vm.apache.org> Subject: =?utf-8?q?=5BJackrabbit_Wiki=5D_Update_of_=22Composite_Blob_Store=22_by_M?= =?utf-8?q?attRyan?= Auto-Submitted: auto-generated Dear Wiki user, You have subscribed to a wiki page or wiki category on "Jackrabbit Wiki" fo= r change notification. The "Composite Blob Store" page has been changed by MattRyan: https://wiki.apache.org/jackrabbit/Composite%20Blob%20Store?action=3Ddiff&r= ev1=3D12&rev2=3D13 =3D Composite Blob Store =3D - = - NOTE: (Thomas Mueller): Motivation is missing (what problems do we want = to solve). - NOTE: (Matt Ryan): Added motivation section below, is this sufficient or= is more needed here? - = = NOTE: The current status of this component is a '''proposed feature'''. = @@ -22, +18 @@ * Choose different types of storage for different types of binary data i= n Oak based on some configurable criteria. = =3D=3D Technical Details =3D=3D - Configuration specifies which delegate blob stores comprise the composite= blob store, in some order of preference. The exact order is defined using= a strategy pattern implementation (see "Delegate Traversal" below) which c= an be customized by end users. In the case of conflicting delegates where = there are multiple matches, the first matching delegate will generally be t= he one used, although the traversal strategy implementation can provide dif= ferent conflict resolution behavior. + To use the composite blob store, delegate blob stores are configured as d= ata store factories. The configuration of each data store factory must spe= cify the following: + * Any standard configuration required to configure this data store + * A role (any string value) identifying this delegate + * Any configuration pertaining to this data store's role as a delegate + For example, if configuring an S3DataStore as a delegate, a user would: + * Configure standard S3DataStore values, like the access key, secret key,= and bucket name + * Define a role for this data store, e.g. "role=3DS3DS_1" + * Add other configuration, if any, to configure this data store as a dele= gate (like whether it is a readOnly store) = - NOTE: (Thomas Mueller): What storage filter to use if the binary was cre= ated without node (which is actually the normal case). What if the storage = filter changes? Store it in multiple locations? That would require more spa= ce which needs to be avoided (to save cost). - NOTE: (Matt Ryan): I moved this concept to a rejected status for 1.8 and= possible future for > 1.8, let me know if this addresses your concerns for= now. - = + After configuring the delegates, the composite is configured using the PI= D "org.apache.jackrabbit.oak.plugins.blob.datastore.CompositeDataStore". A= single configuration entry is required, which is a listing of the roles th= is composite manages. + For example, suppose there are two delegate data stores. In the configur= ation of one delegate, it defines "role=3DS3DS_1". In the configuration of= the second delegate, it defines "role=3DS3DS_2". To use these two delegat= es, the composite data store configuration would include the line "roles=3D= S3DS_1,S3DS_2". + = =3D=3D=3D Delegate Traversal =3D=3D=3D - "Delegate Traversal" refers to the logic that is used to go through deleg= ates on a read or write request to determine which delegate should be used = for a request. The algorithm used to traverse delegates should be extensib= le with a reasonable default provided. This is effectively an implementati= on of the Strategy pattern. Any user of the system could provide their own= implementation to deliver whatever custom logic they wish to provide. - = - The proposed default implementation is called the Intelligent Delegate Tr= aversal Strategy. It is "intelligent" because it attempts to interpret the= provided configuration and apply a priority based on the interpretation of= the configuration. Another possible default could be e.g. the Simple Dele= gate Traversal Strategy. This strategy simply attempts to use the delegate= s in the raw order they are specified in the configuration, with no logic a= pplied. The Intelligent Delegate Traversal Strategy will probably provide = the most unsurprising results for end users. + "Delegate Traversal" refers to the logic that is used to go through deleg= ates on a read or write request to determine which delegate should be used = for a request. The algorithm used to traverse delegates is extensible. Th= e default implementation is called the Intelligent Delegate Traversal Strat= egy. It is "intelligent" because it attempts to interpret the provided con= figuration and apply a priority based on the interpretation of the configur= ation. Another possible default could be e.g. the Simple Delegate Traversa= l Strategy. This strategy simply attempts to use the delegates in the raw = order they are specified in the configuration, with no logic applied. The = Intelligent Delegate Traversal Strategy will probably provide the most unsu= rprising results for end users. = =3D=3D=3D=3D Intelligent Delegate Traversal Strategy =3D=3D=3D=3D = @@ -39, +40 @@ A delegate may be specified as a read-only delegate, in which case it wil= l not accept any write requests. If it would otherwise have been chosen fo= r a write request, the request will defer to the next delegate in the trave= rsal that matches the request, if any. = =3D=3D=3D=3D=3D=3D Delegate Write Preference =3D=3D=3D=3D=3D=3D - The write algorithm is fairly simple: Excluding delegates that are read-= only, iterate through delegates to select the first that can accept the wri= te, and perform the write. Return the result of this write as the result o= f the composite blob store write. + The write algorithm is fairly simple: Excluding delegates that are read-= only, iterate through delegates to select the first that can accept the wri= te, and perform the write. Return the result of this write as the result o= f the composite blob store write. Priority is given to delegates that alre= ady have a matching blob ID. - = - Note that any other versions of the blob in other delegate blob stores ma= y continue to exist, but would not be used as the most recently written one= would be returned for most read requests. If other versions end up abando= ned they should eventually be garbage collected by the standard data store = garbage collection task. = =3D=3D=3D=3D=3D Reads =3D=3D=3D=3D=3D The composite blob store fulfills read requests by deferring the read to = delegate blob stores. @@ -52, +51 @@ * If no delegate can satisfy the read request, iterate through read-only= delegates and select the first that can satisfy the read request. * Return the result of the delegate read, or an appropriate "not found" = error message if no delegate has a match. = - NOTE: (Thomas Mueller): Bloom filters should be mentioned (are they used= , if yes how, if not why not). What about storing the (original) location i= n the binary reference (as a hint to speed up reading). - = The response to a read request is the result of the first successful read= from a delegate. In this way the top priority result is always selected. = Read-only delegates take lower precedence to writable delegates, as writa= ble delegates may contain more up-to-date information which would be prefer= red. = - =3D=3D=3D=3D Read-Only Delegates =3D=3D=3D=3D + =3D=3D=3D Read-Only Delegates =3D=3D=3D The composite blob store supports the notion of a read-only delegate blob= store. One or more of the delegate blob stores can be configured in read-= only mode, meaning that it can be used to satisfy read requests but not wri= te requests. An example use case for this scenario is where two content re= positories are used, one for a production environment and one for a staging= environment. The staging repository can be configured with a composite bl= ob store that accesses to the production storage location in read-only mode= , so tests can execute in staging using production data without modifying p= roduction data or the production store. = Reads issued to a read-only delegate would be processed as normal. Read-= only delegates are not considered for write requests, causing the composite= blob store to move on to the next delegate to attempt to fulfill the write= request. = - Note that configuring all delegates of a composite blob store would make = the blob store useless for storing blobs and thus should not be an allowed = condition - at least one delegate blob store must not be a read-only delega= te. + Note that configuring all delegates of a composite blob store as read-onl= y delegates would make the blob store useless for storing blobs and thus sh= ould not be an allowed condition - at least one delegate blob store must no= t be a read-only delegate. = + =3D=3D=3D Blob ID / Delegate Mapping =3D=3D=3D + In order to avoid issuing read requests to delegates that do not contain = the blob ID in question, the composite blob store must maintain a mapping o= f each blob ID to the delegate containing it. Bloom filters should be used= for this purpose. This mapping must be created at startup and maintained = as the system runs, and should be rebuilt every time data store garbage col= lection runs (among other things, the filter may need to be resized for the= current number of blob IDs present). - = - =3D=3D=3D Curation =3D=3D=3D - = - Curation is the process of evaluating the blobs in a blob store to determ= ine if that blob store is still the correct location for blobs to reside. I= n the case of the composite blob store, a reason to curate data may be to g= radually move data for which the determination of "correct" location can ch= ange over time. - = - Since writes and reads use the same order of delegate preference, reads w= ill come from the most recently written location, so if a blob can be found= it will always be the most up-to-date version of it. Any other blobs, if = not referenced anywhere else in the system, will eventually be garbage coll= ected. - = - It would be possible to add custom curators, but for now those are reject= ed features (see below). - = - =3D=3D=3D Rejected Features =3D=3D=3D - * For the initial implementation of composite blob store, the concept of= [[Composite Blob Store Storage Filters|storage filters]] was considered bu= t rejected for Oak 1.8. - * There are currently no cold storage blob stores, so the concept of [[C= omposite Blob Store Cold Storage Delegates|cold storage delegates]] was als= o rejected for Oak 1.8. - * Background curation is rejected for Oak 1.8. = =3D=3D Use Cases =3D=3D + There are many possible use cases for the composite blob store. In order= to manage the implementation of the capability, functionality is being add= ed and supported one use case at a time. When this capability is released,= the first supported use case will be the Staging Environment (listed below= ). Other use cases are listed here for reference but are not currently sup= ported. + = =3D=3D=3D Staging Environment =3D=3D=3D The composite blob store can be used to address a production/staging depl= oyment use case, where one Oak repository is the production repository and = another is the staging repository. The production repository accesses a si= ngle blob store. The staging repository uses a composite blob store to acc= ess a staging blob store as well as the production blob store in read-only = mode. Thus staging can serve blobs out of either blob store but can only m= odify blobs on the staging blob store. =20