pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zhai...@apache.org
Subject [incubator-pulsar] branch master updated: [documentation] Cherry-pick tiered storage documentation to new website (#2364)
Date Wed, 15 Aug 2018 05:24:10 GMT
This is an automated email from the ASF dual-hosted git repository.

zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
     new f352203  [documentation] Cherry-pick tiered storage documentation to new website
(#2364)
f352203 is described below

commit f352203d1a21752ec07eea8c216a3d62fa7cab04
Author: Sijie Guo <guosijie@gmail.com>
AuthorDate: Tue Aug 14 22:24:07 2018 -0700

    [documentation] Cherry-pick tiered storage documentation to new website (#2364)
    
    ### Motivation
    
    Cherry-pick #2152
---
 site2/docs/cookbooks-tiered-storage.md | 125 +++++++++++++++++++++++++++------
 1 file changed, 105 insertions(+), 20 deletions(-)

diff --git a/site2/docs/cookbooks-tiered-storage.md b/site2/docs/cookbooks-tiered-storage.md
index c0ec597..5d75679 100644
--- a/site2/docs/cookbooks-tiered-storage.md
+++ b/site2/docs/cookbooks-tiered-storage.md
@@ -6,6 +6,11 @@ sidebar_label: Tiered Storage
 
 Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded to long term
storage, thereby freeing up space in BookKeeper and reducing storage costs. This cookbook
walks you through using tiered storage in your Pulsar cluster.
 
+Tiered storage currently uses [Apache Jclouds](https://jclouds.apache.org) to supports
+[Amazon S3](https://aws.amazon.com/s3/) and [Google Cloud Storage](https://cloud.google.com/storage/)(GCS
for short)
+for long term storage. With Jclouds, it is easy to add support for more
+[cloud storage providers](https://jclouds.apache.org/reference/providers/#blobstore-providers)
in the future.
+
 ## When should I use Tiered Storage?
 
 Tiered storage should be used when you have a topic for which you want to keep a very long
backlog for a long time. For example, if you have a topic containing user actions which you
use to train your recommendation systems, you may want to keep that data for a long time,
so that if you change your recommendation algorithm you can rerun it against your full user
history.
@@ -18,44 +23,62 @@ A topic in Pulsar is backed by a log, known as a managed ledger. This
log is com
 
 The Tiered Storage offloading mechanism takes advantage of this segment oriented architecture.
When offloading is requested, the segments of the log are copied, one-by-one, to tiered storage.
All segments of the log, apart from the segment currently being written to can be offloaded.
 
-## Amazon S3
+On the broker, the administrator must configure the bucket and credentials for the cloud
storage service.
+The configured bucket must exist before attempting to offload. If it does not exist, the
offload operation will fail.
 
-Tiered storage currently supports S3 for long term storage. On the broker, the administrator
must configure a S3 bucket and the AWS region where the bucket exists. Offloaded data will
be placed into this bucket.
+Pulsar uses multi-part objects to upload the segment data. It is possible that a broker could
crash while uploading the data.
+We recommend you add a life cycle rule your bucket to expire incomplete multi-part upload
after a day or two to avoid
+getting charged for incomplete uploads.
 
-The configured S3 bucket must exist before attempting to offload. If it does not exist, the
offload operation will fail.
+## Configuring the offload driver
 
-Pulsar users multipart objects to update the segment data. It is possible that a broker could
crash while uploading the data. We recommend you add a lifecycle rule your S3 bucket to expire
incomplete multipart upload after a day or two to avoid getting charged for incomplete uploads.
+Offloading is configured in ```broker.conf```. 
 
-### Configuring the broker
+At a minimum, the administrator must configure the driver, the bucket and the authenticating
credentials.
+There is also some other knobs to configure, like the bucket region, the max block size in
backed storage, etc.
 
-Offloading is configured in ```broker.conf```. 
+Currently we support driver of types:
 
-At a minimum, the user must configure the driver, the region and the bucket.
+- `aws-s3`: [Simple Cloud Storage Service](https://aws.amazon.com/s3/)
+- `google-cloud-storage`: [Google Cloud Storage](https://cloud.google.com/storage/)
+
+> Driver names are case-insensitive for driver's name. There is a third driver type, `s3`,
which is identical to `aws-s3`,
+> though it requires that you specify an endpoint url using `s3ManagedLedgerOffloadServiceEndpoint`.
This is useful if
+> using a S3 compatible data store, other than AWS.
 
 ```conf
-managedLedgerOffloadDriver=S3
-s3ManagedLedgerOffloadRegion=eu-west-3
-s3ManagedLedgerOffloadBucket=pulsar-topic-offload
+managedLedgerOffloadDriver=aws-s3
 ```
 
-It is also possible to specify the s3 endpoint directly, using `s3ManagedLedgerOffloadServiceEndpoint`.
This is useful if you are using a non-AWS storage service which provides an S3 compatible
API. 
+### "aws-s3" Driver configuration
 
-> If the endpoint is specified directly, then the region must _not_ be set.
+#### Bucket and Region
 
-> The broker.conf of all brokers must have the same configuration for driver, region and
bucket for offload to avoid data becoming unavailable as topics move from one broker to another.
+Buckets are the basic containers that hold your data.
+Everything that you store in Cloud Storage must be contained in a bucket.
+You can use buckets to organize your data and control access to your data,
+but unlike directories and folders, you cannot nest buckets.
 
-Pulsar also provides some knobs to configure the size of requests sent to S3.
+```conf
+s3ManagedLedgerOffloadBucket=pulsar-topic-offload
+```
 
-- `s3ManagedLedgerOffloadMaxBlockSizeInBytes` configures the maximum size of a "part" sent
during a multipart upload. This cannot be smaller than 5MB. Default is 64MB.
-- `s3ManagedLedgerOffloadReadBufferSizeInBytes` configures the block size for each individual
read when reading back data from S3. Default is 1MB.
+Bucket Region is the region where bucket located. Bucket Region is not a required
+but a recommended configuration. If it is not configured, It will use the default region.
 
-In both cases, these should not be touched unless you know what you are doing.
+With AWS S3, the default region is `US East (N. Virginia)`. Page
+[AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) contains
more information.
 
-> The broker must be rebooted for any changes in the configuration to take effect.
+```conf
+s3ManagedLedgerOffloadRegion=eu-west-3
+```
 
-### Authenticating with S3
+#### Authentication with AWS
 
-To be able to access S3, you need to authenticate with S3. Pulsar does not provide any direct
means of configuring authentication for S3, but relies on the mechanisms supported by the
[DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html).
+To be able to access AWS S3, you need to authenticate with AWS S3.
+Pulsar does not provide any direct means of configuring authentication for AWS S3,
+but relies on the mechanisms supported by the
+[DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html).
 
 Once you have created a set of credentials in the AWS IAM console, they can be configured
in a number of ways.
 
@@ -87,6 +110,68 @@ If you are running in EC2 you can also use instance profile credentials,
provide
 
 > The broker must be rebooted for credentials specified in pulsar_env to take effect.
 
+#### Configuring the size of block read/write
+
+Pulsar also provides some knobs to configure the size of requests sent to AWS S3.
+
+- ```s3ManagedLedgerOffloadMaxBlockSizeInBytes```  configures the maximum size of
+  a "part" sent during a multipart upload. This cannot be smaller than 5MB. Default is 64MB.
+- ```s3ManagedLedgerOffloadReadBufferSizeInBytes``` configures the block size for
+  each individual read when reading back data from AWS S3. Default is 1MB.
+
+In both cases, these should not be touched unless you know what you are doing.
+
+### "google-cloud-storage" Driver configuration
+
+Buckets are the basic containers that hold your data. Everything that you store in
+Cloud Storage must be contained in a bucket. You can use buckets to organize your data and
+control access to your data, but unlike directories and folders, you cannot nest buckets.
+
+```conf
+gcsManagedLedgerOffloadBucket=pulsar-topic-offload
+```
+
+Bucket Region is the region where bucket located. Bucket Region is not a required but
+a recommended configuration. If it is not configured, It will use the default region.
+
+Regarding GCS, buckets are default created in the `us multi-regional location`, 
+page [Bucket Locations](https://cloud.google.com/storage/docs/bucket-locations) contains
more information.
+
+```conf
+gcsManagedLedgerOffloadRegion=europe-west3
+```
+
+#### Authentication with GCS
+
+The administrator needs to configure `gcsManagedLedgerOffloadServiceAccountKeyFile` in `broker.conf`
+for the broker to be able to access the GCS service. `gcsManagedLedgerOffloadServiceAccountKeyFile`
is
+a Json file, containing the GCS credentials of a service account.
+[Service Accounts section of this page](https://support.google.com/googleapi/answer/6158849)
contains
+more information of how to create this key file for authentication. More information about
google cloud IAM
+is available [here](https://cloud.google.com/storage/docs/access-control/iam).
+
+Usually these are the steps to create the authentication file:
+1. Open the API Console Credentials page.
+2. If it's not already selected, select the project that you're creating credentials for.
+3. To set up a new service account, click New credentials and then select Service account
key.
+4. Choose the service account to use for the key.
+5. Download the service account's public/private key as a JSON file that can be loaded by
a Google API client library.
+
+```conf
+gcsManagedLedgerOffloadServiceAccountKeyFile="/Users/hello/Downloads/project-804d5e6a6f33.json"
+```
+
+#### Configuring the size of block read/write
+
+Pulsar also provides some knobs to configure the size of requests sent to GCS.
+
+- ```gcsManagedLedgerOffloadMaxBlockSizeInBytes``` configures the maximum size of a "part"
sent
+  during a multipart upload. This cannot be smaller than 5MB. Default is 64MB.
+- ```gcsManagedLedgerOffloadReadBufferSizeInBytes``` configures the block size for each individual
+  read when reading back data from GCS. Default is 1MB.
+
+In both cases, these should not be touched unless you know what you are doing.
+
 ## Configuring offload to run automatically
 
 Namespace policies can be configured to offload data automatically once a threshold is reached.
The threshold is based on the size of data that the topic has stored on the pulsar cluster.
Once the topic reaches the threshold, an offload operation will be triggered. Setting a negative
value to the threshold will disable automatic offloading. Setting the threshold to 0 will
cause the broker to offload data as soon as it possiby can.


Mime
View raw message