sdap-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eamonf...@apache.org
Subject [incubator-sdap-nexus] branch bug_fixes updated: Readme updates
Date Thu, 03 Dec 2020 21:08:23 GMT
This is an automated email from the ASF dual-hosted git repository.

eamonford pushed a commit to branch bug_fixes
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git


The following commit(s) were added to refs/heads/bug_fixes by this push:
     new dcc47cb  Readme updates
dcc47cb is described below

commit dcc47cb8ee42099f99144b339d04b94d93e3bf82
Author: Eamon Ford <eamonford@gmail.com>
AuthorDate: Thu Dec 3 13:08:05 2020 -0800

    Readme updates
---
 helm/README.md   | 79 ++++++++++++++++++++++++++++++++++++++++++++++----------
 helm/values.yaml | 14 +++++-----
 2 files changed, 73 insertions(+), 20 deletions(-)

diff --git a/helm/README.md b/helm/README.md
index 529472f..1363981 100644
--- a/helm/README.md
+++ b/helm/README.md
@@ -19,15 +19,17 @@ The helm chart deploys all the required components of the NEXUS application
(Spa
     - [Option 1: Local deployment with ingress enabled](#option-1-local-deployment-with-ingress-enabled)
     - [Option 2: No ingress enabled](#option-2-no-ingress-enabled)
   - [Uninstalling the Chart](#uninstalling-the-chart)
-  - [Configuration](#configuration)
+  - [Parameters](#parameters)
     - [SDAP Webapp (Analyis) Parameters](#sdap-webapp-analyis-parameters)
     - [SDAP Ingestion Parameters](#sdap-ingestion-parameters)
     - [Cassandra Parameters](#cassandra-parameters)
     - [Solr/Zookeeper Parameters](#solrzookeeper-parameters)
     - [RabbitMQ Parameters](#rabbitmq-parameters)
     - [Ingress Parameters](#ingress-parameters)
-  - [Restricting Pods to Specific Nodes](#restricting-pods-to-specific-nodes)
-  - [Notes about Persistent Volume Storage Classes](#notes-about-persistent-volume-storage-classes)
+  - [Configuration Examples](#configuration-examples)
+    - [Restricting Pods to Specific Nodes](#restricting-pods-to-specific-nodes)
+    - [Persistence](#persistence)
+    - [Ingesting Granules Stored on S3](#ingesting-granules-stored-on-s3)
 
 ## Prerequisites
 
@@ -100,9 +102,9 @@ To uninstall/delete the `nexus` deployment:
 The command removes all the Kubernetes components associated with the chart and deletes the
release.
 
 
-## Configuration
+## Parameters
 
-There are two ways to override configuration values for the chart. The first is to use the
`--set` flag when installing the chart, for example:
+There are two ways to override configuration parameters for the chart. The first is to use
the `--set` flag when installing the chart, for example:
 
     $ helm install nexus incubator-sdap-nexus/helm --namespace=sdap --dependency-update --set
cassandra.replicas=3 --set solr.replicas=3
 
@@ -240,7 +242,9 @@ See the [nginx-ingress Helm chart docs](https://github.com/helm/charts/tree/mast
 | `nginx-ingress.defaultBackend.enabled`| Use default backend component	     | `false`  
                                  |
 
 
-## Restricting Pods to Specific Nodes
+## Configuration Examples
+
+### Restricting Pods to Specific Nodes
 
 Sometimes you may wish to restrict pods to run on specific nodes, for example if you have
"UAT" and "SIT" nodes within the same cluster. You can configure 
 affinity and tolerations for all the components, as in the following example:
@@ -327,16 +331,13 @@ solr:
               - uat
 ```
 
-## Notes about Persistent Volume Storage Classes
+### Persistence
 
 The SDAP Helm chart uses [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
for RabbitMQ, Solr, Zookeeper, Cassandra, and optionally the Collection Manager ingestion
component (if Solr ingestion history is disabled).
 In most use cases you will want to use the same storage class for all of these components.
 
-
-**Example**
-
-If you are deploying SDAP on AWS and you want to use
-[EBS GP2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#EBSVolumeTypes_gp2)
volumes for persistence storage, you would need to use the following configuration values
for the SDAP Helm chart:
+For example, if you are deploying SDAP on AWS and you want to use [EBS gp2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#EBSVolumeTypes_gp2)
volumes for persistence storage, you would need to set the following 
+configuration values for the SDAP Helm chart:
 
 ```yaml
 rabbitmq:
@@ -356,6 +357,58 @@ solr:
 
 ingestion:
   history:
-    storageClass: hostpath # This is only needed if Solr ingestion history is disabled, as
follows:
+    storageClass: gp2 # This is only needed if Solr ingestion history is disabled, as follows:
     solrEnabled: false 
+```
+
+### Ingesting Granules Stored on S3
+
+SDAP supports ingesting granules that are stored in an AWS S3 bucket. To enable this, you
must provide the name of the S3 bucket to read from, as well as the S3 credentials as environment
variables.
+(See the [AWS docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
for the list of possible AWS credentials environment variables.)
+> **Note:** it is not yet possible to configure SDAP to ingest from both S3 and a local
or NFS directory simultaneously.
+
+The following is an example configuration that enables ingestion from S3: 
+
+```yaml
+ingestion:
+  granules:
+    s3:
+      bucket: my-nexus-bucket
+      awsCredsEnvs:
+        AWS_ACCESS_KEY_ID: my-secret
+        AWS_SECRET_ACCESS_KEY: my-secret
+        AWS_DEFAULT_REGION: us-west-2
+```
+
+When S3 ingestion is enabled, the `path` property of all collection entries in the collections
config must be an S3 path or prefix. Due to S3 limitations, wildcards are not supported. The
following
+is an example of a collections config to be used with the S3 ingestion configuration above:

+
+```yaml
+collections:
+  - id: "CSR-RL06-Mascons_LAND"
+    path: "s3://my-nexus-bucket/CSR-RL06-Mascons-land/CSR_GRACE_RL06_Mascons_v01-land.nc"
# full S3 path
+    priority: 1
+    projection: Grid
+    dimensionNames:
+      latitude: lat
+      longitude: lon
+      time: time
+      variable: lwe_thickness
+    slices:
+      time: 1
+      lat: 60
+      lon: 60
+  - id: "TELLUS_GRAC-GRFO_MASCON_CRI_GRID_RL06_V2_LAND"
+    path: "s3://nexus-ingest-grace/grace-fo-land/" # S3 prefix
+    priority: 1
+    projection: Grid
+    dimensionNames:
+      latitude: lat
+      longitude: lon
+      time: time
+      variable: lwe_thickness
+    slices:
+      time: 1
+      lat: 60
+      lon: 60
 ```
\ No newline at end of file
diff --git a/helm/values.yaml b/helm/values.yaml
index be4b815..102e304 100644
--- a/helm/values.yaml
+++ b/helm/values.yaml
@@ -70,13 +70,13 @@ ingestion:
     ## the granule files are stored.
     path: /var/lib/sdap/granules
 
-    # s3:
-    #   bucket: nexus-ingest
-    #   awsCredsEnvs:
-    #     AWS_ACCESS_KEY_ID: secret
-    #     AWS_SECRET_ACCESS_KEY: secret
-    #     AWS_SESSION_TOKEN: secret
-    #     AWS_DEFAULT_REGION: us-west-2
+    s3:
+      bucket: 
+      awsCredsEnvs:
+      # AWS_ACCESS_KEY_ID: secret
+      # AWS_SECRET_ACCESS_KEY: secret
+      # AWS_SESSION_TOKEN: secret
+      # AWS_DEFAULT_REGION: us-west-2
 
 
   ## Where to find the Collections Config file


Mime
View raw message