sdap-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eamonf...@apache.org
Subject [incubator-sdap-nexus] branch bug_fixes updated: Readme
Date Mon, 07 Dec 2020 17:33:04 GMT
This is an automated email from the ASF dual-hosted git repository.

eamonford pushed a commit to branch bug_fixes
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git


The following commit(s) were added to refs/heads/bug_fixes by this push:
     new cb0dea4  Readme
cb0dea4 is described below

commit cb0dea45eb000ebd4db4b98311f553b290e6e5ea
Author: Eamon Ford <eamon.d.ford@jpl.nasa.gov>
AuthorDate: Mon Dec 7 11:32:47 2020 -0600

    Readme
---
 helm/README.md | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 62 insertions(+), 10 deletions(-)

diff --git a/helm/README.md b/helm/README.md
index 0bf629e..a992183 100644
--- a/helm/README.md
+++ b/helm/README.md
@@ -27,8 +27,9 @@ The helm chart deploys all the required components of the NEXUS application
(Spa
     - [RabbitMQ Parameters](#rabbitmq-parameters)
     - [Ingress Parameters](#ingress-parameters)
   - [Ingestion](#ingestion)
-    - [Ingesting Granules Stored on S3](#ingesting-granules-stored-on-s3)
-    - [Ingesting Granules Stored on an NFS Host](#ingesting-granules-stored-on-an-nfs-host)
+    - [Ingesting from a Local Directory](#ingesting-from-a-local-directory)
+    - [Ingesting from S3](#ingesting-from-s3)
+    - [Ingesting from an NFS Host](#ingesting-from-an-nfs-host)
   - [Other Configuration Examples](#other-configuration-examples)
     - [Restricting Pods to Specific Nodes](#restricting-pods-to-specific-nodes)
     - [Persistence](#persistence)
@@ -245,12 +246,63 @@ See the [nginx-ingress Helm chart docs](https://github.com/helm/charts/tree/mast
 
 ## Ingestion
 
-SDAP supports ingestion granules from the local filesystem, an S3 bucket, or an NFS. 
-> **Note:** it is not yet possible to configure SDAP to ingest from both S3 and a local
or NFS directory simultaneously
+SDAP supports ingesting granules from either a local directory, an AWS S3 bucket, or an NFS
server. (It is not yet possible to configure SDAP to ingest from multiple
+of these sources simultanously.)
 
-### Ingesting Granules Stored on S3
+### Ingesting from a Local Directory
 
-SDAP supports ingesting granules that are stored in an AWS S3 bucket. To enable this, you
must provide the name of the S3 bucket to read from, as well as the S3 credentials as environment
variables.
+To ingest granules that are stored on the local filesystem, you must provide the path to
the directory where the granules are stored. This directory will be mounted as a volume
+in the ingestion pods.
+
+The following is an example configuration for ingesting granules from a local directory:

+
+```yaml
+ingestion:
+  granules:
+    path: /share/granules
+    mountPath: /data
+```
+
+The `ingestion.granules.mountPath` property sets the mount path in the ingestion pods where
the granule directory will be mounted. **The root directory of the `path` property of all
collection entries in the collections config must match this value.** This is because the
`path` property of collections in the collections config describes to the 
+ingestion pods where to find the mounted granules.
+
+The following is an example of a collections config to be used with the NFS ingestion configuration
above: 
+
+```yaml
+# collections.yml
+
+collections:
+  - id: "CSR-RL06-Mascons_LAND"
+    path: "/data/CSR-RL06-Mascons-land/CSR_GRACE_RL06_Mascons_v01-land.nc" 
+    priority: 1
+    projection: Grid
+    dimensionNames:
+      latitude: lat
+      longitude: lon
+      time: time
+      variable: lwe_thickness
+    slices:
+      time: 1
+      lat: 60
+      lon: 60
+  - id: "TELLUS_GRAC-GRFO_MASCON_CRI_GRID_RL06_V2_LAND"
+    path: "/data/grace-fo-land/"
+    priority: 1
+    projection: Grid
+    dimensionNames:
+      latitude: lat
+      longitude: lon
+      time: time
+      variable: lwe_thickness
+    slices:
+      time: 1
+      lat: 60
+      lon: 60
+```
+
+### Ingesting from S3
+
+To ingest granules that are stored in an S3 bucket, you must provide the name of the S3 bucket
to read from, as well as the S3 credentials as environment variables.
 (See the [AWS docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
for the list of possible AWS credentials environment variables.)
 
 The following is an example configuration that enables ingestion from S3: 
@@ -301,9 +353,9 @@ collections:
       lon: 60
 ```
 
-### Ingesting Granules Stored on an NFS Host
+### Ingesting from an NFS Host
 
-SDAP supports ingesting granules that are stored on an NFS host. To enable this, you must
provide the NFS host url, and the path to the directory on the NFS server the granules are
located.
+To ingest granules that are stored on an NFS host, you must provide the NFS host url, and
the path to the directory on the NFS server the granules are located.
 
 The following is an example configuration that enables ingestion from an NFS host: 
 
@@ -315,9 +367,9 @@ ingestion:
     mountPath: /data
 ```
 
-When ingesting from either NFS or the local filesystem, the `path` property of all collection
entries in the collections config should have the value from `ingestion.granules.mountPath`
as the root. 
-This is because granule directory on the NFS host will be mounted as a volume onto the SDAP
ingestion pods at `ingestion.granules.mountPath`, and the `path` property of collections in
the collections config describes to the 
+The `ingestion.granules.mountPath` property sets the mount path in the ingestion pods where
the granule directory will be mounted. **The root directory of the `path` property of all
collection entries in the collections config must match this value.** This is because the
`path` property of collections in the collections config describes to the 
 ingestion pods where to find the mounted granules.
+
 The following is an example of a collections config to be used with the NFS ingestion configuration
above: 
 
 ```yaml


Mime
View raw message