climate-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jo...@apache.org
Subject [1/4] climate git commit: CLIMATE-582 - Add initial pass at config docs to sphinx build
Date Thu, 12 Mar 2015 15:47:12 GMT
Repository: climate
Updated Branches:
  refs/heads/master 511a7e175 -> 8e69e625f


CLIMATE-582 - Add initial pass at config docs to sphinx build


Project: http://git-wip-us.apache.org/repos/asf/climate/repo
Commit: http://git-wip-us.apache.org/repos/asf/climate/commit/c1c53afc
Tree: http://git-wip-us.apache.org/repos/asf/climate/tree/c1c53afc
Diff: http://git-wip-us.apache.org/repos/asf/climate/diff/c1c53afc

Branch: refs/heads/master
Commit: c1c53afc1875d267dcbeb9f1876f623e7e074688
Parents: 6e73c8a
Author: Michael Joyce <joyce@apache.org>
Authored: Thu Mar 5 13:58:47 2015 -0800
Committer: Michael Joyce <joyce@apache.org>
Committed: Thu Mar 5 13:58:47 2015 -0800

----------------------------------------------------------------------
 docs/source/conf.py                        |  1 +
 docs/source/config/config_overview.rst     | 69 +++++++++++++++++++
 docs/source/config/dataset_information.rst | 89 +++++++++++++++++++++++++
 docs/source/config/evaluation_settings.rst | 56 ++++++++++++++++
 docs/source/config/metrics_information.rst | 12 ++++
 docs/source/config/plots_settings.rst      | 44 ++++++++++++
 docs/source/index.rst                      |  1 +
 7 files changed, 272 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/conf.py
----------------------------------------------------------------------
diff --git a/docs/source/conf.py b/docs/source/conf.py
index 96ad8cb..1095b7b 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -22,6 +22,7 @@ import sys, os
 sys.path.insert(0, os.path.abspath('../../ocw'))
 sys.path.insert(0, os.path.abspath('../../ocw/data_source'))
 sys.path.insert(0, os.path.abspath('../../ocw-ui/backend'))
+sys.path.insert(0, os.path.abspath('../../ocw-config-runner'))
 
 
 # -- General configuration -----------------------------------------------------

http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/config_overview.rst
----------------------------------------------------------------------
diff --git a/docs/source/config/config_overview.rst b/docs/source/config/config_overview.rst
new file mode 100644
index 0000000..dd985cf
--- /dev/null
+++ b/docs/source/config/config_overview.rst
@@ -0,0 +1,69 @@
+Configuration File Overview
+===========================
+
+Apache Open Climate Workbench includes tools for creating and reading configuration files.
Below is an explanation of the general configuration file structure, and in-depth look at
the various configuration options, and explanations of how to use configuration files in an
evaluation.
+
+Getting Started
+---------------
+
+OCW configuration files are written in `YAML <http://yaml.org/>`_ with type annotations
that are supported by the `PyYAML library <http://pyyaml.org/wiki/PyYAMLDocumentation>`_.
Let's look at an example configuration file to get started.
+
+.. code::
+
+    evaluation:
+        temporal_time_delta: 365
+        spatial_regrid_lats: !!python/tuple [-20, 20, 1]
+        spatial_regrid_lons: !!python/tuple [-20, 20, 1]
+
+    datasets:
+        reference:
+            data_source: local
+            file_count: 1
+            path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc
+            variable: tasmax
+
+        targets:
+            - data_source: local
+              file_count: 1
+              path: /tmp/AFRICA_UC-WRF311_CTL_ERAINT_MM_50km-rg_1989-2008_tasmax.nc
+              variable: tasmax
+    metrics:
+        - Bias
+
+    plots:
+        - type: contour
+          results_indeces:
+              - !!python/tuple [0, 0]
+          lats:
+              range_min: -20
+              range_max: 20
+              range_step: 1
+          lons:
+              range_min: -20
+              range_max: 20
+              range_step: 1
+          output_name: wrf_bias_compared_to_knmi
+          optional_args:
+              gridshape: !!python/tuple [6, 6]
+    
+There are 4 main categories for configuration settings: Evaluation, Datasets, Metrics, and
Plots.
+
+Evaluation Settings
+-------------------
+
+This is where you will set evaluation specific settings such as temporal and spatial bin
sizes to use during dataset preparation. Visit the :doc:`Evaluation Settings <evaluation_settings>`
page for additional information.
+
+Dataset Information
+-------------------
+
+The datasets section is where you specify all the datasets to use for an evaluation. You
can specify what the reference dataset should be as well as giving a list of target datasets.
Visit the :doc:`Dataset Information <dataset_information>` page for additional information.
+
+Metrics Information
+-------------------
+
+You will need to load some metrics if you want to get anything useful out of your evaluation.
Visit the :doc:`Metrics Information <metrics_information>` page to learn how to specify
the metrics that should be used in the evaluation.
+
+Plots Settings
+--------------
+
+This is where you specify what plots to make after running the evaluation. The :doc:`Plots
Settings <plots_settings>` page provides examples for the supported plot types.

http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/dataset_information.rst
----------------------------------------------------------------------
diff --git a/docs/source/config/dataset_information.rst b/docs/source/config/dataset_information.rst
new file mode 100644
index 0000000..3c710e3
--- /dev/null
+++ b/docs/source/config/dataset_information.rst
@@ -0,0 +1,89 @@
+Dataset Information
+===================
+
+Dataset configuration information is passed in the **datasets** section of the configuration
file. You can specify one reference dataset and one or more target datasets for your evaluation::
+
+    datasets:
+        reference:
+            data_source: local
+            file_count: 1
+            path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc
+            variable: tasmax
+
+        targets:
+            - data_source: local
+              file_count: 1
+              path: /tmp/AFRICA_UC-WRF311_CTL_ERAINT_MM_50km-rg_1989-2008_tasmax.nc
+              variable: tasmax
+            - data_source: local
+              file_count: 1
+              path: /tmp/AFRICA_UC-WRF311_CTL_ERAINT_MM_50km-rg_1989-2008_tasmax.nc
+              variable: tasmax
+
+Each **data_source** module requires datasets to be passed in a slightly different manner.
Below is an explanation of the format for each of the supported data sources.
+
+Local Dataset
+-------------
+.. code::
+
+    data_source: local
+    file_count: 1
+    path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc
+    variable: tasmax
+
+The **path** flag is the location where the dataset is located on your computer. The **variable**
flag is the variable that should be pulled out of the NetCDF file once it has been opened.
You pass any optional flags that are accepted by :func:`local.load_file` by using the **optional_args**
flag::
+
+    data_source: local
+    file_count: 1
+    path: /tmp/AFRICA_KNMI-RACMO2.2b_CTL_ERAINT_MM_50km_1989-2008_tasmax.nc
+    variable: tasmax
+    optional_args:
+        elevation_index=0,
+        name='foo'
+
+.. note::
+
+    The **file_count** flag is currently not used. It is there to support planned future
functionality. However, you still need to specify it! Leave it as 1.
+
+
+RCMED Dataset
+-------------
+
+.. code::
+    
+    data_source: rcmed
+    dataset_id: 4
+    parameter_id: 32
+    min_lat: -10
+    max_lat: 10
+    min_lon: -20
+    max_lon: 20
+    start_time: 1997-01-01
+    end_time: 2000-01-01
+
+To load a dataset from the Jet Propulsion Laboratory's RCMED you will need to specify the
above flags. The **dataset_id** and **parameter_id** are dataset specific and can be looked
up on the `RCMES project website <https://rcmes.jpl.nasa.gov/content/rcmes-and-data>`_.
Pretty much any common time format will be accepted for the start and end times. However,
just to be safe you should try to stick with something very standard such as `ISO-8601 <http://www.w3.org/TR/NOTE-datetime>`_
formatted time values. You may also pass any optional parameters that are accepted by :func:`rcmed.parameter_dataset`
with the **optional_args** flag.
+
+ESGF Dataset
+------------
+
+In order to load an ESGF dataset you will need to specify the following parameters in addition
to having an ESGF login::
+
+    data_source: esgf
+    dataset_id: obs4MIPs.CNES.AVISO.mon.v1|esg-datanode.jpl.nasa.gov
+    variable: zosStderr
+    esgf_password: totallynotmypassword
+    esgf_username: totallynotmyusername
+
+The **dataset_id** and **variable** flags are ESGF dataset specific. You can locate them
through and ESGF nodes search page. You may also pass any optional parameters that are accepted
by :func:`esgf.load_dataset` with the **optional_args** flag.
+
+
+OpenDAP Dataset
+---------------
+
+A dataset can be downloaded from an OpenDAP URL with the following settings::
+
+    data_source: dap
+    url: http://test.opendap.org/dap/data/nc/sst.mnmean.nc.gz
+    variable: sst
+
+You may also pass any optional parameters that are accepted by :func:`dap.load` with the
**optional_args** flag.

http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/evaluation_settings.rst
----------------------------------------------------------------------
diff --git a/docs/source/config/evaluation_settings.rst b/docs/source/config/evaluation_settings.rst
new file mode 100644
index 0000000..f10973c
--- /dev/null
+++ b/docs/source/config/evaluation_settings.rst
@@ -0,0 +1,56 @@
+Evaluation Settings
+===================
+
+The evaluation settings section of the configuration file allows you to set attributes that
are critical for making adjustments to the loaded datasets prior to an evaluation run. Here
is an example evaluation settings section of a configuration file. Below, we'll look at each
of the configuration options in detail.
+
+.. code::
+
+    evaluation:
+        temporal_time_delta: 365
+        spatial_regrid_lats: !!python/tuple [-20, 20, 1]
+        spatial_regrid_lons: !!python/tuple [-20, 20, 1]
+
+Temporal Rebin
+--------------
+
+It is often necessary to temporally rebin datasets prior to an evaluation. The **temporal_time_delta**
flag is where you can set the **temporal_resolution** parameter for :func:`dataset_processor.temporal_rebin`.
The value that you pass here is interpreted as the number of days to assign to a :class:`datetime.timedelta`
object before running the :func:`dataset_processor.temporal_rebin` function.
+
+.. note::
+
+    This attribute is only useful if you use the configuration data to create an :class:`evaluation.Evaluation`
object with the :func:`evaluation_creation.generate_evaluation_from_config` config parser
function.
+
+Spatial Regrid
+--------------
+
+.. note::
+
+    Some funcitonality here is still in development. Specifically, passing the spatial_regrid_*
flags as lists of values.
+
+If you need to regrid your datasets onto a new lat/lon grid you will need to set the **spatial_regrid_lats**
and **spatial_regrid_lons** options. These will be passed to the :func:`dataset_processor.spatial_regrid`
function along with each dataset. There are two valid ways to pass these parameters. First,
you can pass them as a list of all values::
+
+    evaluation:
+        spatial_regrid_lats: [-10, -5, 0, 5, 10]
+        spatial_regrid_lons: [-10, -5, 0, 5, 10]
+
+This is generally useful if you only need to pass a few parameters or if the sequence isn't
easy to define as a valid **range** in Python. The other option is to pass **range** information
as a tuple. This requires you to use `PyYAML's Python Type Annotations <http://pyyaml.org/wiki/PyYAMLDocumentation#YAMLtagsandPythontypes>`_
but provides a far more compact representation::
+
+    evaluation:
+        spatial_regrid_lats: !!python/tuple [-20, 20, 1]
+        spatial_regrid_lons: !!python/tuple [-20, 20, 1]
+
+Using this style directly maps to a call to :func:`numpy.arange`::
+
+    # spatial_regrid_lats: !!python/tuple [-20, 20, 1] becomes
+    lats = numpy.arange(-20, 20, 1)
+
+Be sure to pay special attention to the end value for your interval. The :func:`numpy.arange`
function does not include the end value in the returned interval.
+
+Subset Bounds
+-------------
+
+In order to subset the datasets down to an area of interest you will need to pass bounds
information::
+
+    evaluation:
+        subset: [-10, 10, -20, 20, "1997-01-01", "2000-01-01"]
+
+Here you're passing the bounding lat/lon box with the first 4 values as well as the valid
temporal range bounds with the starting and end time values. Pretty much any common time format
will be accepted. However, just to be safe you should try to stick with something very standard
such as `ISO-8601 <http://www.w3.org/TR/NOTE-datetime>`_ formatted time values.

http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/metrics_information.rst
----------------------------------------------------------------------
diff --git a/docs/source/config/metrics_information.rst b/docs/source/config/metrics_information.rst
new file mode 100644
index 0000000..1935e87
--- /dev/null
+++ b/docs/source/config/metrics_information.rst
@@ -0,0 +1,12 @@
+Metrics Information
+===================
+
+.. note::
+
+    At the moment, you can only load metrics that are in :mod:`ocw.metrics`. In the future
you will also be able to specify user defined metrics here as well. However, as a work around
you can define your custom metrics in the :mod:`ocw.metrics` module.
+
+You can set the metrics you want to use in the evaluation in the **metrics** section of the
config. You simply need to supply a list of the metric class names that you want to be used::
+
+    metrics:
+        - Bias
+        - TemporalStdDev

http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/config/plots_settings.rst
----------------------------------------------------------------------
diff --git a/docs/source/config/plots_settings.rst b/docs/source/config/plots_settings.rst
new file mode 100644
index 0000000..2959092
--- /dev/null
+++ b/docs/source/config/plots_settings.rst
@@ -0,0 +1,44 @@
+Plots Settings
+==============
+
+Plotting configuration information is passed in the **plots** section of the configuration
file::
+
+    plots:
+        - type: contour
+          results_indeces:
+              - !!python/tuple [0, 0]
+          lats:
+              range_min: -20
+              range_max: 20
+              range_step: 1
+          lons:
+              range_min: -20
+              range_max: 20
+              range_step: 1
+          output_name: wrf_bias_compared_to_knmi
+          optional_args:
+              gridshape: !!python/tuple [6, 6]
+
+Each type of support plot has a different configuration format expected. Each of these are
explained below. Note, most of these will require you to specify what result data you want
included in the plots with the **results_indeces** flag. This relates the format that an Evaluation
object outputs results in. Check the :class:`evaluation.Evaluation` documentation for more
details.
+
+Contour Maps
+-------------
+
+The contour maps config configures data for OCW's contour plotter :func:`plotting.draw_contour_map`::
+
+    type: contour
+          results_indeces:
+              - !!python/tuple [0, 0]
+          lats:
+              range_min: -20
+              range_max: 20
+              range_step: 1
+          lons:
+              range_min: -20
+              range_max: 20
+              range_step: 1
+          output_name: wrf_bias_compared_to_knmi
+          optional_args:
+              gridshape: !!python/tuple [6, 6]
+
+The **lat** and **lon** parameters are specified as a range of values. Be aware that the
**range_max** element is not included in the output range so you may need to adjust it slightly
if you want a particular value included. The **output_name** parameter is the name of the
resulting output graph. You may also pass any optional parameters that are supported by the
:func:`plotting.draw_contour_map` function with the **optional_args** flag.

http://git-wip-us.apache.org/repos/asf/climate/blob/c1c53afc/docs/source/index.rst
----------------------------------------------------------------------
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 796e8da..2f5bf63 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -20,6 +20,7 @@ Contents:
    ocw/utils
    data_source/data_sources
    ui-backend/backend.rst
+   config/config_overview
 
 
 Indices and tables


Mime
View raw message