climate-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLIMATE-926) Metadata Extractors
Date Mon, 25 Sep 2017 17:49:00 GMT

    [ https://issues.apache.org/jira/browse/CLIMATE-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179426#comment-16179426
] 

ASF GitHub Bot commented on CLIMATE-926:
----------------------------------------

GitHub user agoodm opened a pull request:

    https://github.com/apache/climate/pull/451

    CLIMATE-926 - Metadata Extractors

    For now this is only being used for our soon to be shown CORDEX examples, but I do plan
to make this a more permanent part of OCW with unit tests and documentation.
    
    Example usage in an ipython shell:
    
    ```python
    In [1]: from metadata_extractor import CORDEXMetadataExtractor, obs4MIPSMetadataExtractor
    
    In [2]: obs_extractor = obs4MIPSMetadataExtractor('/proj3/data/obs4mips')
    
    In [3]: models_extractor = CORDEXMetadataExtractor('/proj3/data/CORDEX/NAM-44/*')
    
    In [4]: obs_extractor.instruments
    Out[4]: 
    {'AIRS',
     'AMSRE',
     'ATSR',
     'AVISO',
     'CALIOP',
     'CERES-EBAF',
     'GPCP-1DD',
     'GPCP-SG',
     'HIRS',
     'MISR',
     'MLS',
     'MODIS',
     'MODIS-Level',
     'MODIS-level',
     'OISST',
     'QuikSCAT',
     'RO',
     'SSMI-MERIS',
     'SSMI-NH',
     'SSMI-SH',
     'TES',
     'TRMM-L3',
     'ghgcci'}
    
    In [5]: models_extractor.models
    Out[5]: {'DMI-HIRHAM5', 'MOHC-HadRM3P', 'SMHI-RCA4', 'UQAM-CRCM5'}
    
    In [6]: obs_extractor.query(variable='pr')
    Out[6]: 
    [{'filename': '/proj3/data/obs4mips/pr_GPCP-SG_L3_v2.2_*.nc',
      'instrument': 'GPCP-SG',
      'processing_level': 'L3',
      'variable': 'pr',
      'version': 'v2.2'},
     {'filename': '/proj3/data/obs4mips/pr_GPCP-1DD_L3_v1.2_*.nc',
      'instrument': 'GPCP-1DD',
      'processing_level': 'L3',
      'variable': 'pr',
      'version': 'v1.2'},
     {'filename': '/proj3/data/obs4mips/pr_TRMM-L3_*.nc',
      'instrument': 'TRMM-L3',
      'variable': 'pr'}]
    
    In [7]: obs_extractor.group(models_extractor, 'variable')[0]
    Out[7]: 
    ({'filename': '/proj3/data/obs4mips/rlut_CERES-EBAF_L3B_Ed2-8_*.nc',
      'instrument': 'CERES-EBAF',
      'processing_level': 'L3B',
      'variable': 'rlut',
      'version': 'Ed2-8'},
     [{'domain': 'NAM-44',
       'driving_model': 'ECMWF-ERAINT',
       'ensemble': 'r1i1p1',
       'experiment': 'evaluation',
       'filename': '/proj3/data/CORDEX/NAM-44/rlut/rlut_NAM-44_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadRM3P_v1_mon_*.nc',
       'model': 'MOHC-HadRM3P',
       'time_step': 'mon',
       'variable': 'rlut',
       'version': 'v1'},
      {'domain': 'NAM-44',
       'driving_model': 'ECMWF-ERAINT',
       'ensemble': 'r1i1p1',
       'experiment': 'evaluation',
       'filename': '/proj3/data/CORDEX/NAM-44/rlut/rlut_NAM-44_ECMWF-ERAINT_evaluation_r1i1p1_UQAM-CRCM5_v1_mon_*.nc',
       'model': 'UQAM-CRCM5',
       'time_step': 'mon',
       'variable': 'rlut',
       'version': 'v1'},
      {'domain': 'NAM-44',
       'driving_model': 'ECMWF-ERAINT',
       'ensemble': 'r1i1p1',
       'experiment': 'evaluation',
       'filename': '/proj3/data/CORDEX/NAM-44/rlut/rlut_NAM-44_ECMWF-ERAINT_evaluation_r1i1p1_SMHI-RCA4_v1_mon_*.nc',
       'model': 'SMHI-RCA4',
       'time_step': 'mon',
       'variable': 'rlut',
       'version': 'v1'},
      {'domain': 'NAM-44',
       'driving_model': 'ECMWF-ERAINT',
       'ensemble': 'r1i1p1',
       'experiment': 'evaluation',
       'filename': '/proj3/data/CORDEX/NAM-44/rlut/rlut_NAM-44_ECMWF-ERAINT_evaluation_r1i1p1_DMI-HIRHAM5_v1_mon_*.nc',
       'model': 'DMI-HIRHAM5',
       'time_step': 'mon',
       'variable': 'rlut',
       'version': 'v1'}])
    
    ```


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/agoodm/climate CLIMATE-926

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/climate/pull/451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #451
    
----
commit 8217d12f06987d852f9294da94a5af243116e751
Author: Alex Goodman <agoodm@users.noreply.github.com>
Date:   2017-09-25T17:35:20Z

    CLIMATE-926 - Metadata Extractors

commit ec81037a21dbf2cf1999422127bbe924e1072c4a
Author: Alex Goodman <agoodm@users.noreply.github.com>
Date:   2017-09-25T17:41:07Z

    Fix model attribute in CORDEXMetadataExtractor

----


> Metadata Extractors
> -------------------
>
>                 Key: CLIMATE-926
>                 URL: https://issues.apache.org/jira/browse/CLIMATE-926
>             Project: Apache Open Climate Workbench
>          Issue Type: New Feature
>          Components: general
>            Reporter: Alex Goodman
>            Assignee: Alex Goodman
>             Fix For: 1.3.0
>
>
> Add metadata extractors which scan the data directories (Obs4MIPs and CORDEX), and extract
the metadata from all the files therein based on filename conventions. Additional important
capabilities include narrowing down the list of metadata by field (eg, variable name). The
primary purpose of this in the context of the CORDEX pilot project is to associate a list
of models available for a given domain with an obs4MIPs dataset, which can then be used as
inputs to render the configuration file from a template.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message