incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattmann, Chris A (3980)" <chris.a.mattm...@jpl.nasa.gov>
Subject Re: [PROPOSAL] Climate Model Diagnostic Analyzer
Date Thu, 26 Mar 2015 06:38:58 GMT
Dear Henry,

That’s an omission - I will update the proposal to include them.

Sorry about that! Lei, there aren’t any objections to that correct?

Cheers,
Chris


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Henry Saputra <henry.saputra@gmail.com>
Reply-To: "general@incubator.apache.org" <general@incubator.apache.org>
Date: Wednesday, March 25, 2015 at 11:22 AM
To: "general@incubator.apache.org" <general@incubator.apache.org>
Subject: Re: [PROPOSAL] Climate Model Diagnostic Analyzer

>HI Chris,
>
>Great proposal.
>
>Looks like the people from CMU are excluded from list of initial
>committers?
>They are mentioned in the affiliations section but not in the
>committers section.
>
>
>- Henry
>
>On Sun, Mar 22, 2015 at 10:55 PM, Mattmann, Chris A (3980)
><chris.a.mattmann@jpl.nasa.gov> wrote:
>> Hi Everyone,
>>
>> I am pleased to submit for consideration to the Apache Incubator
>> the Climate Model Diagnostic Analyzer proposal. We are actively
>> soliciting interested mentors in this project related to climate
>> science and analytics and big data.
>>
>> Please find the wiki text of the proposal below and the link up
>> on the wiki here:
>>
>> https://wiki.apache.org/incubator/ClimateModelDiagnosticAnalyzerProposal
>>
>> Thank you for your consideration!
>>
>> Cheers,
>> Chris
>> (on behalf of the Climate Model Diagnostic Analyzer community)
>>
>> = Apache ClimateModelDiagnosticAnalyzer Proposal =
>>
>> == Abstract ==
>>
>> The Climate Model Diagnostic Analyzer (CMDA) provides web services for
>> multi-aspect physics-based and phenomenon-oriented climate model
>> performance evaluation and diagnosis through the comprehensive and
>> synergistic use of multiple observational data, reanalysis data, and
>>model
>> outputs.
>>
>> == Proposal ==
>>
>> The proposed web-based tools let users display, analyze, and download
>> earth science data interactively. These tools help scientists quickly
>> examine data to identify specific features, e.g., trends, geographical
>> distributions, etc., and determine whether a further study is needed.
>>All
>> of the tools are designed and implemented to be general so that data
>>from
>> models, observation, and reanalysis are processed and displayed in a
>> unified way to facilitate fair comparisons. The services prepare and
>> display data as a colored map or an X-Y plot and allow users to download
>> the analyzed data. Basic visual capabilities include 1) displaying
>> two-dimensional variable as a map, zonal mean, and time series 2)
>> displaying three-dimensional variable’s zonal mean, a two-dimensional
>> slice at a specific altitude, and a vertical profile. General analysis
>>can
>> be done using the difference, scatter plot, and conditional sampling
>> services. All the tools support display options for using linear or
>> logarithmic scales and allow users to specify a temporal range and
>>months
>> in a year. The source/input datasets for these tools are CMIP5 model
>> outputs, Obs4MIP observational datasets, and ECMWF reanalysis datasets.
>> They are stored on the server and are selectable by a user through the
>>web
>> services.
>>
>> === Service descriptions ===
>>
>> 1. '''Two dimensional variable services'''
>>
>> * Map of two-dimensional variable:  This services displays a two
>> dimensional variable as a colored longitude and latitude map with values
>> represented by a color scheme. Longitude and latitude ranges can be
>> specified to magnify a specific region.
>>
>> * Two dimensional variable zonal mean:  This service plots the zonal
>>mean
>> value of a two-dimensional variable as a function of the latitude in
>>terms
>> of an X-Y plot.
>>
>> * Two dimensional variable time series:  This service displays the
>>average
>> of a two-dimensional variable over the specific region as function of
>>time
>> as an X-Y plot.
>>
>> 2. '''Three dimensional variable services'''
>>
>> * Map of a two dimensional slice of a three-dimensional variable:  This
>> service displays a two-dimensional slice of a three-dimensional variable
>> at a specific altitude as a colored longitude and latitude map with
>>values
>> represented by a color scheme.
>>
>> * Three dimensional zonal mean:  Zonal mean of the specified
>> three-dimensional variable is computed and displayed as a colored
>> altitude-latitude map.
>>
>> * Vertical profile of a three-dimensional variable:  Compute the area
>> weighted average of a three-dimensional variable over the specified
>>region
>> and display the average as function of pressure level (altitude) as an
>>X-Y
>> plot.
>>
>> 3. '''General services'''
>>
>> * Difference of two variables:  This service displays the differences
>> between the two variables, which can be either a two dimensional
>>variable
>> or a slice of a three-dimensional variable at a specified altitude as
>> colored longitude and latitude maps
>>
>> * Scatter and histogram plots of two variables:  This service displays
>>the
>> scatter plot (X-Y plot) between two specified variables and the
>>histograms
>> of the two variables. The number of samples can be specified and the
>> correlation is computed. The two variables can be either a
>>two-dimensional
>> variable or a slice of a three-dimensional variable at a specific
>>altitude.
>>
>> * Conditional sampling:  This service lets user to sort a physical
>> quantity of two or dimensions according to the values of another
>>variable
>> (environmental condition, e.g. SST) which may be a two-dimensional
>> variable or a slice of a three-dimensional variable at a specific
>> altitude. For a two dimensional quantity, the plot is displayed an X-Y
>> plot, and for a two-dimensional quantity, plot is displayed as a
>> colored-map.
>>
>>
>> == Background and Rationale ==
>>
>> The latest Intergovernmental Panel on Climate Change (IPCC) Fourth
>> Assessment Report stressed the need for the comprehensive and innovative
>> evaluation of climate models with newly available global observations.
>>The
>> traditional approach to climate model evaluation, which is the
>>comparison
>> of a single parameter at a time, identifies symptomatic model biases and
>> errors but fails to diagnose the model problems. The model diagnosis
>> process requires physics-based multi-variable comparisons, which
>>typically
>> involve large-volume and heterogeneous datasets, and computationally
>> demanding and data-intensive operations. We propose to develop a
>> computationally efficient information system to enable the physics-based
>> multi-variable model performance evaluations and diagnoses through the
>> comprehensive and synergistic use of multiple observational data,
>> reanalysis data, and model outputs.
>>
>> Satellite observations have been widely used in model-data
>> inter-comparisons and model evaluation studies. These studies normally
>> involve the comparison of a single parameter at a time using a time and
>> space average. For example, modeling cloud-related processes in global
>> climate models requires cloud parameterizations that provide
>>quantitative
>> rules for expressing the location, frequency of occurrence, and
>>intensity
>> of the clouds in terms of multiple large-scale model-resolved parameters
>> such as temperature, pressure, humidity, and wind. One can evaluate the
>> performance of the cloud parameterization by comparing the cloud water
>> content with satellite data and can identify symptomatic model biases or
>> errors. However, in order to understand the cause of the biases and
>> errors, one has to simultaneously investigate several parameters that
>>are
>> integrated in the cloud parameterization.
>>
>> Such studies, aimed at a multi-parameter model diagnosis, require
>> locating, understanding, and manipulating multi-source observation
>> datasets, model outputs, and (re)analysis outputs that are physically
>> distributed, massive in volume, heterogeneous in format, and provide
>> little information on data quality and production legacy. Additionally,
>> these studies involve various data preparation and processing steps that
>> can easily become computationally demanding since many datasets have to
>>be
>> combined and processed simultaneously. It is notorious that scientists
>> spend more than 60% of their research time on just preparing the dataset
>> before it can be analyzed for their research.
>>
>> To address these challenges, we propose to build Climate Model
>>Diagnostic
>> Analyzer (CMDA) that will enable a streamlined and structured
>>preparation
>> of multiple large-volume and heterogeneous datasets, and provide a
>> computationally efficient approach to processing the datasets for model
>> diagnosis. We will leverage the existing information technologies and
>> scientific tools that we developed in our current NASA ROSES COUND, MAP,
>> and AIST projects. We will utilize the open-source Web-service
>>technology.
>> We will make CMDA complementary to other climate model analysis tools
>> currently available to the research community (e.g., PCMDI’s CDAT and
>> NCAR’s CCMVal) by focusing on the missing capabilities such as
>>conditional
>> sampling, and probability distribution function and cluster analysis of
>> multiple-instrument datasets. The users will be able to use a web
>>browser
>> to interface with CMDA.
>>
>> == Current Status ==
>>
>> The current version of ClimateModelDiagnosticAnalyzer was developed by a
>> team at The Jet Propulsion Laboratory (JPL). The project was initiated
>>as
>> a NASA-sponsored project (ROSES-CMAC) in 2011.
>>
>> == Meritocracy ==
>>
>> The current developers are not familiar with meritocratic open source
>> development at Apache, but would like to encourage this style of
>> development for the project.
>>
>> == Community ==
>>
>> While ClimateModelDiagnosticAnalyzer started as a JPL research project,
>>it
>> has been used in The 2014 Caltech Summer School sponsored by the JPL
>> Center for Climate Sciences. Some 23 students from different
>>institutions
>> over the world participated. We deployed the tool to the Amazon Cloud
>>and
>> let every student each has his or her own virtual machine. Students gave
>> positive feedback mostly on the usability and speed of our web services.
>> We also collected a number of enhancement requests. We seek to further
>> grow the developer and user communities using the Apache open source
>> venue. During incubation we will explicitly seek increased academic
>> collaborations (e.g., with The Carnegie Mellon University) as well as
>> industrial participation.
>>
>> One instance of our web services can be found at:
>> http://cmacws.jpl.nasa.gov:8080/cmac/
>>
>> == Core Developers ==
>>
>> The core developers of the project are JPL scientists and software
>> developers.
>>
>> == Alignment ==
>>
>> Apache is the most natural home for taking the
>> ClimateModelDiagnosticAnalyzer project forward. It is well-aligned with
>> some Apache projects such as Apache Open Climate Workbench.
>> ClimateModelDiagnosticAnalyzer also seeks to achieve an Apache-style
>> development model; it is seeking a broader community of contributors and
>> users in order to achieve its full potential and value to the Climate
>> Science and Big Data community.
>>
>> There are also a number of dependencies that will be mentioned below in
>> the Relationships with Other Apache products section.
>>
>>
>> == Known Risks ==
>>
>> === Orphaned products ===
>>
>> Given the current level of intellectual investment in
>> ClimateModelDiagnosticAnalyzer, the risk of the project being abandoned
>>is
>> very small. The Carnegie Mellon University and JPL are collaborating
>> (2014-2015) to build a service for climate analytics workflow
>> recommendation using fund from NASA. A two-year NASA AIST project
>> (2015-2016) will soon start to add diagnostic analysis methodologies
>>such
>> as conditional sampling method, conditional probability density
>>function,
>> data co-location, and random forest. We will also infuse the provenance
>> technology into CMDA so that the history of the data products and
>> workflows will be automatically collected and saved. This information
>>will
>> also be indexed so that the products and workflows can be searchable by
>> the community of climate scientists and students.
>>
>> === Inexperience with Open Source ===
>>
>> The current developers of ClimateModelDiagnosticAnalyzer are
>>inexperienced
>> with Open Source. However, our Champion Chris Mattmann is experienced
>> (Champions of ApacheOpenClimateWorkbench and AsterixDB) and will be
>> working closely with us, also as the Chief Architect of our JPL section.
>>
>> === Relationships with Other Apache Products ===
>>
>> Clearly there is a direct relationship between this project and the
>>Apache
>> Open Climate Workbench already a top level Apache project and also
>>brought
>> to the ASF by its Champion (and ours) Chris Mattmann. We plan on
>>directly
>> collaborating with the Open Climate Workbench community via our Champion
>> and we also welcome ASF mentors familiar with the OCW project to help
>> mentor our project. In addition our team is extremely welcoming of ASF
>> projects and if there are synergies with them we invite participation in
>> the proposal and in the discussion.
>>
>> === Homogeneous Developers ===
>>
>> The current community is within JPL but we would like to increase the
>> heterogeneity.
>>
>> === Reliance on Salaried Developers ===
>>
>> The initial committers are full-time JPL staff from 2013 to 2014. The
>> other committers from 2014 to 2015 are a mix of CMU faculty, students
>>and
>> JPL staff.
>>
>> === An Excessive Fascination with the Apache Brand ===
>>
>> We believe in the processes, systems, and framework Apache has put in
>> place. Apache is also known to foster a great community around their
>> projects and provide exposure. While brand is important, our fascination
>> with it is not excessive. We believe that the ASF is the right home for
>> ClimateModelDiagnosticAnalyzer and that having
>> ClimateModelDiagnosticAnalyzer inside of the ASF will lead to a better
>> long-term outcome for the Climate Science and Big Data community.
>>
>> === Documentation ===
>>
>> The ClimateModelDiagnosticAnalyzer services and documentation can be
>>found
>> at: http://cmacws.jpl.nasa.gov:8080/cmac/.
>>
>> === Initial Source ===
>>
>> Current source resides in ...
>>
>> === External Dependencies ===
>>
>> ClimateModelDiagnosticAnalyzer depends on a number of open source
>>projects:
>>
>>  * Flask
>>  * Gunicorn
>>  * Tornado Web Server
>>  * GNU octave
>>  * epd python
>>  * NOAA ferret
>>  * GNU plot
>>
>> == Required Resources ==
>>
>> === Developer and user mailing lists ===
>>
>>  * private@cmda.incubator.apache.org (with moderated subscriptions)
>>  * commits@cmda.incubator.apache.org
>>  * dev@cmda.incubator.apache.org
>>  * users@cmda.incubator.apache.org
>>
>> A git repository
>>
>> https://git-wip-us.apache.org/repos/asf/incubator-cmda.git
>>
>> A JIRA issue tracker
>>
>> https://issues.apache.org/jira/browse/CMDA
>>
>> === Initial Committers ===
>>
>> The following is a list of the planned initial Apache committers (the
>> active subset of the committers for the current repository at Google
>>code).
>>
>>  * Seungwon Lee (seungwon.lee@jpl.nasa.gov)
>>  * Lei Pan (lei.pan@jpl.nasa.gov)
>>  * Chengxing Zhai (chengxing.zhai@jpl.nasa.gov)
>>  * Benyang Tang (benyang.tang@jpl.nasa.gov)
>>
>>
>> === Affiliations ===
>>
>> JPL
>>
>>  * Seungwon Lee
>>  * Lei Pan
>>  * Chengxing Zhai
>>  * Benyang Tang
>>
>> CMU
>>
>>  * Jia Zhang
>>  * Wei Wang
>>  * Chris Lee
>>  * Xing Wei
>>
>> == Sponsors ==
>>
>> NASA
>>
>> === Champion ===
>>
>> Chris Mattmann (NASA/JPL)
>>
>> === Nominated Mentors ===
>>
>> TBD
>>
>> === Sponsoring Entity ===
>>
>> The Apache Incubator
>>
>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattmann@nasa.gov
>> WWW:  http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org

Mime
View raw message