climate-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lee, Kyo (US 398L)" <huikyo....@jpl.nasa.gov.INVALID>
Subject Re: [EXTERNAL] Re: [DISCUSS] Retire OCW?
Date Sun, 08 Dec 2019 16:17:56 GMT
I will start to make PRs to integrate Jupyter Notebook scripts for OCW/RCMES.
Of course, anyone is very welcomed to contribute to any parts of OCW.

Kyo

On 12/8/19, 8:09 AM, "Michael Anderson" <michael.arthur.anderson@gmail.com> wrote:

    Is there anything specific I can help with regarding the new code base?
     Otherwise, I'll make a start on consolidating the examples and maybe move
    them to Jupyter.
    
    On Sat, Nov 23, 2019 at 12:20 PM Lewis John McGibbney <lewismc@apache.org>
    wrote:
    
    > I also think that these are excellent tasks.
    > A cleanup of JIRA and branching legacy/unused code would be an excellent
    > start in order to cleanup master.
    > Lewis
    >
    > On 2019/11/21 11:09:53, Michael Anderson <
    > michael.arthur.anderson@gmail.com> wrote:
    > > To expand on that thought:
    > >
    > > Remove the UI.
    > > Remove anything related to virtualization and / or migrate it to AWS /
    > > Kubernetes.
    > > Remove any test or example that is not self contained (i.e. depends on
    > > inputs that are not publicly available).   Shrink the examples to a
    > smaller
    > > number which exercise a larger number of features.
    > > Either move mccsearch to the examples or migrate the core functions to
    > the
    > > core library.
    > >
    > > Regarding the JIRA, I'd suggested that anything more than 2 years old can
    > > closed off.
    > >
    > > I'd suggest this would make it both easier for people looking to
    > contribute
    > > to find an impactful entry point and also simplify the migration path if
    > > it's to be consolidated into Alex's new library.
    > >
    > > On Thu, Nov 21, 2019 at 5:25 AM Michael Anderson <
    > > michael.arthur.anderson@gmail.com> wrote:
    > >
    > > > One of the challenges of working on the current project is that it
    > isn't
    > > > always clear which of the sub folders are deprecated / some of the
    > JIRA are
    > > > quite old and the original requestor is no longer interested.   I'd
    > offer
    > > > that an aggressive pruning of the JIRA and sub folders would make it
    > easier
    > > > for people to engage on features that would be most impactful and long
    > > > lived.   Barring that, that is what I find appealing about Alex's
    > > > suggestion as it clearly delineates what is no longer being supported
    > and
    > > > gives a clearer roadmap on where to make the most impactful
    > contributions.
    > > >
    > > > On Wed, Nov 20, 2019 at 4:49 PM Goodman, Alexander (US 398K)
    > > > <alexander.goodman@jpl.nasa.gov.invalid> wrote:
    > > >
    > > >> Hi Lewis,
    > > >>
    > > >> I can't say I completely disagree with your assessment. I would go
    > even
    > > >> further and argue what you are saying has been true not just for 3
    > months
    > > >> but for at least since the 1.3.0 release, if not longer. But to be
    > fair,
    > > >> most of the contributions to OCW including BCDP have been made
    > possible by
    > > >> JPL/NASA funding our labor. Even though Kyo and I have had some
    > success in
    > > >> obtaining a little bit more funding, it has been a very difficult
    > path when
    > > >> we are the only primary developers, compared to the distant past when
    > many
    > > >> more people were on board which is something I explain in the slides
    > I sent
    > > >> as being one of the primary motivators for developing BCDP. The hope
    > is
    > > >> that a more up to date API could alleviate some of the problems we
    > have
    > > >> been having with maintaining the codebase since it's much smaller
    > (xarray
    > > >> helps make many of the things we were previously doing more elegant
    > and
    > > >> concise after all...), but it is still not quite at the stage where
    > it can
    > > >> fully replace the current API, and I hope we can reach that final
    > step with
    > > >> just a bit more funding.
    > > >>
    > > >> As someone with more knowledge about ASF protocol than I, I do think
    > we
    > > >> should have a discussion about what our future software development
    > > >> practices should be, and we should definitely reach an agreement on
    > what
    > > >> the best approach for adding BCDP to our repository is (and CMDA for
    > that
    > > >> matter). I definitely don't want to just dump everything haphazardly
    > into
    > > >> separate subfolders like we always do (which is something I mentioned
    > in a
    > > >> recent JIRA issue thread that you may recall), but ideally we would
    > like to
    > > >> at least keep the OCW brand alive. However if things continue to
    > operate at
    > > >> the same sluggish pace and scale as they have been for sometime, then
    > > >> perhaps we should assess the merit of continuing to keep such a small
    > > >> project under the ASF's care. If we choose to declare the death of
    > OCW, so
    > > >> be it, but I will still release BCDP as a separate project outside
    > the ASF
    > > >> if I must.
    > > >>
    > > >> Kyo is on vacation for the rest of the week, but I am free to talk
    > with
    > > >> you more about this offline if you are around this week since I'd
    > like to
    > > >> have a very indepth discussion. Would you be interested?
    > > >>
    > > >> Thanks,
    > > >> Alex
    > > >> On 11/20/19, 12:25 PM, "Lewis John McGibbney" <lewismc@apache.org>
    > > >> wrote:
    > > >>
    > > >>     This gets at my original statement. To expand however, let me
    > state,
    > > >> project activity is very low, new contributions are not being
    > reviewed, the
    > > >> roadmap is not clear, there seems to be a proposal to literally
    > drop-in
    > > >> replace current master OCW codebase with BCDP which has yet to be open
    > > >> sourced.
    > > >>     None of this is particularly pro-community growth.
    > > >>     I am not particularly happy with the way the community is being
    > > >> managed here at Apache. Nothing is particularly clear. The community
    > seems
    > > >> very stagnant and pretty much dead. This is in agreement with
    > > >> reporter.apache.org community health score which is -3.47 and
    > indicates
    > > >> that work is required.
    > > >>     This thread actually is the most activity the project has seen
in
    > the
    > > >> last 3 months!
    > > >>     I had to file the PMC report this month... for those who want to
    > see
    > > >> what that looked like please monitor whimsy.
    > > >>     I think some timelines backing up Alex's proposals are needed.
    > > >> Otherwise we are just delaying the slow death of OCW further.
    > > >>     Lewis
    > > >>
    > > >>     On 2019/11/20 16:52:47, Michael Anderson <
    > > >> michael.arthur.anderson@gmail.com> wrote:
    > > >>     > That’s pretty interesting.   I like adding incremental features
    > > >> with the goal of replacing rather than outright demise of the old
    > library.
    > > >>     >
    > > >>     > Sent from my iPhone
    > > >>     >
    > > >>     > > On Nov 20, 2019, at 11:08 AM, Goodman, Alexander (US
398K)
    > > >> <alexander.goodman@jpl.nasa.gov.invalid> wrote:
    > > >>     > >
    > > >>     > > Since the plots didn't attach in the last message, here
they
    > are:
    > > >>     > >
    > > >>     > > https://i.imgur.com/mAuq0R5.png
    > > >>     > > https://i.imgur.com/TeRYSPI.png
    > > >>     > >
    > > >>     > > Thanks,
    > > >>     > > Alex
    > > >>     > > -----Original Message-----
    > > >>     > > From: Goodman, Alexander (US 398K)
    > > >> <alexander.goodman@jpl.nasa.gov.INVALID>
    > > >>     > > Sent: Wednesday, November 20, 2019 8:05 AM
    > > >>     > > To: dev@climate.apache.org
    > > >>     > > Subject: RE: [EXTERNAL] Re: [DISCUSS] Retire OCW?
    > > >>     > >
    > > >>     > > Hi Michael,
    > > >>     > >
    > > >>     > > Please see this overview slide:
    > > >>     > > https://i.imgur.com/VnijQ5C.png
    > > >>     > >
    > > >>     > > Here is also a link of some older slides which have a
lot more
    > > >> details:
    > > >>     > >
    > > >>
    > https://docs.google.com/presentation/d/1nhD3_fZmVcmBnq9NiMBYeDMsxr5kIimI7dE9NNUQJEM/edit?usp=sharing
    > > >>     > >
    > > >>     > > I also created an end-to-end processing benchmark that
I
    > compared
    > > >> with the current OCW, which I have attached in two plots: One with
    > > >> regridding and one without, since the difference is so vast. I first
    > > >> presented these results at the AMS Annual meeting last year in
    > Phoenix.
    > > >>     > >
    > > >>     > > Let me know if you have any questions.
    > > >>     > >
    > > >>     > > Thanks,
    > > >>     > > Alex
    > > >>     > > -----Original Message-----
    > > >>     > > From: Michael Anderson <michael.arthur.anderson@gmail.com>
    > > >>     > > Sent: Tuesday, November 19, 2019 5:59 PM
    > > >>     > > To: dev@climate.apache.org
    > > >>     > > Subject: [EXTERNAL] Re: [DISCUSS] Retire OCW?
    > > >>     > >
    > > >>     > > Do you have an overview of the new library you could
share?
    > It'd
    > > >> be interesting how big the gap between the old and new would be /
    > rough LOE
    > > >> to add parity.
    > > >>     > >
    > > >>     > >> On Tue, Nov 19, 2019 at 7:35 PM Alex Goodman <
    > goodman@apache.org>
    > > >> wrote:
    > > >>     > >>
    > > >>     > >> Hi Lewis,
    > > >>     > >>
    > > >>     > >> To add some more specifics, the new xarray-based
API I have
    > > >> developed
    > > >>     > >> for OCW called BCDP (Big Climate Data Pipeline) has
recently
    > > >> been
    > > >>     > >> approved for release to open source. Keep in mind,
this is a
    > > >>     > >> completely new API that is not backwards compatible,
so we
    > were
    > > >>     > >> planning to maintain both the old and new API under
one
    > > >> repository
    > > >>     > >> (either as separate subfolders or git
    > > >>     > >> branches) until the new API has enough functionality
to fully
    > > >> replace
    > > >>     > >> the old one. To accelerate this process and encourage
more
    > > >>     > >> development, both Kyo and I are seeking some additional
    > funding
    > > >>     > >> sources which we should be more certain about early
next
    > year.
    > > >> When I
    > > >>     > >> first conceived BCDP, I was originally planning on
releasing
    > it
    > > >> to
    > > >>     > >> open source as a separate project outside of OCW,
but after
    > some
    > > >>     > >> discussion we thought that it would be better to
maintain the
    > > >> OCW brand for it if possible.
    > > >>     > >>
    > > >>     > >> Does this make sense?
    > > >>     > >>
    > > >>     > >> Thanks,
    > > >>     > >> Alex
    > > >>     > >>> On 2019/11/20 00:17:22, lewis john mcgibbney
<
    > > >> lewismc@apache.org> wrote:
    > > >>     > >>> I got this from Kyo offline
    > > >>     > >>>
    > > >>     > >>> “There is enough motivation to keep OCW going.
Alex has
    > almost
    > > >>     > >>> refactored OCW and Seungwon Lee's CMDA will be
a part of
    > OCW.”
    > > >>     > >>>
    > > >>     > >>> Some of us will remember that CMDA was retired
from the
    > Apache
    > > >>     > >>> Incubator previously.
    > > >>     > >>>
    > > >>     > >>> It would be great if these plans could be hashed
out on the
    > > >> mailing
    > > >>     > >>> list
    > > >>     > >> so
    > > >>     > >>> we can move forward either way.
    > > >>     > >>>
    > > >>     > >>> Lewis
    > > >>     > >>>
    > > >>     > >>> On Thu, Nov 14, 2019 at 09:32 Lewis John Mcgibbney
<
    > > >>     > >>> lewis.mcgibbney@gmail.com> wrote:
    > > >>     > >>>
    > > >>     > >>>> Hi Folks,
    > > >>     > >>>> Project activity is very low and new contributions
are not
    > > >> being
    > > >>     > >> reviewed.
    > > >>     > >>>> Simply, is it time to retire OCW to the Apache
Attic?
    > > >>     > >>>> Best
    > > >>     > >>>> Lewis
    > > >>     > >>>>
    > > >>     > >>>> --
    > > >>     > >>>>
    > > >>     > >>>> *Lewis*
    > > >>     > >>>> Dr. Lewis J. McGibbney Ph.D, B.Sc
    > > >>     > >>>> *Skype*: lewis.john.mcgibbney
    > > >>     > >>>>
    > > >>     > >>>>
    > > >>     > >>>>
    > > >>     > >>>> --
    > > >>     > >>> http://home.apache.org/~lewismc/
    > > >>     > >>> http://people.apache.org/keys/committer/lewismc
    > > >>     > >>>
    > > >>     > >>
    > > >>     >
    > > >>
    > > >>
    > > >>
    > >
    >
    

Mime
View raw message