Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@avro.apache.org
MIME-Version: 1.0
In-Reply-To: <CAO4re1mn0UmKVwhi=OMdBDR3wxEK5e+3xCR8QUsDxkZ_8PHDdQ@mail.gmail.com>
References: <CAO4re1mn0UmKVwhi=OMdBDR3wxEK5e+3xCR8QUsDxkZ_8PHDdQ@mail.gmail.com>
From: Doug Cutting <cutting@gmail.com>
Date: Mon, 14 Nov 2016 15:36:00 -0800
Message-ID: <CALEq1Z91q1isX1uJLRp4W-uoc5x4_rxjy5r2vJLWPheQ0k17iw@mail.gmail.com>
Subject: Re: [DISCUSS] Community engagement
To: dev@avro.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Mon, 14 Nov 2016 23:36:15 -0000

I disagree that Avro is heading towards the Attic.  Its rate of
contribution has been slow but steady for years.  That's the nature of
this project.  It had a larger number of contributions in its first
few years, when new languages and substantial features were being
added, but since then, we see a relatively steady stream of bug fixes
and minor features.  Here are issues fixed per year:

2009 213
2010 345
2011 240
2012 183
2013 130
2014 103
2015   71
2016 107

This year is actually an uptick from the past few.  Projects go to the
Attic when there are zero commits and zero releases for years, with
fewer than 3 PMC members who'll respond to emails at all.  We're
committing several contributions per week.  It sometimes takes a
little prodding to get enough PMC members to vote on a release, but
they're out there and will eventually vote and help get a release out.

Pre-commit tests would be great to have, facilitating development &
releases.  +1

Breaking the project into multiple products, each released separately,
could be a significant task, especially when you include getting the
first release out for each.  On one hand, it would make per-language
releases easier, but it might also let some build problems languish
undetected.  Build issues are sometimes the result of other projects
and distributions changing and may only be detected when you re-build,
even if code hasn't changed.  Lastly, getting PMC reviews & votes for
more-frequent, single-language releases might be harder than
less-frequent, multi-language releases.  To be clear, I don't strongly
oppose splitting things up, but I don't think it will be easy & may
create some new problems as it resolves existing ones.

I installed a new version of Linux on my laptop a few months ago and
was able to recreate Avro's docker image relatively quickly and
painlessly.  I just now started it for the first time since, and it
took 10 minutes to update.  Building all languages took another 5
minutes.  Each step required only a single command and proceeded
without failure.  It's a little slow, but not prohibitive.  Overall
the process is much better than before we had docker.  Sure, it could
be improved, but it's not unworkable.  I find other steps of releases
more tiresome.

Would it be nice to be able to get a Java release out more quickly and
easily?  For sure, but getting there might not be that simple.  Are
you proposing to drive this effort, following it through to
completion, nursing it if it stumbles, perhaps reverting it if it
somehow fails?  (Hadoop was once split & reunified.)

To some degree, splitting assumes that each implementation has a
sufficient community to maintain itself independently.  We have a
viable community combined, but I worry that breaking the project up
could fragment it into projects that are too tiny to make releases.
Right now we're all forced to care a bit about other implementations.

Doug

On Sun, Nov 13, 2016 at 12:21 PM, Ryan Blue <blue@apache.org> wrote:
> Hi everyone,
>
> We tried to release Avro 1.8.2 this week, but the release vote failed
> because only two PMC members voted on the candidate and we didn't have
> enough binding votes to pass. There was a minor problem (in my opinion)
> with the candidate that could have been the reason why there weren't more
> votes. If there's anyone out there that didn't vote because of this, please
> say so. Otherwise, this appears to be related to declining engagement in
> the community and that's a major problem I want to discuss.
>
> Right now, we aren't getting contributions reviewed, committed, and
> released in time for new contributors to become part of the community and
> refresh the set of active committers and PMC members. If that continues,
> this community is heading for the Attic. I think we can build back an
> active committer base and I'd love to discuss how to do that on this thread.
>
> To get us started, I have a couple of ideas.
>
> I think we need to make it easier to participate. I've brought up these
> ideas before when we moved to git: I think we need to separate the
> implementations into their own repositories and set up CI tests for each
> one.
>
> Right now, a contributor has to wait for a review to get feedback and a
> committer has to build a contribution and run tests. If we set up CI, then
> the contributor gets automated feedback and committers can spend their time
> on substantive review, rather than making sure the patch builds and tests
> pass. Making it easier for contributors and committers will help increase
> participation.
>
> Similarly, the build and release process is too difficult. It took me hours
> to get the docker image built so I could make a release candidate, because
> of a failure rate of about 1/500 downloading and installing packages. I had
> to try ~20 times before one happily completed. While the docker image helps
> a lot, the real problem we need to solve is how difficult it is to build
> all of Avro. The docker image helps, but no one really uses it until it's
> time to check a release. Instead, we all build and test how we are used to
> for a particular language implementation: maven for Java, Rake for ruby,
> etc. That's why the build.sh scripts get broken and we don't notice, and
> why the only problem with the latest RC was that it didn't pass C# tests
> outside of docker.
>
> The current build also makes implementation releases dependent on one
> another. Last release, C and ruby problems caused a multi-week delay, and
> this release we want to get the C# test environment fixed before the next
> candidate. All of this makes it take longer for contributions to make it
> out, which undermines the motivation for people to contribute.
>
> Separating the implementations will allow us to structure each repository
> how the contributors and committers for that language expect it to be. We
> can also set up per-implementation CI easily through Travis CI. And the
> biggest benefit is separating the releases, so that Python, for example,
> can release a bug fix without waiting months for unrelated changes in Java.
>
> From my perspective, these two things are a good place to start. To
> everyone still reading, what do you think?
>
> rb
>
>
> --
> Ryan Blue