airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Homan <jgho...@gmail.com>
Subject Re: PR Review Dashboard?
Date Tue, 28 Aug 2018 17:42:27 GMT
Spark is an order of magnitude larger in terms of contributors,
codebase, issues, etc.  I don't think adding an layer of complexity
into the review process would be helpful right now.  Not everything
that works for one ASF project works for another.

I'd vote against this.

-Jakob

On 28 August 2018 at 10:39, Bolke de Bruin <bdbruin@gmail.com> wrote:
> I’m not in favor of moving to GitHub issues. While JIRA is not perfect it actually
moves discussion to the mailing list. With GitHub issues the stuff just gets lost imho.
>
> I do like our changelogs a lot better now.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
>> Op 27 aug. 2018 om 05:53 heeft Holden Karau <holden@pigscanfly.ca> het volgende
geschreven:
>>
>> Awesome, so I'll give this a shot with JIRA for now and if we end up moving to GH
we can move it over. I've got a fork of the dashboard I've been using in Apache Beam as well
as the Spark one so it shouldn't take me too long to generalize it again.
>>
>>> On Sun, Aug 26, 2018 at 8:50 PM Maxime Beauchemin <maximebeauchemin@gmail.com>
wrote:
>>> I love the idea. I took a quick look at /databricks/spark-pr-dashboard
>>> <https://github.com/databricks/spark-pr-dashboard> and that looks like
a
>>> nice easy start. It would be interesting to try to make a generic project
>>> out of it that would work on top of any project/repo. Basically just
>>> refactor all of the Spark-specific code and configurations into a
>>> `config.py` (assuming there's not too much frontend Spark-specific
>>> code...).
>>>
>>> Of course this tool assumes Jira+Github which works for Airflow, but
>>> probably isn't as common of a setup to really generalize beyond Apache. It
>>> seems like by embracing Github issues and dropping Jira we could be
>>> building something much more relevant.
>>>
>>> Max
>>>
>>> On Fri, Aug 24, 2018 at 7:27 PM Holden Karau <holden@pigscanfly.ca> wrote:
>>>
>>> > Update: we can do this with the dashboard code. Since this would modify
>>> > the JIRAs I’d love a sign-off on turning that feature on from someone
on
>>> > the PMC (or at least a week with no PMC folks saying no).
>>> >
>>> > On Thu, Aug 23, 2018 at 8:00 AM Holden Karau <holden@pigscanfly.ca>
wrote:
>>> >
>>> >> I mean a few ASF projects update JIRA tickets based on PRs automatically.
>>> >> Switching from JIRA to GH issues (or back) is super painful, so I’d
>>> >> probably do more incremental improvements personally, I just don’t
have the
>>> >> time to do something like that.
>>> >>
>>> >> I’ll take a look at some the K8s tools (over in Beam they’re looking
at
>>> >> one of the review tagging tools out of K8s) if the Spark one is too
>>> >> difficult to adapt to our use case.
>>> >>
>>> >> On Thu, Aug 23, 2018 at 2:34 AM Eamon Keane <eamon.keane1@gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Kubernetes is a good place to look as they've invested a lot in
github
>>> >>> bots
>>> >>> and label based workflows. E.g. the cherry-picking script and doc
is
>>> >>> here:
>>> >>>
>>> >>>
>>> >>> https://github.com/kubernetes/kubernetes/blob/master/hack/cherry_pick_pull.sh
>>> >>>
>>> >>> https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md
>>> >>>
>>> >>> And more general overview here:
>>> >>>
>>> >>> https://github.com/kubernetes/community/tree/master/contributors/devel
>>> >>>
>>> >>> On Thu, Aug 23, 2018 at 5:11 AM Maxime Beauchemin <
>>> >>> maximebeauchemin@gmail.com> wrote:
>>> >>>
>>> >>> > I've heard many times in the past about a GH/Jira syncing tool
but
>>> >>> never
>>> >>> > seen it in action. Personally my vote is to move issues to
GH and drop
>>> >>> > Jira. Though in the process this will break the release helper
script
>>> >>> here:
>>> >>> >
>>> >>> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-jira
>>> >>> >
>>> >>> > We'll be working on a Github label-driven release baking magic
script
>>> >>> for
>>> >>> > Superset, maybe we could use the same tooling on both Airflow
and
>>> >>> Superset.
>>> >>> > The idea is that the script would use labels like `target:apache-1.11`
>>> >>> on
>>> >>> > PRs to bake releases. The tool would take as input a base SHA
and
>>> >>> release
>>> >>> > minor release number, and would craft a release branch, fetch
and
>>> >>> > cherry-pick all the right commits in the right order based
on labels,
>>> >>> > generate release tags (on minor versions) and output state
into
>>> >>> > release-info files (listing the base, all cherries, all tags,
...). The
>>> >>> > tricky part is resolving merge conflicts while auto-picking
cherries,
>>> >>> but
>>> >>> > the script would guide the operator through it .
>>> >>> >
>>> >>> > Curious to hear about how other projects do it. I think it
generally
>>> >>> > involves a lot of manual work. Let me know if you know of open
source
>>> >>> > tooling to deal with release management.
>>> >>> >
>>> >>> > Max
>>> >>> >
>>> >>> > On Wed, Aug 22, 2018 at 6:25 PM Holden Karau <holden.karau@gmail.com>
>>> >>> > wrote:
>>> >>> >
>>> >>> > > Thanks for the reminder, forgot to ask at coffee but I'll
ask.
>>> >>> > >
>>> >>> > > On Wed, Aug 22, 2018, 1:52 AM Driesprong, Fokko <fokko@driesprong.frl
>>> >>> >
>>> >>> > > wrote:
>>> >>> > >
>>> >>> > > > Hi Holden,
>>> >>> > > >
>>> >>> > > > Just curious if you got a hold of someone at the
coffee machine :-)
>>> >>> > > >
>>> >>> > > > Cheers, Fokko
>>> >>> > > >
>>> >>> > > > Op di 7 aug. 2018 om 09:17 schreef Holden Karau <
>>> >>> holden@pigscanfly.ca
>>> >>> > >:
>>> >>> > > >
>>> >>> > > > > The JIRA/Github integration tooling I’m a
little more fuzzy on
>>> >>> but
>>> >>> > I’m
>>> >>> > > > > doing coffee with some of the folks who probably
know the details
>>> >>> > this
>>> >>> > > > week
>>> >>> > > > > and I’ll report back.
>>> >>> > > > >
>>> >>> > > > > On Tue, Aug 7, 2018 at 12:15 AM Driesprong,
Fokko
>>> >>> > <fokko@driesprong.frl
>>> >>> > > >
>>> >>> > > > > wrote:
>>> >>> > > > >
>>> >>> > > > > > Hi Holden,
>>> >>> > > > > >
>>> >>> > > > > > Thanks for reaching out. Recently we've
moved to Apache Gitbox
>>> >>> (
>>> >>> > > > > > https://gitbox.apache.org/), so we use
the Github UI directly
>>> >>> > > instead
>>> >>> > > > of
>>> >>> > > > > > having to merge using a CLI (
>>> >>> > > > > >
>>> >>> > >
>>> >>> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-pr
>>> >>> > > > ).
>>> >>> > > > > >
>>> >>> > > > > > Not sure if we're already up to the game
of the dashboard,
>>> >>> which
>>> >>> > > looks
>>> >>> > > > > > awesome btw. But as you also mentioned
in your live Airflow PR,
>>> >>> > we're
>>> >>> > > > > > missing some automation of communication
between Jira and
>>> >>> Github.
>>> >>> > For
>>> >>> > > > > > example, as you mentioned, when a PR is
opened, the status
>>> >>> > > > automagically
>>> >>> > > > > > changes to In Progress. Do you have any
pointer of how this is
>>> >>> set
>>> >>> > up
>>> >>> > > > at,
>>> >>> > > > > > for example the Spark or Beam project?
So we can replicate
>>> >>> this in
>>> >>> > > > > Airflow.
>>> >>> > > > > >
>>> >>> > > > > > Cheers, Fokko
>>> >>> > > > > >
>>> >>> > > > > > 2018-08-07 7:28 GMT+02:00 Holden Karau
<holden.karau@gmail.com
>>> >>> >:
>>> >>> > > > > >
>>> >>> > > > > > > Hi Y'all,
>>> >>> > > > > > >
>>> >>> > > > > > > One of the comments from my livestream
was asking if the
>>> >>> code for
>>> >>> > > the
>>> >>> > > > > > Spark
>>> >>> > > > > > > PR review dashboard <http://spark-prs.appspot.com/>
is OSS
>>> >>> (it
>>> >>> > is
>>> >>> > > > > > > <https://github.com/databricks/spark-pr-dashboard>),
and I
>>> >>> have
>>> >>> > a
>>> >>> > > > fork
>>> >>> > > > > > up
>>> >>> > > > > > > for Beam, and I was wondering if folks
in Airflow would find
>>> >>> > > > something
>>> >>> > > > > > like
>>> >>> > > > > > > this useful? If so I'd be happy to
set that up (if not no
>>> >>> > stress).
>>> >>> > > > > > >
>>> >>> > > > > > > Cheers,
>>> >>> > > > > > >
>>> >>> > > > > > > Holden :)
>>> >>> > > > > > >
>>> >>> > > > > > > --
>>> >>> > > > > > > Cell : 425-233-8271
>>> >>> > > > > > >
>>> >>> > > > > >
>>> >>> > > > > --
>>> >>> > > > > Twitter: https://twitter.com/holdenkarau
>>> >>> > > > >
>>> >>> > > >
>>> >>> > >
>>> >>> >
>>> >>>
>>> >> --
>>> >> Twitter: https://twitter.com/holdenkarau
>>> >> Books (Learning Spark, High Performance Spark, etc.):
>>> >> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> >>
>>> > --
>>> > Twitter: https://twitter.com/holdenkarau
>>> > Books (Learning Spark, High Performance Spark, etc.):
>>> > https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> >
>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Mime
View raw message