airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: PR Review Dashboard?
Date Tue, 28 Aug 2018 17:42:31 GMT
Alright let's keep things this way at least for now so that we can move
forward with the Spark approach.

Max

On Tue, Aug 28, 2018 at 10:39 AM Bolke de Bruin <bdbruin@gmail.com> wrote:

> I’m not in favor of moving to GitHub issues. While JIRA is not perfect it
> actually moves discussion to the mailing list. With GitHub issues the stuff
> just gets lost imho.
>
> I do like our changelogs a lot better now.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 27 aug. 2018 om 05:53 heeft Holden Karau <holden@pigscanfly.ca> het
> volgende geschreven:
> >
> > Awesome, so I'll give this a shot with JIRA for now and if we end up
> moving to GH we can move it over. I've got a fork of the dashboard I've
> been using in Apache Beam as well as the Spark one so it shouldn't take me
> too long to generalize it again.
> >
> >> On Sun, Aug 26, 2018 at 8:50 PM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
> >> I love the idea. I took a quick look at /databricks/spark-pr-dashboard
> >> <https://github.com/databricks/spark-pr-dashboard> and that looks like
> a
> >> nice easy start. It would be interesting to try to make a generic
> project
> >> out of it that would work on top of any project/repo. Basically just
> >> refactor all of the Spark-specific code and configurations into a
> >> `config.py` (assuming there's not too much frontend Spark-specific
> >> code...).
> >>
> >> Of course this tool assumes Jira+Github which works for Airflow, but
> >> probably isn't as common of a setup to really generalize beyond Apache.
> It
> >> seems like by embracing Github issues and dropping Jira we could be
> >> building something much more relevant.
> >>
> >> Max
> >>
> >> On Fri, Aug 24, 2018 at 7:27 PM Holden Karau <holden@pigscanfly.ca>
> wrote:
> >>
> >> > Update: we can do this with the dashboard code. Since this would
> modify
> >> > the JIRAs I’d love a sign-off on turning that feature on from someone
> on
> >> > the PMC (or at least a week with no PMC folks saying no).
> >> >
> >> > On Thu, Aug 23, 2018 at 8:00 AM Holden Karau <holden@pigscanfly.ca>
> wrote:
> >> >
> >> >> I mean a few ASF projects update JIRA tickets based on PRs
> automatically.
> >> >> Switching from JIRA to GH issues (or back) is super painful, so I’d
> >> >> probably do more incremental improvements personally, I just don’t
> have the
> >> >> time to do something like that.
> >> >>
> >> >> I’ll take a look at some the K8s tools (over in Beam they’re looking
> at
> >> >> one of the review tagging tools out of K8s) if the Spark one is too
> >> >> difficult to adapt to our use case.
> >> >>
> >> >> On Thu, Aug 23, 2018 at 2:34 AM Eamon Keane <eamon.keane1@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Kubernetes is a good place to look as they've invested a lot in
> github
> >> >>> bots
> >> >>> and label based workflows. E.g. the cherry-picking script and doc
is
> >> >>> here:
> >> >>>
> >> >>>
> >> >>>
> https://github.com/kubernetes/kubernetes/blob/master/hack/cherry_pick_pull.sh
> >> >>>
> >> >>>
> https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md
> >> >>>
> >> >>> And more general overview here:
> >> >>>
> >> >>>
> https://github.com/kubernetes/community/tree/master/contributors/devel
> >> >>>
> >> >>> On Thu, Aug 23, 2018 at 5:11 AM Maxime Beauchemin <
> >> >>> maximebeauchemin@gmail.com> wrote:
> >> >>>
> >> >>> > I've heard many times in the past about a GH/Jira syncing
tool but
> >> >>> never
> >> >>> > seen it in action. Personally my vote is to move issues to
GH and
> drop
> >> >>> > Jira. Though in the process this will break the release helper
> script
> >> >>> here:
> >> >>> >
> >> >>>
> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-jira
> >> >>> >
> >> >>> > We'll be working on a Github label-driven release baking magic
> script
> >> >>> for
> >> >>> > Superset, maybe we could use the same tooling on both Airflow
and
> >> >>> Superset.
> >> >>> > The idea is that the script would use labels like
> `target:apache-1.11`
> >> >>> on
> >> >>> > PRs to bake releases. The tool would take as input a base
SHA and
> >> >>> release
> >> >>> > minor release number, and would craft a release branch, fetch
and
> >> >>> > cherry-pick all the right commits in the right order based
on
> labels,
> >> >>> > generate release tags (on minor versions) and output state
into
> >> >>> > release-info files (listing the base, all cherries, all tags,
> ...). The
> >> >>> > tricky part is resolving merge conflicts while auto-picking
> cherries,
> >> >>> but
> >> >>> > the script would guide the operator through it .
> >> >>> >
> >> >>> > Curious to hear about how other projects do it. I think it
> generally
> >> >>> > involves a lot of manual work. Let me know if you know of
open
> source
> >> >>> > tooling to deal with release management.
> >> >>> >
> >> >>> > Max
> >> >>> >
> >> >>> > On Wed, Aug 22, 2018 at 6:25 PM Holden Karau <
> holden.karau@gmail.com>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > Thanks for the reminder, forgot to ask at coffee but
I'll ask.
> >> >>> > >
> >> >>> > > On Wed, Aug 22, 2018, 1:52 AM Driesprong, Fokko
> <fokko@driesprong.frl
> >> >>> >
> >> >>> > > wrote:
> >> >>> > >
> >> >>> > > > Hi Holden,
> >> >>> > > >
> >> >>> > > > Just curious if you got a hold of someone at the
coffee
> machine :-)
> >> >>> > > >
> >> >>> > > > Cheers, Fokko
> >> >>> > > >
> >> >>> > > > Op di 7 aug. 2018 om 09:17 schreef Holden Karau
<
> >> >>> holden@pigscanfly.ca
> >> >>> > >:
> >> >>> > > >
> >> >>> > > > > The JIRA/Github integration tooling I’m a
little more fuzzy
> on
> >> >>> but
> >> >>> > I’m
> >> >>> > > > > doing coffee with some of the folks who probably
know the
> details
> >> >>> > this
> >> >>> > > > week
> >> >>> > > > > and I’ll report back.
> >> >>> > > > >
> >> >>> > > > > On Tue, Aug 7, 2018 at 12:15 AM Driesprong,
Fokko
> >> >>> > <fokko@driesprong.frl
> >> >>> > > >
> >> >>> > > > > wrote:
> >> >>> > > > >
> >> >>> > > > > > Hi Holden,
> >> >>> > > > > >
> >> >>> > > > > > Thanks for reaching out. Recently we've
moved to Apache
> Gitbox
> >> >>> (
> >> >>> > > > > > https://gitbox.apache.org/), so we use
the Github UI
> directly
> >> >>> > > instead
> >> >>> > > > of
> >> >>> > > > > > having to merge using a CLI (
> >> >>> > > > > >
> >> >>> > >
> >> >>>
> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-pr
> >> >>> > > > ).
> >> >>> > > > > >
> >> >>> > > > > > Not sure if we're already up to the game
of the dashboard,
> >> >>> which
> >> >>> > > looks
> >> >>> > > > > > awesome btw. But as you also mentioned
in your live
> Airflow PR,
> >> >>> > we're
> >> >>> > > > > > missing some automation of communication
between Jira and
> >> >>> Github.
> >> >>> > For
> >> >>> > > > > > example, as you mentioned, when a PR is
opened, the status
> >> >>> > > > automagically
> >> >>> > > > > > changes to In Progress. Do you have any
pointer of how
> this is
> >> >>> set
> >> >>> > up
> >> >>> > > > at,
> >> >>> > > > > > for example the Spark or Beam project?
So we can replicate
> >> >>> this in
> >> >>> > > > > Airflow.
> >> >>> > > > > >
> >> >>> > > > > > Cheers, Fokko
> >> >>> > > > > >
> >> >>> > > > > > 2018-08-07 7:28 GMT+02:00 Holden Karau
<
> holden.karau@gmail.com
> >> >>> >:
> >> >>> > > > > >
> >> >>> > > > > > > Hi Y'all,
> >> >>> > > > > > >
> >> >>> > > > > > > One of the comments from my livestream
was asking if the
> >> >>> code for
> >> >>> > > the
> >> >>> > > > > > Spark
> >> >>> > > > > > > PR review dashboard <http://spark-prs.appspot.com/>
is
> OSS
> >> >>> (it
> >> >>> > is
> >> >>> > > > > > > <https://github.com/databricks/spark-pr-dashboard>),
> and I
> >> >>> have
> >> >>> > a
> >> >>> > > > fork
> >> >>> > > > > > up
> >> >>> > > > > > > for Beam, and I was wondering if
folks in Airflow would
> find
> >> >>> > > > something
> >> >>> > > > > > like
> >> >>> > > > > > > this useful? If so I'd be happy to
set that up (if not
> no
> >> >>> > stress).
> >> >>> > > > > > >
> >> >>> > > > > > > Cheers,
> >> >>> > > > > > >
> >> >>> > > > > > > Holden :)
> >> >>> > > > > > >
> >> >>> > > > > > > --
> >> >>> > > > > > > Cell : 425-233-8271
> >> >>> > > > > > >
> >> >>> > > > > >
> >> >>> > > > > --
> >> >>> > > > > Twitter: https://twitter.com/holdenkarau
> >> >>> > > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >> --
> >> >> Twitter: https://twitter.com/holdenkarau
> >> >> Books (Learning Spark, High Performance Spark, etc.):
> >> >> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> >> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> >> >>
> >> > --
> >> > Twitter: https://twitter.com/holdenkarau
> >> > Books (Learning Spark, High Performance Spark, etc.):
> >> > https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> >> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> >> >
> >
> >
> > --
> > Twitter: https://twitter.com/holdenkarau
> > Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9
> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message