airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow Developers Meeting - 08/03 Notes
Date Fri, 09 Sep 2016 15:27:02 GMT
> Maybe we should keep them for Airflow 1.8.

+1

On Sun, Aug 28, 2016 at 8:37 AM, Alex Van Boxel <alex@vanboxel.be> wrote:
> Hey Maxime, I started to list them all and seems to be a lot of them (one
> depending on the other). I think they had some breaking changes. Maybe we
> should keep them for Airflow 1.8. I'll keep running on master for now.
>
>
>
> On Wed, Aug 24, 2016 at 1:59 AM Jakob Homan <jghoman@gmail.com> wrote:
>
>> ASF provides per-projects blog infrastructure:
>> http://www.apache.org/dev/project-blogs
>>
>> -Jakob
>>
>>
>> On 23 August 2016 at 15:07, Maxime Beauchemin
>> <maximebeauchemin@gmail.com> wrote:
>> > I'm planning on working on a minimal apache release this week. If someone
>> > can point to the commit or PRs I can add them to `branch-1.7.2-apache`
>> and
>> > start moving towards the release.
>> >
>> > Max
>> >
>> > On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel <alex@vanboxel.be>
>> wrote:
>> >
>> >> All stuff is in master (I'm running pre-production on master), if it
>> needs
>> >> to be cherry-picked to a branch I can always test it. I would be grate
>> if
>> >> it's in the next release so I can start blogging about it.
>> >>
>> >> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin <bdbruin@gmail.com>
>> wrote:
>> >>
>> >> > (Sorry for the late response, I was on holiday)
>> >> >
>> >> > I think the G* operators just need to be cherry picked. This will
>> make us
>> >> > deviate slightly from the
>> >> > previous release, but makes sure we don’t have to ‘fix’ history
>> >> afterwards.
>> >> >
>> >> > Anyone against this?
>> >> >
>> >> > - B.
>> >> >
>> >> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <alex@vanboxel.be>
>> het
>> >> > volgende geschreven:
>> >> > >
>> >> > > Sorry, Bolke. What needs to be done for Google Cloud
>> operators/hooks?
>> >> > >
>> >> > > If you bring me up-to-speed I can do this. I'm currently working
an
>> a
>> >> CI
>> >> > > setup for the Google stuff).
>> >> > >
>> >> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bdbruin@gmail.com>
>> >> wrote:
>> >> > >
>> >> > >> I have indeed a branch with cherry picked commits. This branch
is
>> >> > >> available in the apache repo. The branch is available as
>> >> > >> “branch-1.7.2-apache”. What is left to do is to decide
weather we
>> do
>> >> > >> upgrade the google cloud operators/hooks as part of this release
as
>> >> the
>> >> > >> licenses were only added after these changes came in. I prefer
this
>> >> > >> approach as we stay away from custom commits as much as possible
>> then.
>> >> > >>
>> >> > >> - Bolke
>> >> > >>
>> >> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
>> >> gurer.kiratli@airbnb.com
>> >> > .INVALID>
>> >> > >> het volgende geschreven:
>> >> > >>>
>> >> > >>> Agenda
>> >> > >>>
>> >> > >>>
>> >> > >>>  -
>> >> > >>>
>> >> > >>>  Committers sync-up: progress and plans
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     Max to do a recap since last release
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     Airbnb
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     Anyone else?
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Cooperation Best Practices
>> >> > >>>  -
>> >> > >>>
>> >> > >>>  Solicit for feedback for Impersonation Design Review
>> >> > >>>  -
>> >> > >>>
>> >> > >>>  Release Schedule, Management
>> >> > >>>  -
>> >> > >>>
>> >> > >>>  Roadmap discussion
>> >> > >>>
>> >> > >>>
>> >> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
>> >> Maxime
>> >> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George
Ke, Arthur
>> >> > Wiedmer,
>> >> > >>> Gurer Kiratli
>> >> > >>>
>> >> > >>> Notes
>> >> > >>>
>> >> > >>>
>> >> > >>>  -
>> >> > >>>
>> >> > >>>  Recap for the overall project
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Max]
>> >> > >>>
>> >> > >>
>> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Airbnb Recap
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Paul] Airbnb is continuing work on features related
to
>> migrating
>> >> > >>>     more jobs onto Airflow - DB connection scalability,
>> >> impersonation,
>> >> > >>>     refreshing web UI, DAG git versioning, task resource
isolation
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Other Recaps
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Chris] Improvements to Import functionality. Joy
did some
>> work
>> >> > >>>     around CLIs and mark success in collaboration with
Bolke.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Jeremiah] Came up with the Merge tool. Focusing on
the work
>> >> around
>> >> > >>>     configuration.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Meeting cadence
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Max] We can have monthly meetings. Let’s start
with monthly
>> we
>> >> can
>> >> > >>>     change the cadence. We should invite all contributors.
We
>> should
>> >> > >>> specify in
>> >> > >>>     the invite that everyone is welcome. Currently posting
this on
>> >> > >>> the mailing
>> >> > >>>     list and wiki.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     *Action Item*  Gurer to schedule monthly meetings
next one at
>> >> > >> Airbnb.
>> >> > >>>     Airbnb is welcoming folks to come onsite
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Blog/Documentation
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Max, Chris, Jeremiah] Having a blog would be useful.
We can
>> >> > >>>     structure our thoughts.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Andrew] Blog should have better language with screenshots
and
>> >> all.
>> >> > >>>     This raises the level of effort bar and this might
mean that
>> >> > >>> nobody really
>> >> > >>>     does it.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Max] Maybe use Medium? We can try it out.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Max] Haven’t looked at the documentation for a
long time. I
>> >> don’t
>> >> > >>>     really know about its quality.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Arthur] Looking at the email list responses, seems
like the
>> >> > >>>     documentation is missing certain elements.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Max] Maybe we can interview people who just started
and
>> identify
>> >> > >>>     gaps.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     *Action Item* [Jeremiah] We can also ask the mailing
list
>> about
>> >> > >> this.
>> >> > >>>     He will send this out.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Cooperation guidelines
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Max, Arthur]
>> >> > >>>
>> >> > >>
>> >> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Chris] Let’s link the guidelines to contributor
md . This
>> should
>> >> > be
>> >> > >>>     discoverable
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Release
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Chris, Max] Bolke might have branch with High charts
and
>> >> Licensing
>> >> > >>>     cherry picked. We can use this as a dry run. We can
test the
>> >> > >> process.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Max] Big release is probably around September.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Andrew] If we want to have a true practice, we need
to be
>> >> > >> diligently
>> >> > >>>     following Apache guidelines.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Max] We might want to revive the email thread about
how we
>> >> should
>> >> > >> do
>> >> > >>>     this.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Chris] It would be great for someone who hasn’t
done this
>> before
>> >> > to
>> >> > >>>     do this.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Andrew] Building a script for this is smart otherwise
the
>> manual
>> >> > >>>     work is painful from experience.
>> >> > >>>
>> >> > >>
>> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
>> >> > >>>
>> >> > >>>
>> >> > >>> https://cwiki.apache.org/confluence/display/JCLOUDS/
>> >> Validate+a+Release
>> >> > >>>
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     *Action Item* [Max] Max will own this. This will happen
by
>> 08/19.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Arthur] For the main release around mid September.
We need to
>> >> > >>>     confirm this date.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Roadmap
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Max] Maybe each company puts their own section? Or
should we
>> >> have
>> >> > a
>> >> > >>>     more cohesive roadmap?
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     *Action Item* Gurer to schedule a roadmap discussion
>> >> > >>>     -
>> >> > >>>
>> >> > >>>  Misc
>> >> > >>>  -
>> >> > >>>
>> >> > >>>     [Max] For executor thinking about sticking with Celery
and
>> maybe
>> >> > >>>     having Docker implemented on top of it.
>> >> > >>>     -
>> >> > >>>
>> >> > >>>     [Max] We might want to explore Sphinx for generating
>> >> documentation
>> >> > >>>     from code.
>> >> > >>>     - [Max] Please review the impersonation design review.
This is
>> >> > >> pretty
>> >> > >>>     simple and reversible.
>> >> > >>
>> >> > >>
>> >> >
>> >> >
>> >>
>>

Mime
View raw message