airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow 1.9.0 status
Date Tue, 31 Oct 2017 17:17:01 GMT
Both blockers have been merged. Are we ready for an RC?

On Mon, Oct 30, 2017 at 11:55 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> AIRFLOW-1764 is design/integration mistake (maybe worth a CVE). But 1765
> is not a vulnerability, although *maybe* unexpected behaviour.
>
> Bolke.
>
>
> > On 30 Oct 2017, at 19:51, Chris Riccomini <criccomini@apache.org> wrote:
> >
> > I marked them as blocker. Still would like to understand how important
> this
> > is, though. The API is experimental, so I could see an argument for them
> > not being blocker provided the vulnerability is limited to experimental
> > areas.
> >
> > On Mon, Oct 30, 2017 at 11:06 AM, Ash Berlin-Taylor <ash@firemirror.com>
> > wrote:
> >
> >> I think we need to add
> >>
> >> https://issues.apache.org/jira/browse/AIRFLOW-1764
> >> https://issues.apache.org/jira/browse/AIRFLOW-1765
> >>
> >> to that list as critical security fixes. I'll tackled 1765 tomorrow (UK
> >> time) if no one gets round to it before then.
> >>
> >> -ash
> >>
> >>> On 30 Oct 2017, at 17:58, Chris Riccomini <criccomini@apache.org>
> wrote:
> >>>
> >>> Hey all,
> >>>
> >>> Current blockers for 1.9.0:
> >>>
> >>> https://issues.apache.org/jira/browse/AIRFLOW-1711
> >>> https://issues.apache.org/jira/browse/AIRFLOW-1018
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On Tue, Oct 24, 2017 at 11:06 AM, Chris Riccomini <
> criccomini@apache.org
> >>>
> >>> wrote:
> >>>
> >>>> Hey all,
> >>>>
> >>>> Talked with @Bolke on Gitter. Here are the currently blocking issues
> for
> >>>> 1.9.0:
> >>>>
> >>>> AIRFLOW-1744      |Bug         ||Blocker   ||Open      |task.retries
> can
> >>>> be False
> >>>> AIRFLOW-1731      |Bug         ||Blocker   ||Open      |Import custom
> >>>> config on PYTHONPATH
> >>>> AIRFLOW-1641      |Bug         ||Blocker   ||In Progress|Task gets
> stuck
> >>>> in queued state
> >>>>
> >>>> PRs are out for them. After these three are merged, I'm planning to
> cut
> >> an
> >>>> RC.
> >>>>
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Fri, Oct 20, 2017 at 7:49 AM, Bolke de Bruin <bdbruin@gmail.com>
> >> wrote:
> >>>>
> >>>>> I upgraded our production environment today. Some observations
> >>>>>
> >>>>> 1. Database migration (add max tries) didn’t workout of the box
> >> correctly
> >>>>> for Postgres. “max_tries = False” didn’t work and needed to be casted
> >> to an
> >>>>> int. Not sure how this gets to be false. This is a block imho
> >>>>> 2. SSHExecuteOperator not having backwards compatibility sucks. We
> >> needed
> >>>>> to rework quite a lot of dags
> >>>>> 3. We use LdAP for logins. We havent configured the ‘superuser’
> >>>>> ‘data_profiler’ groups and thus should have automatic superuser
> >> privileges.
> >>>>> We don’t at the moment (eg. we cannot manage connections or see the
> >> admin
> >>>>> screens).
> >>>>>
> >>>>> After fixing #1, the rest seems quite normal. Happy with the new
> >> logging.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>>
> >>>>>> On 20 Oct 2017, at 12:16, Driesprong, Fokko <fokko@driesprong.frl>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Bolke,
> >>>>>>
> >>>>>> This one is a blocker: https://issues.apache.org/jira
> >>>>> /browse/AIRFLOW-1731.
> >>>>>>
> >>>>>> This requirers that we are sure that the /plugin/ folder is on the
> >> path:
> >>>>>> https://github.com/apache/incubator-airflow/blob/eb2f589099b
> >>>>> 87743482c2eb16261b49e284dcd96/airflow/plugins_manager.py
> >>>>>>
> >>>>>> And maybe add an additional test and updating the docs a bit.
> >> Hopefully
> >>>>>> somewhere in the next few days, however I'm a bit busy with the
> Spark
> >>>>>> Summit.
> >>>>>>
> >>>>>> Cheers, Fokko
> >>>>>>
> >>>>>> 2017-10-19 21:03 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com>:
> >>>>>>
> >>>>>>> Im planning to upgrade our production to 1.9.0alphaX tomorrow.
> >>>>>>>
> >>>>>>> What are the issues that are open?
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Bolke
> >>>>>>>
> >>>>>>>> On 9 Oct 2017, at 20:24, Chris Riccomini <criccomini@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> K, I will plan on cutting an alpha1 later this week. Something for
> >> you
> >>>>>>> guys to play with.
> >>>>>>>>
> >>>>>>>> On Mon, Oct 9, 2017 at 11:23 AM, Bolke de Bruin <
> bdbruin@gmail.com
> >>>>>>> <mailto:bdbruin@gmail.com>> wrote:
> >>>>>>>> hi Chris,
> >>>>>>>>
> >>>>>>>> We are still running pre alpha. I am a bit preoccupied with
> >> preparing
> >>>>>>> for a conference and the team is readying a release of one of our
> >> core
> >>>>>>> products. So it will probably will be after this week when I get my
> >>>>> hands
> >>>>>>> dirty again.
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Bolke
> >>>>>>>>
> >>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>
> >>>>>>>>> Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <
> >> criccomini@apache.org
> >>>>>>> <mailto:criccomini@apache.org>> het volgende geschreven:
> >>>>>>>>>
> >>>>>>>>> Hey all,
> >>>>>>>>>
> >>>>>>>>> Checking in. Has anyone tested this in their environments? Stable
> >>>>>>> releases
> >>>>>>>>> depend on community involvement.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Chris
> >>>>>>>>>
> >>>>>>>>> On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in
> >> alpha0,
> >>>>>>> but
> >>>>>>>>>> will be included in alpha1.
> >>>>>>>>>>
> >>>>>>>>>>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu
> >> <fenglu@google.com.invalid
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Chris,
> >>>>>>>>>>>
> >>>>>>>>>>> I know it's annoying to have last minute commit com in, but
> this
> >>>>> is a
> >>>>>>>>>>> highly desirable feature for folks using GCP operators, is it
> >>>>>>> possible to
> >>>>>>>>>>> include AIRFLOW-1635
> >>>>>>>>>>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 <
> >>>>>>> https://github.com/apache/incubator-airflow/commit/b3e985a3>
> >>>>>>>>>>> 146272ecfd3ceaaa0d8567e4e9e117d4>
> >>>>>>>>>>> in?
> >>>>>>>>>>> More than happy to offer help if there's something I can do.
> >>>>>>>>>>> Thanks a lot.
> >>>>>>>>>>>
> >>>>>>>>>>> Feng
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have cut a 1.9.0alpha0 release of Airflow. You can download
> it
> >>>>>>> here:
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> >>>>>>> 9.0alpha0/ <https://dist.apache.org/repos/dist/dev/incubator/
> >>>>>>> airflow/1.9.0alpha0/>
> >>>>>>>>>>>>
> >>>>>>>>>>>> The bin tarball can be installed with pip:
> >>>>>>>>>>>>
> >>>>>>>>>>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
> >>>>>>>>>>>>
> >>>>>>>>>>>> The goal is to have the community install and run this to
> expose
> >>>>> any
> >>>>>>>>>>> bugs
> >>>>>>>>>>>> before we move on to official release candidates.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Here are the outstanding blocker bugs for 1.9.0:
> >>>>>>>>>>>>
> >>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator
> >> are
> >>>>>>>>>>> marked
> >>>>>>>>>>>> as
> >>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> >>>>>>> exception
> >>>>>>>>>>>> for
> >>>>>>>>>>>> @on
> >>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log
> >> to
> >>>>>>>>>>> stdout
> >>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>> exception
> >>>>>>>>>>> for
> >>>>>>>>>>>> @once
> >>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes it
> >> to
> >>>>>>> fail
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Chris
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <
> >>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <
> >>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Works for me. Will try and cut a beta before end of week.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Blockers for 1.9.0 are:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> SubDagOperator
> >>>>> are
> >>>>>>>>>>>> marked
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task causes
> it
> >>>>> to
> >>>>>>>>>>> fail
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <
> >>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can I suggest releasing a beta? The stable branch is only
> cut
> >>>>> at
> >>>>>>> RC
> >>>>>>>>>>>>>>> time. Betas allow us a broader exposure. It also gives us a
> >>>>>>> point of
> >>>>>>>>>>>>>>> reference.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In addition the list below are mostly longer standing
> issues
> >>>>> that
> >>>>>>>>>>> are
> >>>>>>>>>>>>>>> also part of the 1.8.x branch. Maybe only consider 1611,
> >> 1525,
> >>>>>>> 1258,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> 976 as blocker?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Verstuurd vanaf mijn iPad
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>>>>>>>> het volgende geschreven:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta
> >>>>>>>>>>> release,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>>>> seeing as there are several outstanding bugs, I'm going to
> >>>>>>> delay.
> >>>>>>>>>>>> Here
> >>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>> the bugs that I'm tracking:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE
> issue
> >>>>>>>>>>>>>>>> AIRFLOW-1258 |Bug         |TaskInstances within
> >> SubDagOperator
> >>>>>>> are
> >>>>>>>>>>>>>>> marked as
> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_
> run()
> >>>>>>>>>>>> exception
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not
> >> log
> >>>>>>> to
> >>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas()
> >>>>>>>>>>> exception
> >>>>>>>>>>>>>>> for @once
> >>>>>>>>>>>>>>>> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated
> >> if
> >>>>>>>>>>> Email
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>> Not
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> AIRFLOW-976  |Bug         |Mark success running task
> causes
> >> it
> >>>>>>> to
> >>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> These are the priority issues. Once they're merged, I'll
> cut
> >>>>> the
> >>>>>>>>>>>>>>>> v1-9-stable and beta release.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If you can help clean this up, that would be really
> >>>>> appreciated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Marked it for 1.9.0.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <
> >>>>>>>>>>> cjones@simpli.fi <mailto:cjones@simpli.fi>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is there any chance we could include AIRFLOW-988 in
> 1.9.0?
> >>>>> SLA
> >>>>>>>>>>>>>>> callbacks
> >>>>>>>>>>>>>>>>>> are not working correctly without emails... Its not a
> >> major
> >>>>>>> bug,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> does cause us some annoyance in our current deployment.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Link to Jira:
> >>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Link to PR:
> >>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2415>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>> Charlie Jones
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> CHARLIE JONES
> >>>>>>>>>>>>>>>>>> Data Engineer
> >>>>>>>>>>>>>>>>>> cjones@simpli.fi <mailto:cjones@simpli.fi>  |  M:
> >>>>>>> 972.821.7631 <tel:972.821.7631>
> >>>>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Programmatic Performance.* Localized.*
> >>>>>>>>>>>>>>>>>> __________________________________________________
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>>>>>>>>>>>>>>>>> 800.840.0768 <tel:800.840.0768>  |  www.simpli.fi <
> >>>>>>> http://www.simpli.fi/>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Merged.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>>>>>>>>>>>>>>>>> ryan.buckley@bluecore.com <mailto:ryan.buckley@bluecore
> .
> >> com
> >>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Would it be possible to include AIRFLOW-1587?
> >>>>>>>>>>>>>>>>>>>> Running dags from the UI is currently broken on the
> >> 1.9.0
> >>>>>>>>>>> branch
> >>>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> this issue.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590
> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2590>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>> Ryan
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>>>>>>>>>>>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
> >>>>> release:
> >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> incubator-airflow/pull/2631
> >> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2631>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Currently importing a custom logging configuration is
> >> not
> >>>>>>>>>>> work
> >>>>>>>>>>>> (as
> >>>>>>>>>>>>>>>>>> far
> >>>>>>>>>>>>>>>>>>>> as I
> >>>>>>>>>>>>>>>>>>>>> know). Any feedback on the PR would also be
> >> appreciated.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> >>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Done!
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
> >>>>>>>>>>>>>>>>>>>>>> michael.crawford@modernizingmedicine.com <mailto:
> >>>>>>> michael.crawford@modernizingmedicine.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix
> in?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> <
> >>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>>
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/pull/2626
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2626> <
> >>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/pull/2626
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/pull/2626>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> It keeps the connection type from getting blanked
> out
> >>>>> on
> >>>>>>>>>>> edit
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>>> types.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>> Mike
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking
> >>>>>>> forward
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>>>>>> fixes!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
> >>>>>>>>>>>>>>>>>>>> alex.guziel@airbnb.com <mailto:alex.guziel@airbnb.com
> >
> >>>>>>>>>>>>>>>>>>>>> .
> >>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Can we get this in?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/browse/AIRFLOW-1519
> >> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://issues.apache.org/
> jira/browse/AIRFLOW-1621
> >> <
> >>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/commit/
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/
> >> incubator-airflow/commit/
> >>>>> <
> >>>>>>> https://github.com/apache/incubator-airflow/commit/>
> >>>>>>>>>>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good
> >>>>> thing
> >>>>>>> to
> >>>>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> 1.9.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
> Riccomini <
> >>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> criccomini@apache.org>
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
> >>>>> then.
> >>>>>>>>>>>>>>>>>> Initial
> >>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>> stands, that I will start locking down what can
> >> get
> >>>>>>> into
> >>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
> Bruin <
> >>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a
> >>>>>>> particular
> >>>>>>>>>>>>>>>>>> fixed
> >>>>>>>>>>>>>>>>>>>>> point
> >>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a
> tarball
> >>>>>>> than
> >>>>>>>>>>> to
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> git
> >>>>>>>>>>>>>>>>>>>>>> pull.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly
> >>>>>>>>>>> different?
> >>>>>>>>>>>>>>>>>>> IIRC,
> >>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de
> >> Bruin <
> >>>>>>>>>>>>>>>>>>>>>> bdbruin@gmail.com <mailto:bdbruin@gmail.com>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs
> >>>>> right
> >>>>>>>>>>> away?
> >>>>>>>>>>>>>>>>>>>> Isn’t a
> >>>>>>>>>>>>>>>>>>>>>>>>> beta
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> bit smarter?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Bolke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini <
> >>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm
> planning
> >>>>> to
> >>>>>>>>>>> cut
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote.
> >> Once
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> stable
> >>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cut, I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry
> >>>>>>> picked
> >>>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> branch,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to
> >> get
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> release
> >>>>>>>>>>>>>>>>>>>>> out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris
> >>>>> Riccomini
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> criccomini@apache.org <mailto:
> >>>>> criccomini@apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are
> the
> >>>>>>>>>>>>>>>>>> outstanding
> >>>>>>>>>>>>>>>>>>> PRs
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
> Vulnerability in
> >>>>>>>>>>> Variable
> >>>>>>>>>>>>>>>>>>>>> endpoint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
> logging in
> >>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix log source of
> >>>>> local
> >>>>>>>>>>>>>>>>>> loggers
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename the
> logger to
> >>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix minor
> LICENSE &
> >>>>>>> NOTICE
> >>>>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
> duplicate
> >>>>> and
> >>>>>>>>>>>>>>>>>> unneeded
> >>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator to
> >>>>> operate
> >>>>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
> >>>>>>>>>>> |airflow/jobs.py:create_dag_ru
> >>>>>>>>>>>>>>>>>> n()
> >>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
> >>>>> shouldn't
> >>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>> paused
> >>>>>>>>>>>>>>>>>>>>>>>>>> DAGs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler DAG
> >>>>> processes
> >>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
> displayed
> >>>>> over
> >>>>>>>>>>> task
> >>>>>>>>>>>>>>>>>>>>> instances
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
> >>>>>>>>>>> |airflow/jobs.py:manage_slas()
> >>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @once
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark success
> running
> >>>>>>> task
> >>>>>>>>>>>>>>>>>> causes
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> fail
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>> BackfillJobTest.test_backfill_
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> examples
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
> >>>>>>>>>>>>>>>>>>>>> tests.CoreTest.test_scheduler_
> >>>>>>>>>>>>>>>>>>>>>>>>> job
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> real
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor tests
> and
> >>>>> build
> >>>>>>>>>>>>>>>>>> matrix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators should
> not
> >>>>>>> push
> >>>>>>>>>>>>>>>>>> XComs
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>> default
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add maximum size
> for
> >>>>>>> XComs
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add Dataflow
> >> semantics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
> unexpectedly
> >>>>>>>>>>> added to
> >>>>>>>>>>>>>>>>>>> hive
> >>>>>>>>>>>>>>>>>>>>> conf
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get
> >>>>> cherry-picked
> >>>>>>>>>>> into
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> 1.9.0
> >>>>>>>>>>>>>>>>>>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want
> >> in,
> >>>>>>>>>>> please
> >>>>>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> fix
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to 1.9.0.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our
> >> dev
> >>>>>>>>>>>> cluster,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying
> >> stability.
> >>>>>>> If
> >>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>> run
> >>>>>>>>>>>>>>>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0
> >> test
> >>>>>>>>>>> branch
> >>>>>>>>>>>>>>>>>>>>> somewhere,
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> verify
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. **
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message