airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Lu <fen...@google.com.INVALID>
Subject Re: Airflow 1.9.0 status
Date Tue, 03 Oct 2017 23:13:52 GMT
Hi Chris,

I know it's annoying to have last minute commit com in, but this is a
highly desirable feature for folks using GCP operators, is it possible to
include AIRFLOW-1635
<https://github.com/apache/incubator-airflow/commit/b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4>
in?
More than happy to offer help if there's something I can do.
Thanks a lot.

Feng

On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Hey all,
>
> I have cut a 1.9.0alpha0 release of Airflow. You can download it here:
>
>   https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/
>
> The bin tarball can be installed with pip:
>
>   pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz
>
> The goal is to have the community install and run this to expose any bugs
> before we move on to official release candidates.
>
> Here are the outstanding blocker bugs for 1.9.0:
>
>   AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>   AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked
> as
>   AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
> for
> @on
>   AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
>   AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
> @once
>   AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>
> Cheers,
> Chris
>
> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <criccomini@apache.org>
> wrote:
>
> > Welp. Work got in the way, so I'll cut the beta on Monday. :)
> >
> > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <criccomini@apache.org>
> > wrote:
> >
> >> Works for me. Will try and cut a beta before end of week.
> >>
> >> Blockers for 1.9.0 are:
> >>
> >> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> marked
> >> as
> >> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> >>
> >>
> >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bdbruin@gmail.com>
> >> wrote:
> >>
> >>> Hi Chris
> >>>
> >>> Can I suggest releasing a beta? The stable branch is only cut at RC
> >>> time. Betas allow us a broader exposure. It also gives us a point of
> >>> reference.
> >>>
> >>> In addition the list below are mostly longer standing issues that are
> >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258,
> and
> >>> 976 as blocker?
> >>>
> >>> Cheers
> >>> Bolke
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
> >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <
> criccomini@apache.org>
> >>> het volgende geschreven:
> >>> >
> >>> > Hey all,
> >>> >
> >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release,
> but
> >>> > seeing as there are several outstanding bugs, I'm going to delay.
> Here
> >>> are
> >>> > the bugs that I'm tracking:
> >>> >
> >>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
> >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> >>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
> >>> marked as
> >>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run()
> exception
> >>> for
> >>> > @on
> >>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to
> >>> stdout
> >>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception
> >>> for @once
> >>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email
> is
> >>> Not
> >>> > be
> >>> > AIRFLOW-976  |Bug         |Mark success running task causes it to
> fail
> >>> >
> >>> > These are the priority issues. Once they're merged, I'll cut the
> >>> > v1-9-stable and beta release.
> >>> >
> >>> > If you can help clean this up, that would be really appreciated.
> >>> >
> >>> > Cheers,
> >>> > Chris
> >>> >
> >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
> >>> criccomini@apache.org>
> >>> > wrote:
> >>> >
> >>> >> Marked it for 1.9.0.
> >>> >>
> >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cjones@simpli.fi>
> >>> wrote:
> >>> >>>
> >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0?
SLA
> >>> callbacks
> >>> >>> are not working correctly without emails... Its not a major
bug,
> but
> >>> it
> >>> >>> does cause us some annoyance in our current deployment.
> >>> >>>
> >>> >>> Link to Jira:
> >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
> >>> >>>
> >>> >>> Link to PR:
> >>> >>> https://github.com/apache/incubator-airflow/pull/2415
> >>> >>>
> >>> >>> Thanks!
> >>> >>> Charlie Jones
> >>> >>>
> >>> >>> CHARLIE JONES
> >>> >>> Data Engineer
> >>> >>> cjones@simpli.fi  |  M: 972.821.7631
> >>> >>> __________________________________________________
> >>> >>>
> >>> >>>
> >>> >>> Programmatic Performance.* Localized.*
> >>> >>> __________________________________________________
> >>> >>>
> >>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
> >>> >>> 800.840.0768  |  www.simpli.fi
> >>> >>>
> >>> >>>
> >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
> >>> criccomini@apache.org>
> >>> >>> wrote:
> >>> >>>
> >>> >>>> Merged.
> >>> >>>>
> >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
> >>> >>> ryan.buckley@bluecore.com>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> Would it be possible to include AIRFLOW-1587?
> >>> >>>>> Running dags from the UI is currently broken on the
1.9.0 branch
> >>> due
> >>> >>> to
> >>> >>>>> this issue.
> >>> >>>>>
> >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
> >>> >>>>>
> >>> >>>>> Thanks,
> >>> >>>>> Ryan
> >>> >>>>>
> >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
> >>> >>> <fokko@driesprong.frl
> >>> >>>>>
> >>> >>>>> wrote:
> >>> >>>>>
> >>> >>>>>> Hi All,
> >>> >>>>>>
> >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0
release:
> >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
> >>> >>>>>>
> >>> >>>>>> Currently importing a custom logging configuration
is not work
> (as
> >>> >>> far
> >>> >>>>> as I
> >>> >>>>>> know). Any feedback on the PR would also be appreciated.
> >>> >>>>>>
> >>> >>>>>> Cheers, Fokko
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <
> criccomini@apache.org
> >>> >:
> >>> >>>>>>
> >>> >>>>>>> Done!
> >>> >>>>>>>
> >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford
<
> >>> >>>>>>> michael.crawford@modernizingmedicine.com>
wrote:
> >>> >>>>>>>
> >>> >>>>>>>> Can you slide the aws and emr connection
type fix in?
> >>> >>>>>>>>
> >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
<
> >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
> >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
<
> >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
> >>> >>>>>>>>
> >>> >>>>>>>> It keeps the connection type from getting
blanked out on edit
> >>> >>> for
> >>> >>>>> these
> >>> >>>>>>>> types.
> >>> >>>>>>>>
> >>> >>>>>>>> Thanks,
> >>> >>>>>>>> Mike
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris
Riccomini <
> >>> >>>>> criccomini@apache.org>
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Absolutely. Just cherry-picked. I've
been looking forward to
> >>> >>>> these
> >>> >>>>>>> fixes!
> >>> >>>>>>>>>
> >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex
Guziel <
> >>> >>>>> alex.guziel@airbnb.com
> >>> >>>>>> .
> >>> >>>>>>>> invalid
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>
> >>> >>>>>>>>>> Can we get this in?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
> >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
> >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
> >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> It speeds up loading times a lot,
so it's a good thing to
> >>> >>> have
> >>> >>>> in
> >>> >>>>>> 1.9.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM,
Chris Riccomini <
> >>> >>>>>>>> criccomini@apache.org>
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta
next week, then.
> >>> >>> Initial
> >>> >>>>>>> warning
> >>> >>>>>>>>>>> stands, that I will start locking
down what can get into
> >>> >>> 1.9.0
> >>> >>>> at
> >>> >>>>>>> that
> >>> >>>>>>>>>>> point.
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10
AM, Bolke de Bruin <
> >>> >>>>>> bdbruin@gmail.com>
> >>> >>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>>> No vote indeed, just to
gather feedback on a particular
> >>> >>> fixed
> >>> >>>>>> point
> >>> >>>>>>> in
> >>> >>>>>>>>>>>> time. It also gives a bit
more trust to a tarball than to
> a
> >>> >>>> git
> >>> >>>>>>> pull.
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>> Bolke
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>>> On 20 Sep 2017, at
20:09, Chris Riccomini <
> >>> >>>>> criccomini@apache.org
> >>> >>>>>>>
> >>> >>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>
> >>> >>>>>>>>>>>>> I can do a beta. Is
the process significantly different?
> >>> >>>> IIRC,
> >>> >>>>>> it's
> >>> >>>>>>>>>>>>> basically the same,
just no vote, right?
> >>> >>>>>>>>>>>>>
> >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017
at 10:56 AM, Bolke de Bruin <
> >>> >>>>>>> bdbruin@gmail.com>
> >>> >>>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>> Are you sure you
want to go ahead and do RCs right away?
> >>> >>>>> Isn’t a
> >>> >>>>>>>>>> beta
> >>> >>>>>>>>>>> a
> >>> >>>>>>>>>>>>>> bit smarter?
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>> - Bolke
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> On 20 Sep 2017,
at 19:41, Chris Riccomini <
> >>> >>>>>> criccomini@apache.org
> >>> >>>>>>>>
> >>> >>>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> Hey all,
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> I want to send
out a warning that I'm planning to cut
> >>> >>> the
> >>> >>>>>> stable
> >>> >>>>>>>>>>> branch
> >>> >>>>>>>>>>>>>>> next week,
and begin the RC1 release vote. Once the
> >>> >>> stable
> >>> >>>>>> branch
> >>> >>>>>>>>>> is
> >>> >>>>>>>>>>>>>> cut, I
> >>> >>>>>>>>>>>>>>> will be locking
down what commits get cherry picked
> into
> >>> >>>> the
> >>> >>>>>>>>>> branch,
> >>> >>>>>>>>>>>> and
> >>> >>>>>>>>>>>>>>> will only be
doing PRs that are required to get the
> >>> >>> release
> >>> >>>>>> out.
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> Cheers,
> >>> >>>>>>>>>>>>>>> Chris
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> On Mon, Sep
18, 2017 at 11:19 AM, Chris Riccomini <
> >>> >>>>>>>>>>>> criccomini@apache.org
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>> wrote:
> >>> >>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> Hey all,
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> An update
on the 1.9.0 release. Here are the
> >>> >>> outstanding
> >>> >>>> PRs
> >>> >>>>>>> that
> >>> >>>>>>>>>>> are
> >>> >>>>>>>>>>>>>>>> slated
to be included into 1.9.0:
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> ISSUE ID
    |STATUS    |DESCRIPTION
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1617
|Open      |XSS Vulnerability in Variable
> >>> >>>>>> endpoint
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1611
|Open      |Customize logging in Airflow
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1605
|Reopened  |Fix log source of local
> >>> >>> loggers
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1604
|Open      |Rename the logger to log
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1525
|Open      |Fix minor LICENSE & NOTICE
> >>> >>> issue
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1499
|In Progres|Eliminate duplicate and
> >>> >>> unneeded
> >>> >>>>> code
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1198
|Open      |HDFSOperator to operate HDFS
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1055
|Open      |airflow/jobs.py:create_dag_ru
> >>> >>> n()
> >>> >>>>>>>>>> exception
> >>> >>>>>>>>>>>> for
> >>> >>>>>>>>>>>>>>>> @on
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1019
|Open      |active_dagruns shouldn't
> >>> >>> include
> >>> >>>>>> paused
> >>> >>>>>>>>>>> DAGs
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1018
|Open      |Scheduler DAG processes can
> >>> >>> not
> >>> >>>> log
> >>> >>>>>> to
> >>> >>>>>>>>>>> stdout
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1015
|Open      |TreeView displayed over task
> >>> >>>>>> instances
> >>> >>>>>>>>>>>>>>>> AIRFLOW-1013
|Open      |airflow/jobs.py:manage_slas()
> >>> >>>>>> exception
> >>> >>>>>>>>>> for
> >>> >>>>>>>>>>>>>>>> @once
> >>> >>>>>>>>>>>>>>>> AIRFLOW-976
 |Open      |Mark success running task
> >>> >>> causes
> >>> >>>> it
> >>> >>>>>> to
> >>> >>>>>>>>>> fail
> >>> >>>>>>>>>>>>>>>> AIRFLOW-914
 |Open      |Refactor
> >>> >>>>>> BackfillJobTest.test_backfill_
> >>> >>>>>>>>>>>>>> examples
> >>> >>>>>>>>>>>>>>>> to
> >>> >>>>>>>>>>>>>>>> AIRFLOW-913
 |Open      |Refactor
> >>> >>>>>> tests.CoreTest.test_scheduler_
> >>> >>>>>>>>>> job
> >>> >>>>>>>>>>>> to
> >>> >>>>>>>>>>>>>>>> real
> >>> >>>>>>>>>>>>>>>> AIRFLOW-912
 |Open      |Refactor tests and build
> >>> >>> matrix
> >>> >>>>>>>>>>>>>>>> AIRFLOW-888
 |Open      |Operators should not push
> >>> >>> XComs
> >>> >>>> by
> >>> >>>>>>>>>> default
> >>> >>>>>>>>>>>>>>>> AIRFLOW-828
 |Open      |Add maximum size for XComs
> >>> >>>>>>>>>>>>>>>> AIRFLOW-825
 |Open      |Add Dataflow semantics
> >>> >>>>>>>>>>>>>>>> AIRFLOW-788
 |Open      |Context unexpectedly added to
> >>> >>>> hive
> >>> >>>>>> conf
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> I will
be locking down what can get cherry-picked into
> >>> >>> the
> >>> >>>>>> 1.9.0
> >>> >>>>>>>>>>>> branch
> >>> >>>>>>>>>>>>>>>> shortly,
so if you have something you want in, please
> >>> >>> set
> >>> >>>>> the
> >>> >>>>>>> fix
> >>> >>>>>>>>>>>>>> version
> >>> >>>>>>>>>>>>>>>> to 1.9.0.
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> We (at
WePay) have deployed 1.9.0 into our dev
> cluster,
> >>> >>>> and
> >>> >>>>> it
> >>> >>>>>>> has
> >>> >>>>>>>>>>>> been
> >>> >>>>>>>>>>>>>>>> running
smoothly for several days.
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> ** I could
really use help verifying stability. If you
> >>> >>> run
> >>> >>>>>>>>>> Airflow,
> >>> >>>>>>>>>>>> it's
> >>> >>>>>>>>>>>>>>>> in your
best interest to deploy the 1.9.0 test branch
> >>> >>>>>> somewhere,
> >>> >>>>>>>>>> and
> >>> >>>>>>>>>>>>>> verify
> >>> >>>>>>>>>>>>>>>> it's working
for your workload. **
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>>> Cheers,
> >>> >>>>>>>>>>>>>>>> Chris
> >>> >>>>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>>>
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>>
> >>> >>>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message