airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow 1.9.0 status
Date Thu, 28 Sep 2017 17:49:14 GMT
Hey all,

I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
seeing as there are several outstanding bugs, I'm going to delay. Here are
the bugs that I'm tracking:

AIRFLOW-1611 |Bug         |Customize logging in Airflow
AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked as
AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception for
@on
AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for @once
AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is Not
be
AIRFLOW-976  |Bug         |Mark success running task causes it to fail

These are the priority issues. Once they're merged, I'll cut the
v1-9-stable and beta release.

If you can help clean this up, that would be really appreciated.

Cheers,
Chris

On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <criccomini@apache.org>
wrote:

> Marked it for 1.9.0.
>
> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cjones@simpli.fi> wrote:
>
>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA callbacks
>> are not working correctly without emails... Its not a major bug, but it
>> does cause us some annoyance in our current deployment.
>>
>> Link to Jira:
>> https://issues.apache.org/jira/browse/AIRFLOW-988
>>
>> Link to PR:
>> https://github.com/apache/incubator-airflow/pull/2415
>>
>> Thanks!
>> Charlie Jones
>>
>> CHARLIE JONES
>> Data Engineer
>> cjones@simpli.fi  |  M: 972.821.7631
>> __________________________________________________
>>
>>
>> Programmatic Performance.* Localized.*
>> __________________________________________________
>>
>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>> 800.840.0768  |  www.simpli.fi
>>
>>
>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <criccomini@apache.org>
>> wrote:
>>
>> > Merged.
>> >
>> > On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>> ryan.buckley@bluecore.com>
>> > wrote:
>> >
>> > > Would it be possible to include AIRFLOW-1587?
>> > > Running dags from the UI is currently broken on the 1.9.0 branch due
>> to
>> > > this issue.
>> > >
>> > > https://github.com/apache/incubator-airflow/pull/2590
>> > >
>> > > Thanks,
>> > > Ryan
>> > >
>> > > On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>> <fokko@driesprong.frl
>> > >
>> > > wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > I would like to include AIRFLOW-1611 in the 1.9.0 release:
>> > > > https://github.com/apache/incubator-airflow/pull/2631
>> > > >
>> > > > Currently importing a custom logging configuration is not work (as
>> far
>> > > as I
>> > > > know). Any feedback on the PR would also be appreciated.
>> > > >
>> > > > Cheers, Fokko
>> > > >
>> > > >
>> > > > 2017-09-25 23:27 GMT+02:00 Chris Riccomini <criccomini@apache.org>:
>> > > >
>> > > > > Done!
>> > > > >
>> > > > > On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>> > > > > michael.crawford@modernizingmedicine.com> wrote:
>> > > > >
>> > > > > > Can you slide the aws and emr connection type fix in?
>> > > > > >
>> > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>> > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1636>
>> > > > > > https://github.com/apache/incubator-airflow/pull/2626 <
>> > > > > > https://github.com/apache/incubator-airflow/pull/2626>
>> > > > > >
>> > > > > > It keeps the connection type from getting blanked out on
edit
>> for
>> > > these
>> > > > > > types.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mike
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > > On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>> > > criccomini@apache.org>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Absolutely. Just cherry-picked. I've been looking forward
to
>> > these
>> > > > > fixes!
>> > > > > > >
>> > > > > > > On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>> > > alex.guziel@airbnb.com
>> > > > .
>> > > > > > invalid
>> > > > > > >> wrote:
>> > > > > > >
>> > > > > > >> Can we get this in?
>> > > > > > >>
>> > > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> > > > > > >> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> > > > > > >>
>> > > > > > >> https://github.com/apache/incubator-airflow/commit/
>> > > > > > >> b6d2e0a46978e93e16576604624f57d1388814f2
>> > > > > > >> https://github.com/apache/incubator-airflow/commit/
>> > > > > > >> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> > > > > > >>
>> > > > > > >> It speeds up loading times a lot, so it's a good
thing to
>> have
>> > in
>> > > > 1.9.
>> > > > > > >>
>> > > > > > >> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini
<
>> > > > > > criccomini@apache.org>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >>> Sounds good. I'll plan on stable+beta next
week, then.
>> Initial
>> > > > > warning
>> > > > > > >>> stands, that I will start locking down what
can get into
>> 1.9.0
>> > at
>> > > > > that
>> > > > > > >>> point.
>> > > > > > >>>
>> > > > > > >>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de
Bruin <
>> > > > bdbruin@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>>
>> > > > > > >>>> No vote indeed, just to gather feedback
on a particular
>> fixed
>> > > > point
>> > > > > in
>> > > > > > >>>> time. It also gives a bit more trust to
a tarball than to a
>> > git
>> > > > > pull.
>> > > > > > >>>>
>> > > > > > >>>> Bolke
>> > > > > > >>>>
>> > > > > > >>>>> On 20 Sep 2017, at 20:09, Chris Riccomini
<
>> > > criccomini@apache.org
>> > > > >
>> > > > > > >>> wrote:
>> > > > > > >>>>>
>> > > > > > >>>>> I can do a beta. Is the process significantly
different?
>> > IIRC,
>> > > > it's
>> > > > > > >>>>> basically the same, just no vote, right?
>> > > > > > >>>>>
>> > > > > > >>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke
de Bruin <
>> > > > > bdbruin@gmail.com>
>> > > > > > >>>> wrote:
>> > > > > > >>>>>
>> > > > > > >>>>>> Are you sure you want to go ahead
and do RCs right away?
>> > > Isn’t a
>> > > > > > >> beta
>> > > > > > >>> a
>> > > > > > >>>>>> bit smarter?
>> > > > > > >>>>>>
>> > > > > > >>>>>> - Bolke
>> > > > > > >>>>>>
>> > > > > > >>>>>>> On 20 Sep 2017, at 19:41, Chris
Riccomini <
>> > > > criccomini@apache.org
>> > > > > >
>> > > > > > >>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Hey all,
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> I want to send out a warning
that I'm planning to cut
>> the
>> > > > stable
>> > > > > > >>> branch
>> > > > > > >>>>>>> next week, and begin the RC1
release vote. Once the
>> stable
>> > > > branch
>> > > > > > >> is
>> > > > > > >>>>>> cut, I
>> > > > > > >>>>>>> will be locking down what commits
get cherry picked into
>> > the
>> > > > > > >> branch,
>> > > > > > >>>> and
>> > > > > > >>>>>>> will only be doing PRs that
are required to get the
>> release
>> > > > out.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Cheers,
>> > > > > > >>>>>>> Chris
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Mon, Sep 18, 2017 at 11:19
AM, Chris Riccomini <
>> > > > > > >>>> criccomini@apache.org
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> Hey all,
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> An update on the 1.9.0
release. Here are the
>> outstanding
>> > PRs
>> > > > > that
>> > > > > > >>> are
>> > > > > > >>>>>>>> slated to be included into
1.9.0:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> ISSUE ID     |STATUS  
 |DESCRIPTION
>> > > > > > >>>>>>>> AIRFLOW-1617 |Open    
 |XSS Vulnerability in Variable
>> > > > endpoint
>> > > > > > >>>>>>>> AIRFLOW-1611 |Open    
 |Customize logging in Airflow
>> > > > > > >>>>>>>> AIRFLOW-1605 |Reopened
 |Fix log source of local
>> loggers
>> > > > > > >>>>>>>> AIRFLOW-1604 |Open    
 |Rename the logger to log
>> > > > > > >>>>>>>> AIRFLOW-1525 |Open    
 |Fix minor LICENSE & NOTICE
>> issue
>> > > > > > >>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
duplicate and
>> unneeded
>> > > code
>> > > > > > >>>>>>>> AIRFLOW-1198 |Open    
 |HDFSOperator to operate HDFS
>> > > > > > >>>>>>>> AIRFLOW-1055 |Open    
 |airflow/jobs.py:create_dag_ru
>> n()
>> > > > > > >> exception
>> > > > > > >>>> for
>> > > > > > >>>>>>>> @on
>> > > > > > >>>>>>>> AIRFLOW-1019 |Open    
 |active_dagruns shouldn't
>> include
>> > > > paused
>> > > > > > >>> DAGs
>> > > > > > >>>>>>>> AIRFLOW-1018 |Open    
 |Scheduler DAG processes can
>> not
>> > log
>> > > > to
>> > > > > > >>> stdout
>> > > > > > >>>>>>>> AIRFLOW-1015 |Open    
 |TreeView displayed over task
>> > > > instances
>> > > > > > >>>>>>>> AIRFLOW-1013 |Open    
 |airflow/jobs.py:manage_slas()
>> > > > exception
>> > > > > > >> for
>> > > > > > >>>>>>>> @once
>> > > > > > >>>>>>>> AIRFLOW-976  |Open    
 |Mark success running task
>> causes
>> > it
>> > > > to
>> > > > > > >> fail
>> > > > > > >>>>>>>> AIRFLOW-914  |Open    
 |Refactor
>> > > > BackfillJobTest.test_backfill_
>> > > > > > >>>>>> examples
>> > > > > > >>>>>>>> to
>> > > > > > >>>>>>>> AIRFLOW-913  |Open    
 |Refactor
>> > > > tests.CoreTest.test_scheduler_
>> > > > > > >> job
>> > > > > > >>>> to
>> > > > > > >>>>>>>> real
>> > > > > > >>>>>>>> AIRFLOW-912  |Open    
 |Refactor tests and build
>> matrix
>> > > > > > >>>>>>>> AIRFLOW-888  |Open    
 |Operators should not push
>> XComs
>> > by
>> > > > > > >> default
>> > > > > > >>>>>>>> AIRFLOW-828  |Open    
 |Add maximum size for XComs
>> > > > > > >>>>>>>> AIRFLOW-825  |Open    
 |Add Dataflow semantics
>> > > > > > >>>>>>>> AIRFLOW-788  |Open    
 |Context unexpectedly added to
>> > hive
>> > > > conf
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> I will be locking down
what can get cherry-picked into
>> the
>> > > > 1.9.0
>> > > > > > >>>> branch
>> > > > > > >>>>>>>> shortly, so if you have
something you want in, please
>> set
>> > > the
>> > > > > fix
>> > > > > > >>>>>> version
>> > > > > > >>>>>>>> to 1.9.0.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> We (at WePay) have deployed
1.9.0 into our dev cluster,
>> > and
>> > > it
>> > > > > has
>> > > > > > >>>> been
>> > > > > > >>>>>>>> running smoothly for several
days.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> ** I could really use help
verifying stability. If you
>> run
>> > > > > > >> Airflow,
>> > > > > > >>>> it's
>> > > > > > >>>>>>>> in your best interest to
deploy the 1.9.0 test branch
>> > > > somewhere,
>> > > > > > >> and
>> > > > > > >>>>>> verify
>> > > > > > >>>>>>>> it's working for your workload.
**
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Cheers,
>> > > > > > >>>>>>>> Chris
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>
>> > > > > > >>>>
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message