airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From siddharth anand <san...@apache.org>
Subject Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5
Date Wed, 15 Mar 2017 21:06:12 GMT
Here's the JIRA :
https://issues.apache.org/jira/browse/AIRFLOW-989

I confirmed it is a regression from 1.7.1.3, which I installed via pip and
tested against the same DAG in the JIRA.

The issue occurs if a leaf / last / terminal downstream task is not
cleared. You won't see this issue if you clear the entire DAG Run or clear
a task and all of its downstream tasks. If you truly want to only clear and
rerun a task, but not its downstream tasks, you can use the CLI to execute
a specific task (e.g. vial airflow run).

This is a change in behavior -- if we do go ahead with the release, then
this JIRA should be in a list of JIRAs of known issues related to the new
version.
-s

On Wed, Mar 15, 2017 at 9:17 AM, Chris Riccomini <criccomini@apache.org>
wrote:

> @Sid, does this happen if you clear downstream as well?
>
> On Wed, Mar 15, 2017 at 9:04 AM, Chris Riccomini <criccomini@apache.org>
> wrote:
>
> > Has anyone been able to reproduce Sid's issue?
> >
> > On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbruin@gmail.com>
> > wrote:
> >
> >> That is not an airflow error, but a Kerberos error. Try executing the
> >> kinit command on the command line by yourself.
> >>
> >> Bolke
> >>
> >> Sent from my iPhone
> >>
> >> > On 14 Mar 2017, at 23:11, Ruslan Dautkhanov <dautkhanov@gmail.com>
> >> wrote:
> >> >
> >> > `airflow kerberos` is broken in 1.8-rc5
> >> > https://issues.apache.org/jira/browse/AIRFLOW-987
> >> > Hopefully fix can be part of the 1.8 release.
> >> >
> >> >
> >> >
> >> > --
> >> > Ruslan Dautkhanov
> >> >
> >> >> On Tue, Mar 14, 2017 at 6:19 PM, siddharth anand <sanand@apache.org>
> >> wrote:
> >> >>
> >> >> FYI,
> >> >> I've just hit a major bug in the release candidate related to "clear
> >> task"
> >> >> behavior.
> >> >>
> >> >> I've been running airflow in both stage and prod since yesterday on
> >> rc5 and
> >> >> have reproduced this in both environments. I will file a JIRA for
> this
> >> >> tonight, but wanted to send a note over email as well.
> >> >>
> >> >> In my example, I have a 2 task DAG. For a given DAG run that has
> >> completed
> >> >> successfully, if I
> >> >> 1) clear task2 (leaf task in this case), the previously-successful
> DAG
> >> Run
> >> >> goes back to Running, requeues, and executes the task successfully.
> >> The DAG
> >> >> Run the returns from Running to Success.
> >> >> 2) clear task1 (root task in this case), the previously-successful
> DAG
> >> Run
> >> >> goes back to Running, DOES NOT requeue or execute the task at all.
> The
> >> DAG
> >> >> Run the returns from Running to Success though it never ran the task.
> >> >>
> >> >> 1) is expected and previous behavior. 2) is a regression.
> >> >>
> >> >> The only workaround is to use the CLI to run the task cleared. Here
> are
> >> >> some images :
> >> >> *After Clearing the Tasks*
> >> >> https://www.dropbox.com/s/wmuxt0krwx6wurr/Screenshot%
> >> >> 202017-03-14%2014.09.34.png?dl=0
> >> >>
> >> >> *After DAG Runs return to Success*
> >> >> https://www.dropbox.com/s/qop933rzgdzchpd/Screenshot%
> >> >> 202017-03-14%2014.09.49.png?dl=0
> >> >>
> >> >> This is a major regression because it will force everyone to use the
> >> CLI
> >> >> for things that they would normally use the UI for.
> >> >>
> >> >> -s
> >> >>
> >> >>
> >> >> -s
> >> >>
> >> >>
> >> >>> On Tue, Mar 14, 2017 at 1:32 PM, Daniel Huang <dxhuang@gmail.com>
> >> wrote:
> >> >>>
> >> >>> +1 (non-binding)!
> >> >>>
> >> >>> On Tue, Mar 14, 2017 at 11:35 AM, siddharth anand <
> sanand@apache.org>
> >> >>> wrote:
> >> >>>
> >> >>>> +1 (binding)
> >> >>>>
> >> >>>>
> >> >>>> On Tue, Mar 14, 2017 at 8:42 AM, Maxime Beauchemin <
> >> >>>> maximebeauchemin@gmail.com> wrote:
> >> >>>>
> >> >>>>> +1 (binding)
> >> >>>>>
> >> >>>>> On Tue, Mar 14, 2017 at 3:59 AM, Alex Van Boxel <alex@vanboxel.be
> >
> >> >>>> wrote:
> >> >>>>>
> >> >>>>>> +1 (binding)
> >> >>>>>>
> >> >>>>>> Note: we had to revert all our ONE_SUCCESS with ALL_SUCCESS
> trigger
> >> >>>> rules
> >> >>>>>> where the parent nodes where joining with a SKIP. But
I can of
> >> >> should
> >> >>>>> have
> >> >>>>>> known this was coming. Apart of that I had a successful
run last
> >> >>> night.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Tue, Mar 14, 2017 at 1:37 AM siddharth anand <
> sanand@apache.org
> >> >>>
> >> >>>>> wrote:
> >> >>>>>>
> >> >>>>>> I'm going to deploy this to staging now. Fab work Bolke!
> >> >>>>>> -s
> >> >>>>>>
> >> >>>>>> On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov <
> >> >> dan.davydov@airbnb.com
> >> >>> .
> >> >>>>>> invalid
> >> >>>>>>> wrote:
> >> >>>>>>
> >> >>>>>>> I'll test this on staging as soon as I get a chance
(the testing
> >> >> is
> >> >>>>>>> non-blocking on the rc5). Bolke very much in particular
:).
> >> >>>>>>>
> >> >>>>>>> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin
<
> >> >>> jlowin@apache.org>
> >> >>>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>>> +1 (binding) extremely impressed by the work
and diligence all
> >> >>>>>>> contributors
> >> >>>>>>>> have put in to getting these blockers fixed,
Bolke in
> >> >> particular.
> >> >>>>>>>>
> >> >>>>>>>> On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer
<
> >> >>> arthur@apache.org>
> >> >>>>>>> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> +1 (binding)
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks again for steering us through Bolke.
> >> >>>>>>>>>
> >> >>>>>>>>> Best,
> >> >>>>>>>>> Arthur
> >> >>>>>>>>>
> >> >>>>>>>>> On Sun, Mar 12, 2017 at 9:59 PM, Bolke
de Bruin <
> >> >>>> bdbruin@gmail.com
> >> >>>>>>
> >> >>>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>> Dear All,
> >> >>>>>>>>>>
> >> >>>>>>>>>> Finally, I have been able to make the
FIFTH RELEASE
> >> >> CANDIDATE
> >> >>>> of
> >> >>>>>>>> Airflow
> >> >>>>>>>>>> 1.8.0 available at: https://dist.apache.org/repos/
> >> >>>>>>>>>> dist/dev/incubator/airflow/ <https://dist.apache.org/
> >> >>>>>>>>>> repos/dist/dev/incubator/airflow/>
, public keys are
> >> >>> available
> >> >>>>> at
> >> >>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
> >> >>> airflow/
> >> >>>> <
> >> >>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
> >> >>> airflow/>
> >> >>>> .
> >> >>>>>> It
> >> >>>>>>> is
> >> >>>>>>>>>> tagged with a local version “apache.incubating”
so it
> >> >> allows
> >> >>>>>>> upgrading
> >> >>>>>>>>> from
> >> >>>>>>>>>> earlier releases.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Issues fixed since rc4:
> >> >>>>>>>>>>
> >> >>>>>>>>>> [AIRFLOW-900] Double trigger should
not kill original task
> >> >>>>> instance
> >> >>>>>>>>>> [AIRFLOW-900] Fixes bugs in LocalTaskJob
for double run
> >> >>>>> protection
> >> >>>>>>>>>> [AIRFLOW-932] Do not mark tasks removed
when backfilling
> >> >>>>>>>>>> [AIRFLOW-961] run onkill when SIGTERMed
> >> >>>>>>>>>> [AIRFLOW-910] Use parallel task execution
for backfills
> >> >>>>>>>>>> [AIRFLOW-967] Wrap strings in native
for py2 ldap
> >> >>> compatibility
> >> >>>>>>>>>> [AIRFLOW-941] Use defined parameters
for psycopg2
> >> >>>>>>>>>> [AIRFLOW-719] Prevent DAGs from ending
prematurely
> >> >>>>>>>>>> [AIRFLOW-938] Use test for True in
task_stats queries
> >> >>>>>>>>>> [AIRFLOW-937] Improve performance of
task_stats
> >> >>>>>>>>>> [AIRFLOW-933] use ast.literal_eval
rather eval because
> >> >>>>>>> ast.literal_eval
> >> >>>>>>>>>> does not execute input.
> >> >>>>>>>>>> [AIRFLOW-919] Running tasks with no
start date shouldn't
> >> >>> break
> >> >>>> a
> >> >>>>>> DAGs
> >> >>>>>>>> UI
> >> >>>>>>>>>> [AIRFLOW-897] Prevent dagruns from
failing with unfinished
> >> >>>> tasks
> >> >>>>>>>>>> [AIRFLOW-861] make pickle_info endpoint
be login_required
> >> >>>>>>>>>> [AIRFLOW-853] use utf8 encoding for
stdout line decode
> >> >>>>>>>>>> [AIRFLOW-856] Make sure execution date
is set for local
> >> >>> client
> >> >>>>>>>>>> [AIRFLOW-830][AIRFLOW-829][AIRFLOW-88]
Reduce Travis log
> >> >>>>> verbosity
> >> >>>>>>>>>> [AIRFLOW-794] Access DAGS_FOLDER and
SQL_ALCHEMY_CONN
> >> >>>> exclusively
> >> >>>>>>> from
> >> >>>>>>>>>> settings
> >> >>>>>>>>>> [AIRFLOW-694] Fix config behaviour
for empty envvar
> >> >>>>>>>>>> [AIRFLOW-365] Set dag.fileloc explicitly
and use for Code
> >> >>> view
> >> >>>>>>>>>> [AIRFLOW-931] Do not set QUEUED in
TaskInstances
> >> >>>>>>>>>> [AIRFLOW-899] Tasks in SCHEDULED state
should be white in
> >> >> the
> >> >>>> UI
> >> >>>>>>>> instead
> >> >>>>>>>>>> of black
> >> >>>>>>>>>> [AIRFLOW-895] Address Apache release
incompliancies
> >> >>>>>>>>>> [AIRFLOW-893][AIRFLOW-510] Fix crashing
webservers when a
> >> >>>> dagrun
> >> >>>>>> has
> >> >>>>>>> no
> >> >>>>>>>>>> start date
> >> >>>>>>>>>> [AIRFLOW-793] Enable compressed loading
in S3ToHiveTransfer
> >> >>>>>>>>>> [AIRFLOW-863] Example DAGs should have
recent start dates
> >> >>>>>>>>>> [AIRFLOW-869] Refactor mark success
functionality
> >> >>>>>>>>>> [AIRFLOW-856] Make sure execution date
is set for local
> >> >>> client
> >> >>>>>>>>>> [AIRFLOW-814] Fix Presto*CheckOperator.__init__
> >> >>>>>>>>>> [AIRFLOW-844] Fix cgroups directory
creation
> >> >>>>>>>>>>
> >> >>>>>>>>>> No known issues anymore.
> >> >>>>>>>>>>
> >> >>>>>>>>>> I would also like to raise a VOTE for
releasing 1.8.0 based
> >> >>> on
> >> >>>>>>> release
> >> >>>>>>>>>> candidate 5, i.e. just renaming release
candidate 5 to
> >> >> 1.8.0
> >> >>>>>> release.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Please respond to this email by:
> >> >>>>>>>>>>
> >> >>>>>>>>>> +1,0,-1 with *binding* if you are a
PMC member or
> >> >>> *non-binding*
> >> >>>>> if
> >> >>>>>>> you
> >> >>>>>>>>> are
> >> >>>>>>>>>> not.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks!
> >> >>>>>>>>>> Bolke
> >> >>>>>>>>>>
> >> >>>>>>>>>> My VOTE: +1 (binding)
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>> --
> >> >>>>>>  _/
> >> >>>>>> _/ Alex Van Boxel
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message