airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3
Date Mon, 13 Feb 2017 19:59:50 GMT
https://github.com/apache/incubator-airflow/pull/2075 <https://github.com/apache/incubator-airflow/pull/2075>


Is (part of) the fix. I can include it retroactively if needed, but I don’t consider it
blocking.

Bolke


> On 13 Feb 2017, at 20:56, Dan Davydov <dan.davydov@airbnb.com.INVALID> wrote:
> 
> Can you give more details/a repro case Sid? FWIW mark success and clear
> both work for me.
> 
> On Mon, Feb 13, 2017 at 11:51 AM, siddharth anand <sanand@apache.org> wrote:
> 
>> Folks!
>> I need to change my vote.. -1 (Binding).
>> 
>> 
>> Mark Success/Clear is broken in the UI. It's a regression.
>> 
>> -s
>> 
>> On Mon, Feb 13, 2017 at 10:53 AM, Alex Van Boxel <alex@vanboxel.be> wrote:
>> 
>>> +1 (binding)
>>> 
>>> On Mon, Feb 13, 2017 at 7:45 PM siddharth anand <sanand@apache.org>
>> wrote:
>>> 
>>>> +1 (binding)
>>>> 
>>>> On Mon, Feb 13, 2017 at 8:57 AM, Chris Riccomini <
>> criccomini@apache.org>
>>>> wrote:
>>>> 
>>>>> +1 (binding)
>>>>> 
>>>>> On Sun, Feb 12, 2017 at 8:54 AM, Jeremiah Lowin <jlowin@apache.org>
>>>> wrote:
>>>>> 
>>>>>> Interesting -- I also run on Kubernetes with a git-sync sidecar,
>> but
>>>> the
>>>>>> containers wait for the synced repo to apprar before starting since
>>> it
>>>>>> contains some dependencies -- I assume that's why I didn't
>> experience
>>>> the
>>>>>> same issue.
>>>>>> 
>>>>>> On Sun, Feb 12, 2017 at 6:29 AM Bolke de Bruin <bdbruin@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> Although the race condition doesn't explain why “num_runs =
None”
>>>>>> resolved
>>>>>>> the issue for you earlier, but it does give a clue now: the PR
>> that
>>>>>>> introduced “num_runs = -1” was there to be able to work with
>> empty
>>>> dag
>>>>>>> dirs, maybe it wasn’t fully covered yet.
>>>>>>> 
>>>>>>> Bolke
>>>>>>> 
>>>>>>>> On 12 Feb 2017, at 12:26, Bolke de Bruin <bdbruin@gmail.com>
>>>> wrote:
>>>>>>>> 
>>>>>>>> Ok great! Thanks! That sounds like a race condition: module
not
>>>>>>> available yet at time of reading. I would expect that it resolves
>>>>> itself
>>>>>>> after a while.
>>>>>>>> 
>>>>>>>> After talking to some people at the Warsaw BigData conf I
have
>>> some
>>>>>>> ideas around syncing dags, Spoiler: no dependency on git.
>>>>>>>> 
>>>>>>>> - Bolke
>>>>>>>> 
>>>>>>>>> On 12 Feb 2017, at 11:17, Alex Van Boxel <alex@vanboxel.be>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Running ok, in staging... @bolke I'm running patch-less.
I've
>>>>> switched
>>>>>>> my
>>>>>>>>> Kubernetes from:
>>>>>>>>> 
>>>>>>>>> - each container (webserver/scheduler/worker) had a
>> git-sync'er
>>>>>> (getting
>>>>>>>>> the dags from git)
>>>>>>>>>> this meant that the scheduler had 0 dags at startup,
and
>> should
>>>>> have
>>>>>>>>> picked them up later
>>>>>>>>> 
>>>>>>>>> to
>>>>>>>>> 
>>>>>>>>> - single NFS share that shares airflow_home over each
>> container
>>>>>>>>>> the git sync'er is now a seperate container running
before
>> the
>>>>> other
>>>>>>>>> containers
>>>>>>>>> 
>>>>>>>>> This resolved my mystery DAG crashes.
>>>>>>>>> 
>>>>>>>>> I'll be updating production to a patchless RC3 today,
you get
>> my
>>>>> vote
>>>>>>> after
>>>>>>>>> that.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Sun, Feb 12, 2017 at 4:59 AM Boris Tyukin <
>>>> boris@boristyukin.com
>>>>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> awesome! thanks Jeremiah
>>>>>>>>>> 
>>>>>>>>>> On Sat, Feb 11, 2017 at 12:53 PM, Jeremiah Lowin
<
>>>>> jlowin@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Boris, I submitted a PR to address your second
point --
>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2068.
>>> Thanks!
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, Feb 11, 2017 at 10:42 AM Boris Tyukin
<
>>>>>> boris@boristyukin.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I am running LocalExecutor and not doing
crazy things but
>> use
>>>> DAG
>>>>>>>>>>>> generation heavily - everything runs fine
as before. As I
>>>>> mentioned
>>>>>>> in
>>>>>>>>>>>> other threads only had a few issues:
>>>>>>>>>>>> 
>>>>>>>>>>>> 1) had to upgrade MySQL which was a PAIN.
Cloudera CDH is
>>>> running
>>>>>> old
>>>>>>>>>>>> version of MySQL which was compatible with
1.7.1 but not
>>>>> compatible
>>>>>>> now
>>>>>>>>>>>> with 1.8 because of fractional seconds support
PR.
>>>>>>>>>>>> 
>>>>>>>>>>>> 2) when you install airflow, there are two
new example DAGs
>>>>>>>>>>>> (last_task_only) which are going back very
far in the past
>>> and
>>>>>>>>>> scheduled
>>>>>>>>>>> to
>>>>>>>>>>>> run every hour - a bunch of dags triggered
on the first
>> start
>>>> of
>>>>>>>>>>> scheduler
>>>>>>>>>>>> and hosed my CPU
>>>>>>>>>>>> 
>>>>>>>>>>>> Everything else was fine and I LOVE lots
of small UI
>> changes,
>>>>> which
>>>>>>>>>>> reduced
>>>>>>>>>>>> a lot my use of cli.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks again for the amazing work and an
awesome project!
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Sat, Feb 11, 2017 at 9:17 AM, Jeremiah
Lowin <
>>>>> jlowin@apache.org
>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I was able to deploy successfully. +1
(binding)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Feb 10, 2017 at 7:37 PM Maxime
Beauchemin <
>>>>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> +1 (binding)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Feb 10, 2017 at 3:44 PM,
Arthur Wiedmer <
>>>>>>>>>>>>> arthur.wiedmer@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> +1 (binding)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Feb 10, 2017 3:13 PM, "Dan
Davydov" <
>>>>> dan.davydov@airbnb.com.
>>>>>>>>>>>>> invalid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Our staging looks good, all
the DAGs there pass.
>>>>>>>>>>>>>>>> +1 (binding)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Feb 10, 2017 at 10:21
AM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Running in all environments.
Will vote after the
>> weekend
>>>> to
>>>>>>>>>>> make
>>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>>> things are working properly,
but so far so good.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Feb 10, 2017
at 6:05 AM, Bolke de Bruin <
>>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Dear All,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Let’s try again!
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have made the THIRD
RELEASE CANDIDATE of Airflow
>>> 1.8.0
>>>>>>>>>>>>> available
>>>>>>>>>>>>>>> at:
>>>>>>>>>>>>>>>>>> 
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>> 
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
>>>>>>>>>> ,
>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> keys
>>>>>>>>>>>>>>>>>> are available at
https://dist.apache.org/repos/
>>>>>>>>>>>>>>> dist/release/incubator/
>>>>>>>>>>>>>>>>>> airflow/ <
>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>>>>>>>>>> airflow/>
>>>>>>>>>>>>>>>>>> . It is tagged with
a local version
>> “apache.incubating”
>>>> so
>>>>>>>>>> it
>>>>>>>>>>>>>> allows
>>>>>>>>>>>>>>>>>> upgrading from earlier
releases.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Two issues have been
fixed since release candidate 2:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> * trigger_dag could
create dags with fractional
>>> seconds,
>>>>>>>>>> not
>>>>>>>>>>>>>>> supported
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> logging and UI at
the moment
>>>>>>>>>>>>>>>>>> * local api client
trigger_dag had hardcoded
>> execution
>>> of
>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Known issue:
>>>>>>>>>>>>>>>>>> * Airflow on kubernetes
and num_runs -1 (default) can
>>>>>>>>>> expose
>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>> issues.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have extensively
discussed this with Alex
>> (reporter)
>>>> and
>>>>>>>>>> we
>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>> this a known issue
with a workaround available as we
>>> are
>>>>>>>>>>> unable
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> replicate this in
a different environment.
>> UPDATING.md
>>>> has
>>>>>>>>>>> been
>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>> with the work around.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> As these issues are
confined to a very specific area
>>> and
>>>>>>>>>> full
>>>>>>>>>>>>> unit
>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>> were added I would
also like to raise a VOTE for
>>>> releasing
>>>>>>>>>>>> 1.8.0
>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> release candidate
3, i.e. just renaming release
>>>> candidate 3
>>>>>>>>>>> to
>>>>>>>>>>>>>> 1.8.0
>>>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Please respond to
this email by:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> +1,0,-1 with *binding*
if you are a PMC member or
>>>>>>>>>>> *non-binding*
>>>>>>>>>>>>> if
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> not.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> My VOTE: +1 (binding)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> _/
>>>>>>>>> _/ Alex Van Boxel
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> --
>>>  _/
>>> _/ Alex Van Boxel
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message