airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.8.0 BETA 5
Date Tue, 31 Jan 2017 22:48:22 GMT
Rc 1 isn't out yet, differences will be small. 

Sent from my iPhone

> On 31 Jan 2017, at 22:00, Chris Riccomini <criccomini@apache.org> wrote:
> 
> I'm still on b5. Should I upgrade to rc1, or is it still in progress?
> 
>> On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel <alex@vanboxel.be> wrote:
>> 
>> I identified my root cause: it's was a problem at my site with a faulty
>> rebase. So all green.
>> 
>>> On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel <alex@vanboxel.be> wrote:
>>> 
>>> So bumped to RC1 and this seems fine. I don't get it.
>>> 
>>> So it's a go. Sorry for the noise, but better safe then sorry. I also
>>> found that the scheduler logs setting are not equal from the normal
>> logging
>>> so that's also a go.
>>> 
>>> You get green light from me.
>>> 
>>> On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bdbruin@gmail.com>
>> wrote:
>>> 
>>> And the scheduler doesn't log anything in beta 4?
>>> 
>>> Sent from my iPhone
>>> 
>>>> On 31 Jan 2017, at 19:26, Alex Van Boxel <alex@vanboxel.be> wrote:
>>>> 
>>>> I see it in the scheduler and in the UI. Currently trying to do a new
>>>> upgrade.
>>>> 
>>>>> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bdbruin@gmail.com>
>>> wrote:
>>>>> 
>>>>> Please note I will be holding off on the RC, before we understand this
>>>>> issue better.
>>>>> 
>>>>> Bolke
>>>>> 
>>>>>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bdbruin@gmail.com>
wrote:
>>>>>> 
>>>>>> Hey Alex,
>>>>>> 
>>>>>> Could it actually be that Airflow is doing the right thing? Earlier
>> it
>>>>> was swallowing the errors. Where do you see the errors? In the
>> scheduler
>>>>> logs, UI, processor logs?
>>>>>> 
>>>>>> - Bolke
>>>>>> 
>>>>>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <alex@vanboxel.be>
wrote:
>>>>>>> 
>>>>>>> I'll try to identify the core problem
>>>>>>> 
>>>>>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbruin@gmail.com>
>>> wrote:
>>>>>>>> 
>>>>>>>> Hey Alex
>>>>>>>> 
>>>>>>>> Can you provide some info on the scheduler paths thing. I
don't
>>>>> have/see
>>>>>>>> that issue. Do you mean cli paths or by cfg? Jira would be
nice in
>>> any
>>>>> case.
>>>>>>>> 
>>>>>>>> I don't think the dag processor respects cli parameters.
>>>>>>>> 
>>>>>>>> Bolke
>>>>>>>> 
>>>>>>>> Sent from my iPhone
>>>>>>>> 
>>>>>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <alex@vanboxel.be>
>> wrote:
>>>>>>>>> 
>>>>>>>>> It's quite hard to share my complete dags. I don't have
this
>>> locally,
>>>>>>>> but I
>>>>>>>>> have it in my production environment where I use Celery.
I rolled
>>>>> back to
>>>>>>>>> beta 4 to make it work again.
>>>>>>>>> 
>>>>>>>>> Also @bolke the scheduler logs don't respect the log
path.
>>>>>>>>> 
>>>>>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <
>> dan.davydov@airbnb.com
>>>>>>>> .invalid>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> @Alex
>>>>>>>>>> I'm not able to reproduce locally (assuming the two
python files
>>> are
>>>>> in
>>>>>>>> the
>>>>>>>>>> same folder or is on your PYTHONPATH). I don't see
that import
>>> error
>>>>>>>>>> anyways.
>>>>>>>>>> 
>>>>>>>>>> Just in case, what is your complete DAG definition?
Is anyone
>> else
>>>>> able
>>>>>>>> to
>>>>>>>>>> repro?
>>>>>>>>>> 
>>>>>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel
<
>> alex@vanboxel.be
>>>> 
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Well this means none of my DAG's work anymore:
>>>>>>>>>>> 
>>>>>>>>>>> you just can do this anymore:
>>>>>>>>>>> 
>>>>>>>>>>> file bqschema.py with
>>>>>>>>>>> 
>>>>>>>>>>> def marketing_segment():
>>>>>>>>>>> return [
>>>>>>>>>>>    {"name": "user_id", "type": "integer", "mode":
"nullable"},
>>>>>>>>>>>    {"name": "bucket_date", "type": "timestamp",
"mode":
>>>>>>>> "nullable"},
>>>>>>>>>>>    {"name": "segment_main", "type": "string",
"mode":
>>> "nullable"},
>>>>>>>>>>>    {"name": "segment_sub", "type": "integer",
"mode":
>>> "nullable"},
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> In marketing_segmentation.py:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> import bqschema
>>>>>>>>>>> 
>>>>>>>>>>> Gives an error:
>>>>>>>>>>> 
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> File
>>>>>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>>>>>>>> line 264, in process_file
>>>>>>>>>>> m = imp.load_source(mod_name, filepath)
>>>>>>>>>>> File "/home/airflow/dags/marketing_segmentation.py",
line 17,
>> in
>>>>>>>>>>> <module>
>>>>>>>>>>> import bqschema
>>>>>>>>>>> ImportError: No module named bqschema
>>>>>>>>>>> 
>>>>>>>>>>> *I don't think this is incorrect?!*
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov
<
>>>>> dan.davydov@airbnb.com.
>>>>>>>>>>> invalid>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> The latest commit fixed a regression since
1.7 that files with
>>>>> parsing
>>>>>>>>>>>> errors no longer showed up on the UI.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van
Boxel <
>>> alex@vanboxel.be>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Just installed beta 5 on our dev environment
it lighted up as
>> a
>>>>>>>>>>> christmas
>>>>>>>>>>>>> tree. I got a a screen full of import
errors. I see that the
>>>>> latest
>>>>>>>>>>>> commit
>>>>>>>>>>>>> did something with import errors... is
it coorect?!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke
de Bruin <
>>> bdbruin@gmail.com
>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hey Boris
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The scheduler is a bit more aggressive
and can use multiple
>>>>>>>>>>> processors,
>>>>>>>>>>>>> so
>>>>>>>>>>>>>> higher CPU usage is actually a good
thing.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I case it is really out of hand look
at the new scheduler
>>> options
>>>>>>>>>> and
>>>>>>>>>>>>>> heartbeat options (see PR for updating.md
not in the beta
>>> yet).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris
Tyukin <
>> boris@boristyukin.com
>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I am not sure if it is my config
or something, but looks
>> like
>>>>>>>>>> after
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> upgrade and start of scheduler,
airflow would totally hose
>>> CPU.
>>>>>>>>>> The
>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>> is two new examples that start
running right away - latest
>>> only
>>>>>>>>>> and
>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>> with trigger. Once I pause them,
CPU goes back to idle. Is
>>> this
>>>>>>>>>>>> because
>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>> dags are not paused by default
like it was before?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> As I mentioned before, I also
had to upgrade mysql to 5.7 -
>> if
>>>>>>>>>>>> someone
>>>>>>>>>>>>>>> needs a step by step instruction,
make sure to follow all
>>> steps
>>>>>>>>>>>>> precisely
>>>>>>>>>>>>>>> here for in-place upgrade or
you will have heck of the time
>>>>> (like
>>>>>>>>>>>> me).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>>>>>>>> upgrade-procedure-inplace
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> BTW official Oracle repository
for Oracle Linux only has
>> MySql
>>>>>>>>>> 5.6
>>>>>>>>>>> -
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> 5.7 you have to use MySql community
repo.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07
AM, Bolke de Bruin <
>>>>>>>>>>> bdbruin@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have made the FIFTH beta
of Airflow 1.8.0 available at:
>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/
>> <
>>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
>> ,
>>>>>>>>>>> public
>>>>>>>>>>>>>> keys
>>>>>>>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>>>>>>> dist/release/incubator/
>>>>>>>>>>>>>>>> airflow/ <
>>>>> https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>>>>>>> airflow/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> . It is tagged with a local
version “apache.incubating” so
>> it
>>>>>>>>>>> allows
>>>>>>>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Issues fixed:
>>>>>>>>>>>>>>>> * Parsing errors not showing
up in UI fixing a regression**
>>>>>>>>>>>>>>>> * Scheduler would terminate
immediately if no dag files
>>> present
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ** As this touches the scheduler
logic I though it
>> warranted
>>>>>>>>>>> another
>>>>>>>>>>>>>> beta.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This should be the last beta
in my opinion and we can
>> prepare
>>>>>>>>>>>>> changelog,
>>>>>>>>>>>>>>>> upgrade notes and release
notes for the RC (Feb 2).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> _/
>>>>>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> _/
>>>>>>>>>>> _/ Alex Van Boxel
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> _/
>>>>>>>>> _/ Alex Van Boxel
>>>>>>>> 
>>>>>>> --
>>>>>>> _/
>>>>>>> _/ Alex Van Boxel
>>>>>> 
>>>>> 
>>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>> 
>>> --
>>>  _/
>>> _/ Alex Van Boxel
>>> 
>> --
>>  _/
>> _/ Alex Van Boxel
>> 

Mime
View raw message