airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.8.0 BETA 5
Date Tue, 31 Jan 2017 19:00:36 GMT
Please note I will be holding off on the RC, before we understand this issue better.

Bolke

> On 31 Jan 2017, at 18:06, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
> Hey Alex,
> 
> Could it actually be that Airflow is doing the right thing? Earlier it was swallowing
the errors. Where do you see the errors? In the scheduler logs, UI, processor logs?
> 
> - Bolke
> 
>> On 31 Jan 2017, at 16:25, Alex Van Boxel <alex@vanboxel.be> wrote:
>> 
>> I'll try to identify the core problem
>> 
>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbruin@gmail.com> wrote:
>> 
>>> Hey Alex
>>> 
>>> Can you provide some info on the scheduler paths thing. I don't have/see
>>> that issue. Do you mean cli paths or by cfg? Jira would be nice in any case.
>>> 
>>> I don't think the dag processor respects cli parameters.
>>> 
>>> Bolke
>>> 
>>> Sent from my iPhone
>>> 
>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <alex@vanboxel.be> wrote:
>>>> 
>>>> It's quite hard to share my complete dags. I don't have this locally,
>>> but I
>>>> have it in my production environment where I use Celery. I rolled back to
>>>> beta 4 to make it work again.
>>>> 
>>>> Also @bolke the scheduler logs don't respect the log path.
>>>> 
>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
>>> .invalid>
>>>> wrote:
>>>> 
>>>>> @Alex
>>>>> I'm not able to reproduce locally (assuming the two python files are
in
>>> the
>>>>> same folder or is on your PYTHONPATH). I don't see that import error
>>>>> anyways.
>>>>> 
>>>>> Just in case, what is your complete DAG definition? Is anyone else able
>>> to
>>>>> repro?
>>>>> 
>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <alex@vanboxel.be>
>>> wrote:
>>>>>> 
>>>>>> Well this means none of my DAG's work anymore:
>>>>>> 
>>>>>> you just can do this anymore:
>>>>>> 
>>>>>> file bqschema.py with
>>>>>> 
>>>>>> def marketing_segment():
>>>>>>  return [
>>>>>>      {"name": "user_id", "type": "integer", "mode": "nullable"},
>>>>>>      {"name": "bucket_date", "type": "timestamp", "mode":
>>> "nullable"},
>>>>>>      {"name": "segment_main", "type": "string", "mode": "nullable"},
>>>>>>      {"name": "segment_sub", "type": "integer", "mode": "nullable"},
>>>>>> 
>>>>>> 
>>>>>> In marketing_segmentation.py:
>>>>>> 
>>>>>> 
>>>>>> import bqschema
>>>>>> 
>>>>>> Gives an error:
>>>>>> 
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
>>>>>> line 264, in process_file
>>>>>>  m = imp.load_source(mod_name, filepath)
>>>>>> File "/home/airflow/dags/marketing_segmentation.py", line 17, in
>>>>>> <module>
>>>>>>  import bqschema
>>>>>> ImportError: No module named bqschema
>>>>>> 
>>>>>> *I don't think this is incorrect?!*
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
>>>>>> invalid>
>>>>>> wrote:
>>>>>> 
>>>>>>> The latest commit fixed a regression since 1.7 that files with
parsing
>>>>>>> errors no longer showed up on the UI.
>>>>>>> 
>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <alex@vanboxel.be>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> Just installed beta 5 on our dev environment it lighted up
as a
>>>>>> christmas
>>>>>>>> tree. I got a a screen full of import errors. I see that
the latest
>>>>>>> commit
>>>>>>>> did something with import errors... is it coorect?!
>>>>>>>> 
>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbruin@gmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Boris
>>>>>>>>> 
>>>>>>>>> The scheduler is a bit more aggressive and can use multiple
>>>>>> processors,
>>>>>>>> so
>>>>>>>>> higher CPU usage is actually a good thing.
>>>>>>>>> 
>>>>>>>>> I case it is really out of hand look at the new scheduler
options
>>>>> and
>>>>>>>>> heartbeat options (see PR for updating.md not in the
beta yet).
>>>>>>>>> 
>>>>>>>>> Bolke
>>>>>>>>> 
>>>>>>>>> Sent from my iPhone
>>>>>>>>> 
>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <boris@boristyukin.com>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I am not sure if it is my config or something, but
looks like
>>>>> after
>>>>>>> the
>>>>>>>>>> upgrade and start of scheduler, airflow would totally
hose CPU.
>>>>> The
>>>>>>>>> reason
>>>>>>>>>> is two new examples that start running right away
- latest only
>>>>> and
>>>>>>>>> latest
>>>>>>>>>> with trigger. Once I pause them, CPU goes back to
idle. Is this
>>>>>>> because
>>>>>>>>> now
>>>>>>>>>> dags are not paused by default like it was before?
>>>>>>>>>> 
>>>>>>>>>> As I mentioned before, I also had to upgrade mysql
to 5.7 - if
>>>>>>> someone
>>>>>>>>>> needs a step by step instruction, make sure to follow
all steps
>>>>>>>> precisely
>>>>>>>>>> here for in-place upgrade or you will have heck of
the time (like
>>>>>>> me).
>>>>>>>>>> 
>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
>>>>>>>> upgrade-procedure-inplace
>>>>>>>>>> 
>>>>>>>>>> BTW official Oracle repository for Oracle Linux only
has MySql
>>>>> 5.6
>>>>>> -
>>>>>>>> for
>>>>>>>>>> 5.7 you have to use MySql community repo.
>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin
<
>>>>>> bdbruin@gmail.com
>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi All,
>>>>>>>>>>> 
>>>>>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available
at:
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/
<
>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
,
>>>>>> public
>>>>>>>>> keys
>>>>>>>>>>> are available at https://dist.apache.org/repos/
>>>>>>>> dist/release/incubator/
>>>>>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
>>>>>>>> airflow/
>>>>>>>>>> 
>>>>>>>>>>> . It is tagged with a local version “apache.incubating”
so it
>>>>>> allows
>>>>>>>>>>> upgrading from earlier releases.
>>>>>>>>>>> 
>>>>>>>>>>> Issues fixed:
>>>>>>>>>>> * Parsing errors not showing up in UI fixing
a regression**
>>>>>>>>>>> * Scheduler would terminate immediately if no
dag files present
>>>>>>>>>>> 
>>>>>>>>>>> ** As this touches the scheduler logic I though
it warranted
>>>>>> another
>>>>>>>>> beta.
>>>>>>>>>>> 
>>>>>>>>>>> This should be the last beta in my opinion and
we can prepare
>>>>>>>> changelog,
>>>>>>>>>>> upgrade notes and release notes for the RC (Feb
2).
>>>>>>>>>>> 
>>>>>>>>>>> Cheers
>>>>>>>>>>> Bolke
>>>>>>>>> 
>>>>>>>> --
>>>>>>>> _/
>>>>>>>> _/ Alex Van Boxel
>>>>>>>> 
>>>>>>> 
>>>>>> --
>>>>>> _/
>>>>>> _/ Alex Van Boxel
>>>>>> 
>>>>> 
>>>> --
>>>> _/
>>>> _/ Alex Van Boxel
>>> 
>> -- 
>> _/
>> _/ Alex Van Boxel
> 


Mime
View raw message