airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <a...@vanboxel.be>
Subject Re: Airflow 1.8.0 BETA 5
Date Tue, 31 Jan 2017 19:52:20 GMT
So bumped to RC1 and this seems fine. I don't get it.

So it's a go. Sorry for the noise, but better safe then sorry. I also found
that the scheduler logs setting are not equal from the normal logging so
that's also a go.

You get green light from me.

On Tue, Jan 31, 2017 at 8:32 PM Bolke de Bruin <bdbruin@gmail.com> wrote:

> And the scheduler doesn't log anything in beta 4?
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 19:26, Alex Van Boxel <alex@vanboxel.be> wrote:
> >
> > I see it in the scheduler and in the UI. Currently trying to do a new
> > upgrade.
> >
> >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >>
> >> Please note I will be holding off on the RC, before we understand this
> >> issue better.
> >>
> >> Bolke
> >>
> >>> On 31 Jan 2017, at 18:06, Bolke de Bruin <bdbruin@gmail.com> wrote:
> >>>
> >>> Hey Alex,
> >>>
> >>> Could it actually be that Airflow is doing the right thing? Earlier it
> >> was swallowing the errors. Where do you see the errors? In the scheduler
> >> logs, UI, processor logs?
> >>>
> >>> - Bolke
> >>>
> >>>> On 31 Jan 2017, at 16:25, Alex Van Boxel <alex@vanboxel.be> wrote:
> >>>>
> >>>> I'll try to identify the core problem
> >>>>
> >>>>> On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >>>>>
> >>>>> Hey Alex
> >>>>>
> >>>>> Can you provide some info on the scheduler paths thing. I don't
> >> have/see
> >>>>> that issue. Do you mean cli paths or by cfg? Jira would be nice
in
> any
> >> case.
> >>>>>
> >>>>> I don't think the dag processor respects cli parameters.
> >>>>>
> >>>>> Bolke
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 31 Jan 2017, at 15:10, Alex Van Boxel <alex@vanboxel.be>
wrote:
> >>>>>>
> >>>>>> It's quite hard to share my complete dags. I don't have this
> locally,
> >>>>> but I
> >>>>>> have it in my production environment where I use Celery. I rolled
> >> back to
> >>>>>> beta 4 to make it work again.
> >>>>>>
> >>>>>> Also @bolke the scheduler logs don't respect the log path.
> >>>>>>
> >>>>>> On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> >>>>> .invalid>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Alex
> >>>>>>> I'm not able to reproduce locally (assuming the two python
files
> are
> >> in
> >>>>> the
> >>>>>>> same folder or is on your PYTHONPATH). I don't see that
import
> error
> >>>>>>> anyways.
> >>>>>>>
> >>>>>>> Just in case, what is your complete DAG definition? Is anyone
else
> >> able
> >>>>> to
> >>>>>>> repro?
> >>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <alex@vanboxel.be
> >
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Well this means none of my DAG's work anymore:
> >>>>>>>>
> >>>>>>>> you just can do this anymore:
> >>>>>>>>
> >>>>>>>> file bqschema.py with
> >>>>>>>>
> >>>>>>>> def marketing_segment():
> >>>>>>>> return [
> >>>>>>>>     {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>>>>>>     {"name": "bucket_date", "type": "timestamp", "mode":
> >>>>> "nullable"},
> >>>>>>>>     {"name": "segment_main", "type": "string", "mode":
> "nullable"},
> >>>>>>>>     {"name": "segment_sub", "type": "integer", "mode":
> "nullable"},
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> In marketing_segmentation.py:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> import bqschema
> >>>>>>>>
> >>>>>>>> Gives an error:
> >>>>>>>>
> >>>>>>>> Traceback (most recent call last):
> >>>>>>>> File
> >>>>>>>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>>>>>>> apache.incubating-py2.7.egg/airflow/models.py",
> >>>>>>>> line 264, in process_file
> >>>>>>>> m = imp.load_source(mod_name, filepath)
> >>>>>>>> File "/home/airflow/dags/marketing_segmentation.py",
line 17, in
> >>>>>>>> <module>
> >>>>>>>> import bqschema
> >>>>>>>> ImportError: No module named bqschema
> >>>>>>>>
> >>>>>>>> *I don't think this is incorrect?!*
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <
> >> dan.davydov@airbnb.com.
> >>>>>>>> invalid>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> The latest commit fixed a regression since 1.7 that
files with
> >> parsing
> >>>>>>>>> errors no longer showed up on the UI.
> >>>>>>>>>
> >>>>>>>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel
<
> alex@vanboxel.be>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Just installed beta 5 on our dev environment
it lighted up as a
> >>>>>>>> christmas
> >>>>>>>>>> tree. I got a a screen full of import errors.
I see that the
> >> latest
> >>>>>>>>> commit
> >>>>>>>>>> did something with import errors... is it coorect?!
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin
<
> bdbruin@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey Boris
> >>>>>>>>>>>
> >>>>>>>>>>> The scheduler is a bit more aggressive and
can use multiple
> >>>>>>>> processors,
> >>>>>>>>>> so
> >>>>>>>>>>> higher CPU usage is actually a good thing.
> >>>>>>>>>>>
> >>>>>>>>>>> I case it is really out of hand look at
the new scheduler
> options
> >>>>>>> and
> >>>>>>>>>>> heartbeat options (see PR for updating.md
not in the beta
> yet).
> >>>>>>>>>>>
> >>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>
> >>>>>>>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin
<boris@boristyukin.com
> >
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am not sure if it is my config or
something, but looks like
> >>>>>>> after
> >>>>>>>>> the
> >>>>>>>>>>>> upgrade and start of scheduler, airflow
would totally hose
> CPU.
> >>>>>>> The
> >>>>>>>>>>> reason
> >>>>>>>>>>>> is two new examples that start running
right away - latest
> only
> >>>>>>> and
> >>>>>>>>>>> latest
> >>>>>>>>>>>> with trigger. Once I pause them, CPU
goes back to idle. Is
> this
> >>>>>>>>> because
> >>>>>>>>>>> now
> >>>>>>>>>>>> dags are not paused by default like
it was before?
> >>>>>>>>>>>>
> >>>>>>>>>>>> As I mentioned before, I also had to
upgrade mysql to 5.7 - if
> >>>>>>>>> someone
> >>>>>>>>>>>> needs a step by step instruction, make
sure to follow all
> steps
> >>>>>>>>>> precisely
> >>>>>>>>>>>> here for in-place upgrade or you will
have heck of the time
> >> (like
> >>>>>>>>> me).
> >>>>>>>>>>>>
> >>>>>>>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> >>>>>>>>>> upgrade-procedure-inplace
> >>>>>>>>>>>>
> >>>>>>>>>>>> BTW official Oracle repository for Oracle
Linux only has MySql
> >>>>>>> 5.6
> >>>>>>>> -
> >>>>>>>>>> for
> >>>>>>>>>>>> 5.7 you have to use MySql community
repo.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM,
Bolke de Bruin <
> >>>>>>>> bdbruin@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi All,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I have made the FIFTH beta of Airflow
1.8.0 available at:
> >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/
<
> >>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
,
> >>>>>>>> public
> >>>>>>>>>>> keys
> >>>>>>>>>>>>> are available at https://dist.apache.org/repos/
> >>>>>>>>>> dist/release/incubator/
> >>>>>>>>>>>>> airflow/ <
> >> https://dist.apache.org/repos/dist/release/incubator/
> >>>>>>>>>> airflow/
> >>>>>>>>>>>>
> >>>>>>>>>>>>> . It is tagged with a local version
“apache.incubating” so it
> >>>>>>>> allows
> >>>>>>>>>>>>> upgrading from earlier releases.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Issues fixed:
> >>>>>>>>>>>>> * Parsing errors not showing up
in UI fixing a regression**
> >>>>>>>>>>>>> * Scheduler would terminate immediately
if no dag files
> present
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ** As this touches the scheduler
logic I though it warranted
> >>>>>>>> another
> >>>>>>>>>>> beta.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This should be the last beta in
my opinion and we can prepare
> >>>>>>>>>> changelog,
> >>>>>>>>>>>>> upgrade notes and release notes
for the RC (Feb 2).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>> Bolke
> >>>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> _/
> >>>>>>>>>> _/ Alex Van Boxel
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> _/
> >>>>>>>> _/ Alex Van Boxel
> >>>>>>>>
> >>>>>>>
> >>>>>> --
> >>>>>> _/
> >>>>>> _/ Alex Van Boxel
> >>>>>
> >>>> --
> >>>> _/
> >>>> _/ Alex Van Boxel
> >>>
> >>
> >> --
> >  _/
> > _/ Alex Van Boxel
>
-- 
  _/
_/ Alex Van Boxel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message