airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <a...@vanboxel.be>
Subject Re: Airflow 1.8.0 BETA 5
Date Mon, 30 Jan 2017 23:09:45 GMT
Well this means none of my DAG's work anymore:

you just can do this anymore:

file bqschema.py with

def marketing_segment():
    return [
        {"name": "user_id", "type": "integer", "mode": "nullable"},
        {"name": "bucket_date", "type": "timestamp", "mode": "nullable"},
        {"name": "segment_main", "type": "string", "mode": "nullable"},
        {"name": "segment_sub", "type": "integer", "mode": "nullable"},


In marketing_segmentation.py:


import bqschema

Gives an error:

Traceback (most recent call last):
  File
"/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+apache.incubating-py2.7.egg/airflow/models.py",
line 264, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/home/airflow/dags/marketing_segmentation.py", line 17, in <module>
    import bqschema
ImportError: No module named bqschema

*I don't think this is incorrect?!*



On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.invalid>
wrote:

> The latest commit fixed a regression since 1.7 that files with parsing
> errors no longer showed up on the UI.
>
> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <alex@vanboxel.be> wrote:
>
> > Just installed beta 5 on our dev environment it lighted up as a christmas
> > tree. I got a a screen full of import errors. I see that the latest
> commit
> > did something with import errors... is it coorect?!
> >
> > On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> > > Hey Boris
> > >
> > > The scheduler is a bit more aggressive and can use multiple processors,
> > so
> > > higher CPU usage is actually a good thing.
> > >
> > > I case it is really out of hand look at the new scheduler options and
> > > heartbeat options (see PR for updating.md not in the beta yet).
> > >
> > > Bolke
> > >
> > > Sent from my iPhone
> > >
> > > > On 29 Jan 2017, at 15:35, Boris Tyukin <boris@boristyukin.com>
> wrote:
> > > >
> > > > I am not sure if it is my config or something, but looks like after
> the
> > > > upgrade and start of scheduler, airflow would totally hose CPU. The
> > > reason
> > > > is two new examples that start running right away - latest only and
> > > latest
> > > > with trigger. Once I pause them, CPU goes back to idle. Is this
> because
> > > now
> > > > dags are not paused by default like it was before?
> > > >
> > > > As I mentioned before, I also had to upgrade mysql to 5.7 - if
> someone
> > > > needs a step by step instruction, make sure to follow all steps
> > precisely
> > > > here for in-place upgrade or you will have heck of the time (like
> me).
> > > >
> > > https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> > upgrade-procedure-inplace
> > > >
> > > > BTW official Oracle repository for Oracle Linux only has MySql 5.6 -
> > for
> > > > 5.7 you have to use MySql community repo.
> > > >
> > > >> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <bdbruin@gmail.com
> >
> > > wrote:
> > > >>
> > > >> Hi All,
> > > >>
> > > >> I have made the FIFTH beta of Airflow 1.8.0 available at:
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > >> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> > > keys
> > > >> are available at https://dist.apache.org/repos/
> > dist/release/incubator/
> > > >> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> > airflow/
> > > >
> > > >> . It is tagged with a local version “apache.incubating” so it
allows
> > > >> upgrading from earlier releases.
> > > >>
> > > >> Issues fixed:
> > > >> * Parsing errors not showing up in UI fixing a regression**
> > > >> * Scheduler would terminate immediately if no dag files present
> > > >>
> > > >> ** As this touches the scheduler logic I though it warranted another
> > > beta.
> > > >>
> > > >> This should be the last beta in my opinion and we can prepare
> > changelog,
> > > >> upgrade notes and release notes for the RC (Feb 2).
> > > >>
> > > >> Cheers
> > > >> Bolke
> > >
> > --
> >   _/
> > _/ Alex Van Boxel
> >
>
-- 
  _/
_/ Alex Van Boxel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message