Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
References: <9AEEDD5F-7FC7-4E15-8FE8-556165019D3B@gmail.com>
 <CANRT7T14wQqj8kr4rvzKUzOuA07eYPCn_GkPFQLAXuqx8aaE+A@mail.gmail.com>
 <3246BFBD-BBC3-475B-9BD3-6D23820B3973@gmail.com> <CALCMntsA1vvSFhkyoQUPishOQfoBYJTbPcewcWSF1J4Wwzposw@mail.gmail.com>
 <CABekQG5M1RYqGUH8jcWe0h_8P=Qfg=zWUA5Arjt262HKaE0ryg@mail.gmail.com>
 <CALCMntt3H9p5BaT4gHbKOuG59W8+3NQ9=3aRopEZhiMP1T27LA@mail.gmail.com>
 <CABekQG5zYSdXnE6HVg1gLqAtQLnctEcNVPuP=AypbSpKrVz71g@mail.gmail.com>
 <CALCMntt_-6_nuVrV-1Cz=fZ3TAcCVNw566KV+Ux=0es8wMQELg@mail.gmail.com> <41EAC266-BC94-407B-A97D-9062B8832230@gmail.com>
In-Reply-To: <41EAC266-BC94-407B-A97D-9062B8832230@gmail.com>
From: Alex Van Boxel <alex@vanboxel.be>
Date: Tue, 31 Jan 2017 16:25:06 +0000
Message-ID: <CALCMntvT7KFooo9LgxeLo2Tbyj1==__DDKFGNqcpu9BucKoeEg@mail.gmail.com>
Subject: Re: Airflow 1.8.0 BETA 5
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary=f40304361e2cae6cb705476660f9
archived-at: Tue, 31 Jan 2017 16:25:26 -0000

--f40304361e2cae6cb705476660f9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I'll try to identify the core problem

On Tue, Jan 31, 2017, 16:43 Bolke de Bruin <bdbruin@gmail.com> wrote:

> Hey Alex
>
> Can you provide some info on the scheduler paths thing. I don't have/see
> that issue. Do you mean cli paths or by cfg? Jira would be nice in any ca=
se.
>
> I don't think the dag processor respects cli parameters.
>
> Bolke
>
> Sent from my iPhone
>
> > On 31 Jan 2017, at 15:10, Alex Van Boxel <alex@vanboxel.be> wrote:
> >
> > It's quite hard to share my complete dags. I don't have this locally,
> but I
> > have it in my production environment where I use Celery. I rolled back =
to
> > beta 4 to make it work again.
> >
> > Also @bolke the scheduler logs don't respect the log path.
> >
> > On Tue, Jan 31, 2017 at 1:02 AM Dan Davydov <dan.davydov@airbnb.com
> .invalid>
> > wrote:
> >
> >> @Alex
> >> I'm not able to reproduce locally (assuming the two python files are i=
n
> the
> >> same folder or is on your PYTHONPATH). I don't see that import error
> >> anyways.
> >>
> >> Just in case, what is your complete DAG definition? Is anyone else abl=
e
> to
> >> repro?
> >>
> >>> On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel <alex@vanboxel.be>
> wrote:
> >>>
> >>> Well this means none of my DAG's work anymore:
> >>>
> >>> you just can do this anymore:
> >>>
> >>> file bqschema.py with
> >>>
> >>> def marketing_segment():
> >>>    return [
> >>>        {"name": "user_id", "type": "integer", "mode": "nullable"},
> >>>        {"name": "bucket_date", "type": "timestamp", "mode":
> "nullable"},
> >>>        {"name": "segment_main", "type": "string", "mode": "nullable"}=
,
> >>>        {"name": "segment_sub", "type": "integer", "mode": "nullable"}=
,
> >>>
> >>>
> >>> In marketing_segmentation.py:
> >>>
> >>>
> >>> import bqschema
> >>>
> >>> Gives an error:
> >>>
> >>> Traceback (most recent call last):
> >>>  File
> >>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+
> >>> apache.incubating-py2.7.egg/airflow/models.py",
> >>> line 264, in process_file
> >>>    m =3D imp.load_source(mod_name, filepath)
> >>>  File "/home/airflow/dags/marketing_segmentation.py", line 17, in
> >>> <module>
> >>>    import bqschema
> >>> ImportError: No module named bqschema
> >>>
> >>> *I don't think this is incorrect?!*
> >>>
> >>>
> >>>
> >>> On Mon, Jan 30, 2017 at 11:46 PM Dan Davydov <dan.davydov@airbnb.com.
> >>> invalid>
> >>> wrote:
> >>>
> >>>> The latest commit fixed a regression since 1.7 that files with parsi=
ng
> >>>> errors no longer showed up on the UI.
> >>>>
> >>>> On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel <alex@vanboxel.be>
> >>> wrote:
> >>>>
> >>>>> Just installed beta 5 on our dev environment it lighted up as a
> >>> christmas
> >>>>> tree. I got a a screen full of import errors. I see that the latest
> >>>> commit
> >>>>> did something with import errors... is it coorect?!
> >>>>>
> >>>>> On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin <bdbruin@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Hey Boris
> >>>>>>
> >>>>>> The scheduler is a bit more aggressive and can use multiple
> >>> processors,
> >>>>> so
> >>>>>> higher CPU usage is actually a good thing.
> >>>>>>
> >>>>>> I case it is really out of hand look at the new scheduler options
> >> and
> >>>>>> heartbeat options (see PR for updating.md not in the beta yet).
> >>>>>>
> >>>>>> Bolke
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>>
> >>>>>>> On 29 Jan 2017, at 15:35, Boris Tyukin <boris@boristyukin.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>> I am not sure if it is my config or something, but looks like
> >> after
> >>>> the
> >>>>>>> upgrade and start of scheduler, airflow would totally hose CPU.
> >> The
> >>>>>> reason
> >>>>>>> is two new examples that start running right away - latest only
> >> and
> >>>>>> latest
> >>>>>>> with trigger. Once I pause them, CPU goes back to idle. Is this
> >>>> because
> >>>>>> now
> >>>>>>> dags are not paused by default like it was before?
> >>>>>>>
> >>>>>>> As I mentioned before, I also had to upgrade mysql to 5.7 - if
> >>>> someone
> >>>>>>> needs a step by step instruction, make sure to follow all steps
> >>>>> precisely
> >>>>>>> here for in-place upgrade or you will have heck of the time (like
> >>>> me).
> >>>>>>>
> >>>>>> https://dev.mysql.com/doc/refman/5.7/en/upgrading.html#
> >>>>> upgrade-procedure-inplace
> >>>>>>>
> >>>>>>> BTW official Oracle repository for Oracle Linux only has MySql
> >> 5.6
> >>> -
> >>>>> for
> >>>>>>> 5.7 you have to use MySql community repo.
> >>>>>>>
> >>>>>>>> On Sat, Jan 28, 2017 at 10:07 AM, Bolke de Bruin <
> >>> bdbruin@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> I have made the FIFTH beta of Airflow 1.8.0 available at:
> >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> >>> public
> >>>>>> keys
> >>>>>>>> are available at https://dist.apache.org/repos/
> >>>>> dist/release/incubator/
> >>>>>>>> airflow/ <https://dist.apache.org/repos/dist/release/incubator/
> >>>>> airflow/
> >>>>>>>
> >>>>>>>> . It is tagged with a local version =E2=80=9Capache.incubating=
=E2=80=9D so it
> >>> allows
> >>>>>>>> upgrading from earlier releases.
> >>>>>>>>
> >>>>>>>> Issues fixed:
> >>>>>>>> * Parsing errors not showing up in UI fixing a regression**
> >>>>>>>> * Scheduler would terminate immediately if no dag files present
> >>>>>>>>
> >>>>>>>> ** As this touches the scheduler logic I though it warranted
> >>> another
> >>>>>> beta.
> >>>>>>>>
> >>>>>>>> This should be the last beta in my opinion and we can prepare
> >>>>> changelog,
> >>>>>>>> upgrade notes and release notes for the RC (Feb 2).
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Bolke
> >>>>>>
> >>>>> --
> >>>>>  _/
> >>>>> _/ Alex Van Boxel
> >>>>>
> >>>>
> >>> --
> >>>  _/
> >>> _/ Alex Van Boxel
> >>>
> >>
> > --
> >  _/
> > _/ Alex Van Boxel
>
--=20
  _/
_/ Alex Van Boxel

--f40304361e2cae6cb705476660f9--