Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Message-Id: <0BB6FA0E-94CC-4076-B66F-C0094F1758F4@gmail.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_78EBE853-9101-4089-B92C-893BBD6A8891"
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Re: Airflow 1.8.0 Release Candidate 1
Date: Wed, 8 Feb 2017 13:33:01 +0100
In-Reply-To: <F436C944-17A8-42E1-9D8C-4846620D4208@gmail.com>
Cc: dev@airflow.incubator.apache.org
To: Alex Van Boxel <alex@vanboxel.be>
References: <1BA62FD0-851A-43BB-965E-87B680E247B6@gmail.com>
 <CANRT7T1XVsB67=+dXN7qYAHSDPhxTbzz-K=w4_H5KHqFR3iCKA@mail.gmail.com>
 <CAHEEp7UfxgLTXAHs4PaUTq16GExTPv+2ehrKftva7cV1aFBXuA@mail.gmail.com>
 <CANRT7T0L=iXOqqVEY=Lfg+E4vN+oe-MK0E_aCPXyk_u6vENUGw@mail.gmail.com>
 <B78062D9-5666-41A4-8108-F20A72FC5E14@gmail.com>
 <CADsgxrEhxt_=yOwwrmPc0Ep7Ud96_qEgEC1KbQJYT3BZkj70og@mail.gmail.com>
 <CAEOTQt8S9PGw3qYH60ze3J+Oj=mNGSQK2ciuUypcmPLACFgTZA@mail.gmail.com>
 <CADsgxrGt9_f_AexmMQ+ixO0rszy+P7Wt6ZgHcA087xMmTnDBSw@mail.gmail.com>
 <CABYbY7fMpHZqoesPR7wAH0w9viU9jbfkt1BrMWeeE07yZe7kuQ@mail.gmail.com>
 <CABekQG741sMjDB+1cSo70mdov5Aw_M6PV4QEFyX6G_zZXBttew@mail.gmail.com>
 <CANLtMifU3ZyXhZ723p3qJDT_2o_JuZKpDbCsnzt4=gRzxdLvJg@mail.gmail.com>
 <CANLtMify=94JYzuKrwBC7Qv-2pJdqZOWUYWOL0URNRn5MAYMMg@mail.gmail.com>
 <CABekQG5kG8mRiNFnqS=GvcZYt+iCOGa0e2tc98sOckRgpY0PRw@mail.gmail.com>
 <CANLtMidv5W3h8axVd_bkFNbwHk1YaoBSv=_t3vsveZRGfMJVRA@mail.gmail.com>
 <CALCMntuYhgLfCh49K6k+OOdBvX4it38dxwiVKN2H8Kibf2=Lgg@mail.gmail.com>
 <2DE49F95-BDD1-44F2-AC6A-495B755828D2@gmail.com>
 <CACZXaYu5DavzBws+z4EXattMgoUute8B0uQD1KW_rxxJJGB8Ng@mail.gmail.com>
 <CACZXaYujTeKBdGJZw066N2SNNyMq+XBAwMNDgWrH8bDHd7Qipg@mail.gmail.com>
 <CALCMntsCUW9LMkjsqOMh=EzprzNHpEd8xEVgB27J+bm48R26EA@mail.gmail.com>
 <F436C944-17A8-42E1-9D8C-4846620D4208@gmail.com>
archived-at: Wed, 08 Feb 2017 12:33:44 -0000

--Apple-Mail=_78EBE853-9101-4089-B92C-893BBD6A8891
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Alex,

Do you have anything more to go on? I don=E2=80=99t mind reverting the =
patch, however it code part seems unrelated to what you described and =
the issue wasn=E2=80=99t reproducible. I would really like to see more =
logging and maybe a test in a clean environment plus debugging. =
Preferable I would like to make RC 2 available today and immediately =
raise a vote as the *current* changes are really small, are confined to =
contrib and have been tested by the people using it.

But I am holding off for now due to your concern.

Cheers
Bolke


> On 7 Feb 2017, at 20:56, Bolke de Bruin <bdbruin@gmail.com> wrote:
>=20
> How do you start the scheduler Alex? What are the command line =
parameters? What are the logs when it doesn=E2=80=99t work?
>=20
> Bolke
>=20
>=20
>=20
>> On 7 Feb 2017, at 18:52, Alex Van Boxel <alex@vanboxel.be =
<mailto:alex@vanboxel.be>> wrote:
>>=20
>> Hey Feng,
>>=20
>> The upgrades are all automated (including the workers/web/scheduler). =
And I tripple checked, I now am test running RC1 just with the your line =
reverted (and look ok)
>>=20
>> Could you do me a favour and add a test dag where you do a local =
import. Example:
>>=20
>> bqschema.py
>> def ranking():
>>     return [
>>         {"name": "bucket_date", "type": "timestamp", "mode": =
"nullable"},
>>         {"name": "rank", "type": "integer", "mode": "nullable"},
>>         {"name": "audience_preference", "type": "float", "mode": =
"nullable"},
>>         {"name": "audience_likelihood_share", "type": "float", =
"mode": "nullable"}
>>     ]
>>=20
>> dag.py
>> import bqschema
>> ...
>> all in the same dag folder. We use it to define out BigQuery schema's =
into a seperate file.
>>=20
>>=20
>> On Tue, Feb 7, 2017 at 6:37 PM Feng Lu <fenglu@google.com.invalid =
<mailto:fenglu@google.com.invalid>> wrote:
>> Hi Alex-
>>=20
>> Please see the attached screenshots of my local testing using =
celeryexecutor (on k8s as well).=20
>> All look good and the workflow is successfully completed.
>>=20
>> Curious did you also update the worker image?=20
>> Sorry for the confusion, happy to debug more if you could share with =
me your k8s setup.=20
>>=20
>> Feng
>>=20
>> On Tue, Feb 7, 2017 at 8:37 AM, Feng Lu <fenglu@google.com =
<mailto:fenglu@google.com>> wrote:
>> When num_runs is not explicitly specified, the default is set to -1 =
to match the expectation of SchedulerJob here:
>> <Screen Shot 2017-02-07 at 8.01.26 AM.png>
>> =E2=80=8B
>> Doing so also matches the type of num_runs ('int' in this case).
>> The scheduler will run non-stop as a result regardless whether dag =
files are present (since the num_runs default is now -1: unlimited).=20
>>=20
>> Based on what Alex described, the import error doesn't look like =
directly related to this change.=20
>> Maybe this one? =
https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb73=
0af3af45330fc51b9 =
<https://github.com/apache/incubator-airflow/commit/67cbb966410226c1489bb7=
30af3af45330fc51b9>
>>=20
>> I am still in the middle of running some quick test using celery =
executor, will update the thread once it's done.=20
>>=20
>>=20
>> On Tue, Feb 7, 2017 at 6:56 AM, Bolke de Bruin <bdbruin@gmail.com =
<mailto:bdbruin@gmail.com>> wrote:
>> Hey Alex,
>>=20
>> Thanks for tracking it down. Can you elaborate want went wrong with =
celery? The lines below do not particularly relate to Celery directly, =
so I wonder why we are not seeing it with LocalExecutor?
>>=20
>> Cheers
>> Bolke
>>=20
>> > On 7 Feb 2017, at 15:51, Alex Van Boxel <alex@vanboxel.be =
<mailto:alex@vanboxel.be>> wrote:
>> >
>> > I have to give the RC1 a *-1*. I spend hours, or better days to get =
the RC
>> > running with Celery on our test environment, till I finally found =
the
>> > commit that killed it:
>> >
>> > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb
>> > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest
>> > Closes #2032 from fenglu-g/master
>> >
>> > I was always looking at the wrong this, because the commit only =
changes a
>> > single default parameter from *None to -1*
>> >
>> > I do have the impression I'm the only one running with Celery. Are =
other
>> > people running with it?
>> >
>> > *I propose* *reverting the commit*. Feng, can you elaborate on this =
change?
>> >
>> > Change the default back no *None* in cli.py got it finally working:
>> >
>> > 'num_runs': Arg(
>> >    ("-n", "--num_runs"),
>> >    default=3DNone, type=3Dint,
>> >    help=3D"Set the number of runs to execute before exiting"),
>> >
>> > Thanks.
>> >
>> > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand <sanand@apache.org =
<mailto:sanand@apache.org>> wrote:
>> >
>> > I did get 1.8.0 installed and running at Agari.
>> >
>> > I did run into 2 problems.
>> > 1. Most of our DAGs broke due the way Operators are now imported.
>> > =
https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprec=
ated-features =
<https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#depre=
cated-features>
>> >
>> > According to the documentation, these deprecations would only cause =
an
>> > issue in 2.0. However, I needed to fix them now.
>> >
>> > So, I needed to change "from airflow.operators import =
PythonOperator" to
>> > from "from airflow.operators.python_operator import =
PythonOperator". Am I
>> > missing something?
>> >
>> > 2. I ran into a migration problem that seems to have cleared itself =
up. I
>> > did notice that some dags do not have data in their "DAG Runs" =
column on
>> > the overview page computed. I am looking into that issue presently.
>> > =
https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.=
07.png?dl=3D0 =
<https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45=
.07.png?dl=3D0>
>> >
>> > -s
>> >
>> > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov <dan.davydov@airbnb.com =
<mailto:dan.davydov@airbnb.com>.invalid>
>> > wrote:
>> >
>> >> Bolke, attached is the patch for the cgroups fix. Let me know =
which
>> >> branches you would like me to merge it to. If anyone has =
complaints about
>> >> the patch let me know (but it does not touch the core of airflow, =
only the
>> >> new cgroups task runner).
>> >>
>> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand <sanand@apache.org =
<mailto:sanand@apache.org>> wrote:
>> >>
>> >>> Actually, I see the error is further down..
>> >>>
>> >>>  File
>> >>> =
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
>> >>> line
>> >>> 469, in do_execute
>> >>>
>> >>>    cursor.execute(statement, parameters)
>> >>>
>> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null =
value in
>> >>> column "dag_id" violates not-null constraint
>> >>>
>> >>> DETAIL:  Failing row contains (null, running, 1, f).
>> >>>
>> >>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES =
(%(state)s,
>> >>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': =
u'running',
>> >>> 'dirty': False}]
>> >>>
>> >>> It looks like an autoincrement is missing for this table.
>> >>>
>> >>>
>> >>> I'm running `SQLAlchemy=3D=3D1.1.4` - I see our setup.py =
specifies any
>> > version
>> >>> greater than 0.9.8
>> >>>
>> >>> -s
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand =
<sanand@apache.org <mailto:sanand@apache.org>>
>> >>> wrote:
>> >>>
>> >>>> I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ =
<https://dist.apache.org/repos/dist/dev/incubator/airflow/>
>> >>>> airflow-1.8.0rc1+apache.incubating.tar.gz and then running =
airflow
>> >>>> upgradedb didn't quite work. First, I thought it completed
>> > successfully,
>> >>>> then saw errors some tables were indeed missing. I ran it again =
and
>> >>>> encountered the following exception :
>> >>>>
>> >>>> DB: postgresql://app_cousteau@db-cousteau.ep.stage.agari.com:543 =
<http://app_cousteau@db-cousteau.ep.stage.agari.com:543/>
>> >>> 2/airflow
>> >>>>
>> >>>> [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Will assume transactional DDL.
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 =
->
>> >>>> 211e584da130, add TI state index
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade 211e584da130 =
->
>> >>>> 64de9cddf6c9, add task fails journal table
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 =
->
>> >>>> f2ca10b85618, add dag_stats table
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 =
->
>> >>>> 4addfa1236f1, Add fractional seconds to mysql tables
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 =
->
>> >>>> 8504051e801b, xcom dag task indices
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade 8504051e801b =
->
>> >>>> 5e7d17757c7a, add pid field to TaskInstance
>> >>>>
>> >>>> INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a =
->
>> >>>> 127d2bf2dfa7, Add dag_id/state index on dag_run table
>> >>>>
>> >>>> =
/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
>> >>>> SAWarning: Column 'dag_stats.dag_id' is marked as a member of =
the
>> >>> primary
>> >>>> key for table 'dag_stats', but has no Python-side or server-side
>> > default
>> >>>> generator indicated, nor does it indicate 'autoincrement=3DTrue' =
or
>> >>>> 'nullable=3DTrue', and no explicit value is passed.  Primary key =
columns
>> >>>> typically may not store NULL. Note that as of SQLAlchemy 1.1,
>> >>>> 'autoincrement=3DTrue' must be indicated explicitly for =
composite (e.g.
>> >>>> multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY =
behavior is
>> >>>> expected for one of the columns in the primary key. CREATE TABLE
>> >>> statements
>> >>>> are impacted by this change as well on most backends.
>> >>>>
>> >>>
>> >>
>> >>
>> >
>> > --
>> >  _/
>> > _/ Alex Van Boxel
>>=20
>>=20
>> --=20
>>   _/
>> _/ Alex Van Boxel
>=20


--Apple-Mail=_78EBE853-9101-4089-B92C-893BBD6A8891--