airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Naqvi <ali.na...@conornash.com>
Subject Re: Airflow 1.8.0 Release Candidate 1
Date Tue, 07 Feb 2017 21:24:10 GMT
Hi Sid,
The deprecation warning is appearing only in the webserver, but is not
preventing the scheduler or worker from completing any tasks.
When you say "Most of our DAGs broke" do you mean they cannot be scheduled
at all?

Re upgradedb
we took a snapshot of our db on AWS and used that as a starting point for
our upgrade. For the migration `airflow upgradedb` worked. We are using MySQL
5.6.27.

Cheers,
Ali

On Tue, Feb 7, 2017 at 3:46 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> Apache scrubs patches, so I will cherry pick it in this case. Will figure
> out a better way to this.
>
> Bolke
>
> > On 7 Feb 2017, at 01:30, Dan Davydov <dan.davydov@airbnb.com.INVALID>
> wrote:
> >
> > Bolke, attached is the patch for the cgroups fix. Let me know which
> branches you would like me to merge it to. If anyone has complaints about
> the patch let me know (but it does not touch the core of airflow, only the
> new cgroups task runner).
> >
> > On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand <sanand@apache.org
> <mailto:sanand@apache.org>> wrote:
> > Actually, I see the error is further down..
> >
> >   File
> > "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> line
> > 469, in do_execute
> >
> >     cursor.execute(statement, parameters)
> >
> > sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> > column "dag_id" violates not-null constraint
> >
> > DETAIL:  Failing row contains (null, running, 1, f).
> >
> >  [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> > %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> > 'dirty': False}]
> >
> > It looks like an autoincrement is missing for this table.
> >
> >
> > I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> version
> > greater than 0.9.8
> >
> > -s
> >
> >
> >
> > On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand <sanand@apache.org
> <mailto:sanand@apache.org>> wrote:
> >
> > > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> > > https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
> > > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> > > upgradedb didn't quite work. First, I thought it completed
> successfully,
> > > then saw errors some tables were indeed missing. I ran it again and
> > > encountered the following exception :
> > >
> > > DB: postgresql://app_cousteau@db-cousteau.ep.stage.agari.com:
> 5432/airflow <http://app_cousteau@db-cousteau.ep.stage.agari.com:
> 5432/airflow>
> > >
> > > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> > >
> > > INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
> > >
> > > INFO  [alembic.runtime.migration] Will assume transactional DDL.
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> > > 211e584da130, add TI state index
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
> > > 64de9cddf6c9, add task fails journal table
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> > > f2ca10b85618, add dag_stats table
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> > > 4addfa1236f1, Add fractional seconds to mysql tables
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> > > 8504051e801b, xcom dag task indices
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
> > > 5e7d17757c7a, add pid field to TaskInstance
> > >
> > > INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> > > 127d2bf2dfa7, Add dag_id/state index on dag_run table
> > >
> > > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> > > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
> primary
> > > key for table 'dag_stats', but has no Python-side or server-side
> default
> > > generator indicated, nor does it indicate 'autoincrement=True' or
> > > 'nullable=True', and no explicit value is passed.  Primary key columns
> > > typically may not store NULL. Note that as of SQLAlchemy 1.1,
> > > 'autoincrement=True' must be indicated explicitly for composite (e.g.
> > > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
> > > expected for one of the columns in the primary key. CREATE TABLE
> statements
> > > are impacted by this change as well on most backends.
> > >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message