airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.8.0 BETA 3
Date Thu, 26 Jan 2017 10:40:14 GMT
Hi Alex 

Are you sure you are in a fully clean environment? Sometimes things remain in the airflow
directories. This worked for me:

rm -rf site-packages/airflow*
python setup.py install
airflow initdb

Bolke


> On 26 Jan 2017, at 10:53, Alex Van Boxel <alex@vanboxel.be> wrote:
> 
> About the database error: starting from scratch also gives the same error:
> 
> Fresh install. Delete airflow.db sqllite db. And the : airflow initdb
> 
> same error as above.
> 
> On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <alex@vanboxel.be> wrote:
> 
>> Not directly one I can share. I'll spend some time looking at it. I'll try
>> to create a unittest of it.
>> 
>> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bdbruin@gmail.com> wrote:
>> 
>> Do you have a example dag for this? That makes it easier to work on.
>> 
>> Bolke
>> 
>> Sent from my iPhone
>> 
>>> On 26 Jan 2017, at 08:36, Alex Van Boxel <alex@vanboxel.be> wrote:
>>> 
>>> Another thing that I noticed (but observed it in beta 2 as well). Is the
>>> following:
>>> 
>>> - The following trigger should not fire.
>>> --- Trigger rule is ONE_SUCCESS
>>> --- upstream: UP_FOR_RETRY + SKIPPED
>>> => task get's triggered
>>> => resulting SKIPPED
>>> => DAG marked success, with half of the Tasks never scheduled
>>> 
>>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure
>>> described above)
>>> --- *Does this make sense* ?!
>>> 
>>> Both are a problem, and certainly the combination of both. I'll see if I
>>> can spend some time investigating this.
>>> 
>>> 
>>> 
>>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bdbruin@gmail.com>
>> wrote:
>>>> 
>>>> Mmm that is due to the reverting of one changes to the db. Need to look
>>>> into that how to fix it.
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <alex@vanboxel.be> wrote:
>>>>> 
>>>>> I do seem to have a problem upgrading the MySQL database with the last
>>>>> commit:
>>>>> 
>>>>> 
>>>>> 2017-01-25T23:41:55.662572654Z
>>>>> 
>>>> 
>> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69:
>>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 ->
>>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not
>>>> present
>>>>> 2017-01-25T23:41:55.662613670Z   warnings.warn(msg)
>>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last):
>>>>> 2017-01-25T23:41:55.664582238Z   File "/usr/local/bin/airflow", line
4,
>>>> in
>>>>> <module>
>>>>> 2017-01-25T23:41:55.664758457Z
>>>>> 
>>>> 
>> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating',
>>>>> 'airflow')
>>>>> 2017-01-25T23:41:55.664776553Z   File
>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> line
>>>>> 739, in run_script
>>>>> 2017-01-25T23:41:55.664937644Z
>>>>> self.require(requires)[0].run_script(script_name, ns)
>>>>> 2017-01-25T23:41:55.664952001Z   File
>>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py",
>> line
>>>>> 1494, in run_script
>>>>> 2017-01-25T23:41:55.665072840Z     exec(code, namespace, namespace)
>>>>> 2017-01-25T23:41:55.665086884Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
>>>>> line 28, in <module>
>>>>> 2017-01-25T23:41:55.665211755Z     args.func(args)
>>>>> 2017-01-25T23:41:55.665225157Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py",
>>>>> line 931, in upgradedb
>>>>> 2017-01-25T23:41:55.665326691Z     db_utils.upgradedb()
>>>>> 2017-01-25T23:41:55.665346979Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py",
>>>>> line 292, in upgradedb
>>>>> 2017-01-25T23:41:55.665449297Z     command.upgrade(config, 'heads')
>>>>> 2017-01-25T23:41:55.665462333Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>> line 174, in upgrade
>>>>> 2017-01-25T23:41:55.665521466Z     script.run_env()
>>>>> 2017-01-25T23:41:55.665527232Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>> line 416, in run_env2017-01-25T23:41:55.665638879Z
>>>>> util.load_python_file(self.dir, 'env.py')
>>>>> 2017-01-25T23:41:55.665656437Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py",
>>>>> line 93, in load_python_file
>>>>> 2017-01-25T23:41:55.665682760Z     module = load_module_py(module_id,
>>>> path)
>>>>> 2017-01-25T23:41:55.665699257Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py",
>>>>> line 79, in load_module_py
>>>>> 2017-01-25T23:41:55.665725036Z     mod = imp.load_source(module_id,
>> path,
>>>>> fp)
>>>>> 2017-01-25T23:41:55.665751433Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>> line 88, in <module>
>>>>> 2017-01-25T23:41:55.665778219Z     run_migrations_online()
>>>>> 2017-01-25T23:41:55.665787820Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py",
>>>>> line 83, in run_migrations_online
>>>>> 2017-01-25T23:41:55.665811353Z     context.run_migrations()
>>>>> 2017-01-25T23:41:55.665818933Z   File "<string>", line 8, in
>>>> run_migrations
>>>>> 2017-01-25T23:41:55.666707547Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py",
>>>>> line 797, in run_migrations
>>>>> 2017-01-25T23:41:55.666839945Z
>>>> self.get_context().run_migrations(**kw)
>>>>> 2017-01-25T23:41:55.666859336Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py",
>>>>> line 305, in run_migrations
>>>>> 2017-01-25T23:41:55.666970791Z     for step in
>> self._migrations_fn(heads,
>>>>> self):
>>>>> 2017-01-25T23:41:55.666992265Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py",
>>>>> line 163, in upgrade
>>>>> 2017-01-25T23:41:55.667021804Z     return
>> script._upgrade_revs(revision,
>>>>> rev)
>>>>> 2017-01-25T23:41:55.667032496Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py",
>>>>> line 329, in _upgrade_revs
>>>>> 2017-01-25T23:41:55.667144295Z     revs = list(revs)
>>>>> 2017-01-25T23:41:55.667157407Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 641, in _iterate_revisions
>>>>> 2017-01-25T23:41:55.667302027Z     requested_lowers =
>>>>> self.get_revisions(lower)
>>>>> 2017-01-25T23:41:55.667315913Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 298, in get_revisions
>>>>> 2017-01-25T23:41:55.667390590Z     return
>>>> sum([self.get_revisions(id_elem)
>>>>> for id_elem in id_], ())
>>>>> 2017-01-25T23:41:55.667411867Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 300, in get_revisions
>>>>> 2017-01-25T23:41:55.667419226Z     resolved_id, branch_label =
>>>>> self._resolve_revision_number(id_)
>>>>> 2017-01-25T23:41:55.667434110Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 433, in _resolve_revision_number
>>>>> 2017-01-25T23:41:55.667576273Z     self._revision_map
>>>>> 2017-01-25T23:41:55.667590641Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py",
>>>>> line 240, in __get__
>>>>> 2017-01-25T23:41:55.667648561Z     obj.__dict__[self.__name__] =
>> result =
>>>>> self.fget(obj)
>>>>> 2017-01-25T23:41:55.667658598Z   File
>>>>> 
>>>> 
>> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py",
>>>>> line 151, in _revision_map
>>>>> 2017-01-25T23:41:55.667680311Z     down_revision = map_[downrev]
>>>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5'
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bdbruin@gmail.com>
>>>> wrote:
>>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I have made the THIRD beta of Airflow 1.8.0 available at:
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
>>>> keys
>>>>>> are available at
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
. It
>> is
>>>>>> tagged with a local version “apache.incubating” so it allows
upgrading
>>>> from
>>>>>> earlier releases. This beta is available for testing in a more
>>>> production
>>>>>> like setting (acceptance environment?).
>>>>>> 
>>>>>> I would like to encourage everyone  to try it out, to report back
any
>>>>>> issues so we get to a rock solid release of 1.8.0. When reporting
>>>> issues a
>>>>>> test case or even a fix is highly appreciated.
>>>>>> 
>>>>>> Issues cleared:
>>>>>> 
>>>>>> * Manual trigger not working
>>>>>> * Performance issues with MySQL
>>>>>> * Postgres auto-commit support left over to the driver
>>>>>> * Poison pill taken while task has exited
>>>>>> * Keep cgroups optional
>>>>>> * Funcsigs pinned to 1.0.0
>>>>>> 
>>>>>> Issue(s) remaining (blocker for RC):
>>>>>> * Cgroups not py3 compatible
>>>>>> 
>>>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks
>> for
>>>>>> reporting issues and keep on testing please :). Moving towards RC
I
>>>> tend to
>>>>>> like small bug fixes only. When we mark RC (do we need to vote on
>> this?)
>>>>>> the procedure becomes even more strict. Please remember that the
FINAL
>>>>>> release is dependent on a vote on the IPMC mailinglist.
>>>>>> 
>>>>>> Cheers
>>>>>> Bolke
>>>>> 
>>>>> --
>>>>> _/
>>>>> _/ Alex Van Boxel
>>>> 
>>> --
>>> _/
>>> _/ Alex Van Boxel
>> 
>> --
>>  _/
>> _/ Alex Van Boxel
>> 
> -- 
>  _/
> _/ Alex Van Boxel


Mime
View raw message