airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: [VOTE] Airflow 1.10.0rc1
Date Fri, 13 Jul 2018 13:46:10 GMT
With the SequentialExecutor the webserver also acts as the scheduler
(without parallelism)

2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <carl.johan@tictail.com>:

> Need to run the scheduler also right?
>
> --
> Carl Johan Gustavsson
>
> On 13 July 2018 at 15:37:22, Driesprong, Fokko (fokko@driesprong.frl)
> wrote:
>
> I've installed Airflow, but using a clean install and the sequential
> executor, the tasks are not being picked up.
>
> This is easy to replicate:
>
> docker run -t -i --rm -p 8080:8080 python:3.5 bash
> pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz
> airflow initdb
> airflow webserver
>
> I've enabled the http example dag and kicked of a dag, but it isn't being
> picked up by the SequentialScheduler.
>
> - Fokko
>
>
>
>
> 2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com>:
>
>> Hi Sid,
>>
>> Do you have a JIRA and a PR to address it? I can then consider it for RC2
>>
>> B.
>>
>> > On 12 Jul 2018, at 04:50, Sid Anand <sanand@apache.org> wrote:
>> >
>> > FYI!
>> > I just installed the release candidate. The first thing I noticed is a
>> missing tool tip for the Null State in the Recent Tasks column on the
>> landing page. Since the null globe is new to this UI, users will likely
>> hover over it to inquire what it means... and will be left wanting. Of
>> course, they could click on the globe, which will take them to
>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals
>> =example_bash_operator&flt2_state_equals=null <
>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equal
>> s=example_bash_operator&flt2_state_equals=null>, which will always show
>> an empty list, leaving them a bit more confused.
>> >
>> > -s
>> >
>> > On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <
>> carl.johan@tictail.com.invalid> wrote:
>> > Hi Bolke,
>> >
>> > (Switching email to avoid moderation on my emails.)
>> >
>> > The normal Airflow test suite does not fail as it uses a LC_ALL set to
>> > utf-8.
>> >
>> > I think it is a proper test though, it is a minimal reproducible
>> version of
>> > the code that fails. And the only difference in behaviour is at 3.7
>> which
>> > we don’t support anyway so I’m fairly sure it is broken for all
>> supported
>> > Python 3 versions.
>> >
>> > I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>> > unset and I see the same failure.
>> >
>> > I don’t think this is a big thing though and we could release it without
>> > the fix I made. I think most people run it with a sane LC_ALL, but
>> > apparently we didn’t.
>> > Here’s the log for the test:
>> >
>> > > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>> > root@b99b297df111:/# locale
>> > LANG=C.UTF-8
>> > LANGUAGE=
>> > LC_CTYPE="C.UTF-8"
>> > LC_NUMERIC="C.UTF-8"
>> > LC_TIME="C.UTF-8"
>> > LC_COLLATE="C.UTF-8"
>> > LC_MONETARY="C.UTF-8"
>> > LC_MESSAGES="C.UTF-8"
>> > LC_PAPER="C.UTF-8"
>> > LC_NAME="C.UTF-8"
>> > LC_ADDRESS="C.UTF-8"
>> > LC_TELEPHONE="C.UTF-8"
>> > LC_MEASUREMENT="C.UTF-8"
>> > LC_IDENTIFICATION="C.UTF-8"
>> > LC_ALL=
>> > > unset LANG
>> > root@b99b297df111:/# locale
>> > LANG=
>> > LANGUAGE=
>> > LC_CTYPE="POSIX"
>> > LC_NUMERIC="POSIX"
>> > LC_TIME="POSIX"
>> > LC_COLLATE="POSIX"
>> > LC_MONETARY="POSIX"
>> > LC_MESSAGES="POSIX"
>> > LC_PAPER="POSIX"
>> > LC_NAME="POSIX"
>> > LC_ADDRESS="POSIX"
>> > LC_TELEPHONE="POSIX"
>> > LC_MEASUREMENT="POSIX"
>> > LC_IDENTIFICATION="POSIX"
>> > LC_ALL=
>> > root@b99b297df111:/# pip install -e .[devel]
>> > root@b99b297df111:/airflow# ./run_unit_tests.sh
>> > + export AIRFLOW_HOME=/root/airflow
>> > + AIRFLOW_HOME=/root/airflow
>> > + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>> > + AIRFLOW__CORE__UNIT_TEST_MODE=True
>> > + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> > + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> > + export AIRFLOW_USE_NEW_IMPORTS=1
>> > + AIRFLOW_USE_NEW_IMPORTS=1
>> > +++ dirname ./run_unit_tests.sh
>> > ++ cd .
>> > ++ pwd
>> > + DIR=/airflow
>> > + export PYTHONPATH=:/airflow/tests/test_utils
>> > + PYTHONPATH=:/airflow/tests/test_utils
>> > + nose_args=
>> > + which airflow
>> > + echo 'Initializing the DB'
>> > Initializing the DB
>> > + airflow resetdb
>> > + yes
>> > Traceback (most recent call last):
>> >   File "/usr/local/bin/airflow", line 6, in <module>
>> >     exec(compile(open(__file__).read(), __file__, 'exec'))
>> >   File "/airflow/airflow/bin/airflow", line 21, in <module>
>> >     from airflow import configuration
>> >   File "/airflow/airflow/__init__.py", line 35, in <module>
>> >     from airflow import configuration as conf
>> >   File "/airflow/airflow/configuration.py", line 106, in <module>
>> >     DEFAULT_CONFIG = f.read()
>> >   File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>> decode
>> >     return codecs.ascii_decode(input, self.errors)[0]
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 21082:
>> > ordinal not in range(128)
>> > + '[' '' ']'
>> > + '[' -z '' ']'
>> > + nose_args='--with-coverage     --cover-erase     --cover-html
>> > --cover-package=airflow     --cover-html-dir=airflow/www/
>> static/coverage
>> >   --with-ignore-docstrings     --rednose     --with-timer     -s     -v
>> > --logging-level=DEBUG '
>> > + echo 'Starting the unit tests with the following nose arguments:
>> > --with-coverage' --cover-erase --cover-html --cover-package=airflow
>> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> > --rednose --with-timer -s -v --logging-level=DEBUG
>> > Starting the unit tests with the following nose arguments:
>> --with-coverage
>> > --cover-erase --cover-html --cover-package=airflow
>> > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> > --rednose --with-timer -s -v --logging-level=DEBUG
>> > + nosetests --with-coverage --cover-erase --cover-html
>> > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>> > --with-ignore-docstrings --rednose --with-timer -s -v
>> --logging-level=DEBUG
>> > nose.plugins.cover: ERROR: Coverage not available: unable to import
>> > coverage module
>> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> > position 21082: ordinal not in range(128)) ... ERROR
>> > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> > position 21082: ordinal not in range(128)) ... ERROR
>> > ======================================================================
>> > 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>> 0xe2
>> > in position 21082: ordinal not in range(128))
>> > ----------------------------------------------------------------------
>> >    Traceback (most recent call last):
>> >     /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> > runTest
>> >       raise self.exc_val.with_traceback(self.tb)
>> >     /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> > loadTestsFromName
>> >       addr.filename, addr.module)
>> >     /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> > importFromPath
>> >       return self.importFromDir(dir_path, fqname)
>> >     /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> > importFromDir
>> >       mod = load_module(part_fqname, fh, filename, desc)
>> >     /usr/local/lib/python3.5/imp.py line 245 in load_module
>> >       return load_package(name, filename)
>> >     /usr/local/lib/python3.5/imp.py line 217 in load_package
>> >       return _load(spec)
>> >     <frozen importlib._bootstrap> line 693 in _load
>> >
>> >     <frozen importlib._bootstrap> line 673 in _load_unlocked
>> >
>> >     <frozen importlib._bootstrap_external> line 697 in exec_module
>> >
>> >     <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>> >
>> >     airflow/__init__.py line 35 in <module>
>> >       from airflow import configuration as conf
>> >     airflow/configuration.py line 106 in <module>
>> >       DEFAULT_CONFIG = f.read()
>> >     /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>> >       return codecs.ascii_decode(input, self.errors)[0]
>> >    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> > 21082: ordinal not in range(128)
>> > ======================================================================
>> > 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>> 0xe2
>> > in position 21082: ordinal not in range(128))
>> > ----------------------------------------------------------------------
>> >    Traceback (most recent call last):
>> >     /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> > runTest
>> >       raise self.exc_val.with_traceback(self.tb)
>> >     /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> > loadTestsFromName
>> >       addr.filename, addr.module)
>> >     /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> > importFromPath
>> >       return self.importFromDir(dir_path, fqname)
>> >     /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> > importFromDir
>> >       mod = load_module(part_fqname, fh, filename, desc)
>> >     /usr/local/lib/python3.5/imp.py line 245 in load_module
>> >       return load_package(name, filename)
>> >     /usr/local/lib/python3.5/imp.py line 217 in load_package
>> >       return _load(spec)
>> >     <frozen importlib._bootstrap> line 693 in _load
>> >
>> >     <frozen importlib._bootstrap> line 673 in _load_unlocked
>> >
>> >     <frozen importlib._bootstrap_external> line 697 in exec_module
>> >
>> >     <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>> >
>> >     tests/__init__.py line 25 in <module>
>> >       from .configuration import *
>> >     tests/configuration.py line 28 in <module>
>> >       from airflow import configuration
>> >     airflow/__init__.py line 35 in <module>
>> >       from airflow import configuration as conf
>> >     airflow/configuration.py line 106 in <module>
>> >       DEFAULT_CONFIG = f.read()
>> >     /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>> >       return codecs.ascii_decode(input, self.errors)[0]
>> >    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> > 21082: ordinal not in range(128)
>> >
>> > [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>> > ------------------------------------------------------------
>> -----------------
>> > 2 tests run in 0.060 seconds.
>> > 2 errors (0 tests passed)
>> >
>> >
>> > --
>> > Carl Johan Gustavsson
>> >
>> > On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <mailto:
>> bdbruin@gmail.com>) wrote:
>> >
>> > Hi Carl,
>> >
>> > That is not a real check, ie. Does Airflow have the same issue clean
>> > install on 3.5? Travis’ tests run on 3.5.
>> >
>> > B.
>> >
>> > Verstuurd vanaf mijn iPad
>> >
>> > > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>> > carl.j.gustavsson@gmail.com <mailto:carl.j.gustavsson@gmail.com>> het
>> volgende geschreven:
>> > >
>> > > Hi Bolke,
>> > >
>> > >
>> > > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>> > breaks
>> > >
>> > > Quick repro:
>> > >
>> > > ➜ ~ pyenv local 3.5.5
>> > > ➜ ~ locale
>> > > LANG=
>> > > LC_COLLATE="C"
>> > > LC_CTYPE="C"
>> > > LC_MESSAGES="C"
>> > > LC_MONETARY="C"
>> > > LC_NUMERIC="C"
>> > > LC_TIME="C"
>> > > LC_ALL=
>> > > ➜ ~ cat testweird.txt
>> > > ’
>> > > ➜ ~ python
>> > > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>> > > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>> > > Type "help", "copyright", "credits" or "license" for more information.
>> > > >>> open('testweird.txt').read()
>> > > Traceback (most recent call last):
>> > > File "<stdin>", line 1, in <module>
>> > > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/as
>> cii.py",
>> > line 26, in decode
>> > > return codecs.ascii_decode(input, self.errors)[0]
>> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 0:
>> > ordinal not in range(128)
>> > > >>>
>> > >
>> > >
>> > > Maybe not a blocking change but it is a breaking change from 1.9 I
>> guess.
>> > >
>> > >
>> > > / Carl Johan
>> > >
>> > >
>> > >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com
>> <mailto:bdbruin@gmail.com>) wrote:
>> > >>
>> > >> Hi Carl,
>> > >>
>> > >> Did you this on python 3.5 as well? 3.6 is not an officially
>> supported
>> > (yet). As a workaround is available I won’t consider this blocking btw.
>> > >>
>> > >> Bolke
>> > >>
>> > >> Verstuurd vanaf mijn iPad
>> > >>
>> > >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>> > carl.j.gustavsson@gmail.com <mailto:carl.j.gustavsson@gmail.com>> het
>> volgende geschreven:
>> > >>
>> > >>> Hi,
>> > >>>
>> > >>> First of all, thank you for all the work with the release
>> management.
>> > >>>
>> > >>> I ran in to a weird issue testing the RC1, running under Python
>> 3.6.0 /
>> > Ubuntu 14.04.5, upgrading from a master build from February.
>> > >>>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>> > (most recent call last):
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>> airflow
>> > import configuration
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>> s/airflow/__init__.py",
>> > line 35, in <module>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>> airflow
>> > import configuration as conf
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>> s/airflow/configuration.py",
>> > line 106, in <module>
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> > DEFAULT_CONFIG = f.read()
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> > "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>> line
>> > 26, in decode
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>> > codecs.ascii_decode(input, self.errors)[0]
>> > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 20770:
>> > ordinal not in range(128)
>> > >>>
>> > >>> Removing the ’ on
>> > https://github.com/apache/incubator-airflow/blob/master/airf
>> low/config_templates/default_airflow.cfg#L613 <
>> https://github.com/apache/incubator-airflow/blob/master/air
>> flow/config_templates/default_airflow.cfg#L613>
>> > solved the issue for me, and digging a bit deeper it seems Airflow now
>> > requires setting LC_ALL=en_US.UTF-8 in the environment or similar to
>> force
>> > Python to read the file as utf-8 and not ascii. (I think this was
>> changed
>> > in to default to utf-8 in Python 3.7).
>> > >>>
>> > >>> I see 3 solutions for this
>> > >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>> > similar.
>> > >>> 2. Change the default config file to not contain non-ascii
>> characters.
>> > >>> 3. Always read the file as unicode regardless of the LC_ALL
>> > environment, by the encoding='utf-8’ parameter to open().
>> > >>>
>> > >>> I think 3 is the best solution, and I can prepare a PR for that
if
>> > necessary .
>> > >>>
>> > >>> I guess this counts as -1 (non-binding)
>> > >>>
>> > >>> All the best
>> > >>>
>> > >>> Carl Johan
>> > >>>
>> > >>>
>> > >>>
>> > >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com
>> <mailto:bdbruin@gmail.com>) wrote:
>> > >>>>
>> > >>>> Hey all,
>> > >>>>
>> > >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote
on the
>> > release,
>> > >>>> which will last for 72 hours. Consider this my (binding) +1.
>> > >>>>
>> > >>>> Airflow 1.10.0 RC 1 is available at:
>> > >>>>
>> > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.1
>> 0.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 10.0rc1/> <
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
>> > >>>>
>> > >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>> release
>> > that
>> > >>>> comes with INSTALL instructions.
>> > >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>> Python
>> > "sdist"
>> > >>>> release.
>> > >>>>
>> > >>>> Public keys are available at:
>> > >>>>
>> > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/
<
>> https://dist.apache.org/repos/dist/release/incubator/airflow/> <
>> > https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>>
>> > >>>>
>> > >>>> The amount of JIRAs fixed is over 700. Please have a look at
the
>> > changelog.
>> > >>>>
>> > >>>> Please note that the version number excludes the `rcX` string
as
>> well
>> > >>>> as the "+incubating" string, so it's now simply 1.10.0. This
will
>> > allow us
>> > >>>> to rename the artifact without modifying the artifact checksums
>> when
>> > we
>> > >>>> actually release.
>> > >>>>
>> > >>>>
>> > >>>> Cheers,
>> > >>>> Bolke
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message