airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sid Anand <san...@apache.org>
Subject Re: [VOTE] Airflow 1.10.0rc1
Date Thu, 12 Jul 2018 02:50:20 GMT
FYI!
I just installed the release candidate. The first thing I noticed is a
missing tool tip for the Null State in the Recent Tasks column on the
landing page. Since the null globe is new to this UI, users will likely
hover over it to inquire what it means... and will be left wanting. Of
course, they could click on the globe, which will take them to
http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
which will always show an empty list, leaving them a bit more confused.

-s

On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
<carl.johan@tictail.com.invalid> wrote:

> Hi Bolke,
>
> (Switching email to avoid moderation on my emails.)
>
> The normal Airflow test suite does not fail as it uses a LC_ALL set to
> utf-8.
>
> I think it is a proper test though, it is a minimal reproducible version of
> the code that fails. And the only difference in behaviour is at 3.7 which
> we don’t support anyway so I’m fairly sure it is broken for all supported
> Python 3 versions.
>
> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
> unset and I see the same failure.
>
> I don’t think this is a big thing though and we could release it without
> the fix I made. I think most people run it with a sane LC_ALL, but
> apparently we didn’t.
> Here’s the log for the test:
>
> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> root@b99b297df111:/# locale
> LANG=C.UTF-8
> LANGUAGE=
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_PAPER="C.UTF-8"
> LC_NAME="C.UTF-8"
> LC_ADDRESS="C.UTF-8"
> LC_TELEPHONE="C.UTF-8"
> LC_MEASUREMENT="C.UTF-8"
> LC_IDENTIFICATION="C.UTF-8"
> LC_ALL=
> > unset LANG
> root@b99b297df111:/# locale
> LANG=
> LANGUAGE=
> LC_CTYPE="POSIX"
> LC_NUMERIC="POSIX"
> LC_TIME="POSIX"
> LC_COLLATE="POSIX"
> LC_MONETARY="POSIX"
> LC_MESSAGES="POSIX"
> LC_PAPER="POSIX"
> LC_NAME="POSIX"
> LC_ADDRESS="POSIX"
> LC_TELEPHONE="POSIX"
> LC_MEASUREMENT="POSIX"
> LC_IDENTIFICATION="POSIX"
> LC_ALL=
> root@b99b297df111:/# pip install -e .[devel]
> root@b99b297df111:/airflow# ./run_unit_tests.sh
> + export AIRFLOW_HOME=/root/airflow
> + AIRFLOW_HOME=/root/airflow
> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> + AIRFLOW__CORE__UNIT_TEST_MODE=True
> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + export AIRFLOW_USE_NEW_IMPORTS=1
> + AIRFLOW_USE_NEW_IMPORTS=1
> +++ dirname ./run_unit_tests.sh
> ++ cd .
> ++ pwd
> + DIR=/airflow
> + export PYTHONPATH=:/airflow/tests/test_utils
> + PYTHONPATH=:/airflow/tests/test_utils
> + nose_args=
> + which airflow
> + echo 'Initializing the DB'
> Initializing the DB
> + airflow resetdb
> + yes
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 6, in <module>
>     exec(compile(open(__file__).read(), __file__, 'exec'))
>   File "/airflow/airflow/bin/airflow", line 21, in <module>
>     from airflow import configuration
>   File "/airflow/airflow/__init__.py", line 35, in <module>
>     from airflow import configuration as conf
>   File "/airflow/airflow/configuration.py", line 106, in <module>
>     DEFAULT_CONFIG = f.read()
>   File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>     return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
> ordinal not in range(128)
> + '[' '' ']'
> + '[' -z '' ']'
> + nose_args='--with-coverage     --cover-erase     --cover-html
> --cover-package=airflow     --cover-html-dir=airflow/www/static/coverage
>   --with-ignore-docstrings     --rednose     --with-timer     -s     -v
> --logging-level=DEBUG '
> + echo 'Starting the unit tests with the following nose arguments:
> --with-coverage' --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> Starting the unit tests with the following nose arguments: --with-coverage
> --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> + nosetests --with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
> nose.plugins.cover: ERROR: Coverage not available: unable to import
> coverage module
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> ======================================================================
> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> ----------------------------------------------------------------------
>    Traceback (most recent call last):
>     /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> runTest
>       raise self.exc_val.with_traceback(self.tb)
>     /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> loadTestsFromName
>       addr.filename, addr.module)
>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> importFromPath
>       return self.importFromDir(dir_path, fqname)
>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> importFromDir
>       mod = load_module(part_fqname, fh, filename, desc)
>     /usr/local/lib/python3.5/imp.py line 245 in load_module
>       return load_package(name, filename)
>     /usr/local/lib/python3.5/imp.py line 217 in load_package
>       return _load(spec)
>     <frozen importlib._bootstrap> line 693 in _load
>
>     <frozen importlib._bootstrap> line 673 in _load_unlocked
>
>     <frozen importlib._bootstrap_external> line 697 in exec_module
>
>     <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>
>     airflow/__init__.py line 35 in <module>
>       from airflow import configuration as conf
>     airflow/configuration.py line 106 in <module>
>       DEFAULT_CONFIG = f.read()
>     /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>       return codecs.ascii_decode(input, self.errors)[0]
>    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082: ordinal not in range(128)
> ======================================================================
> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> ----------------------------------------------------------------------
>    Traceback (most recent call last):
>     /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
> runTest
>       raise self.exc_val.with_traceback(self.tb)
>     /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
> loadTestsFromName
>       addr.filename, addr.module)
>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
> importFromPath
>       return self.importFromDir(dir_path, fqname)
>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
> importFromDir
>       mod = load_module(part_fqname, fh, filename, desc)
>     /usr/local/lib/python3.5/imp.py line 245 in load_module
>       return load_package(name, filename)
>     /usr/local/lib/python3.5/imp.py line 217 in load_package
>       return _load(spec)
>     <frozen importlib._bootstrap> line 693 in _load
>
>     <frozen importlib._bootstrap> line 673 in _load_unlocked
>
>     <frozen importlib._bootstrap_external> line 697 in exec_module
>
>     <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>
>     tests/__init__.py line 25 in <module>
>       from .configuration import *
>     tests/configuration.py line 28 in <module>
>       from airflow import configuration
>     airflow/__init__.py line 35 in <module>
>       from airflow import configuration as conf
>     airflow/configuration.py line 106 in <module>
>       DEFAULT_CONFIG = f.read()
>     /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>       return codecs.ascii_decode(input, self.errors)[0]
>    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 21082: ordinal not in range(128)
>
> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>
> -----------------------------------------------------------------------------
> 2 tests run in 0.060 seconds.
> 2 errors (0 tests passed)
>
>
> --
> Carl Johan Gustavsson
>
> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>
> Hi Carl,
>
> That is not a real check, ie. Does Airflow have the same issue clean
> install on 3.5? Travis’ tests run on 3.5.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
> carl.j.gustavsson@gmail.com> het volgende geschreven:
> >
> > Hi Bolke,
> >
> >
> > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
> breaks
> >
> > Quick repro:
> >
> > ➜ ~ pyenv local 3.5.5
> > ➜ ~ locale
> > LANG=
> > LC_COLLATE="C"
> > LC_CTYPE="C"
> > LC_MESSAGES="C"
> > LC_MONETARY="C"
> > LC_NUMERIC="C"
> > LC_TIME="C"
> > LC_ALL=
> > ➜ ~ cat testweird.txt
> > ’
> > ➜ ~ python
> > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> open('testweird.txt').read()
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
> line 26, in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
> ordinal not in range(128)
> > >>>
> >
> >
> > Maybe not a blocking change but it is a breaking change from 1.9 I guess.
> >
> >
> > / Carl Johan
> >
> >
> >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com) wrote:
> >>
> >> Hi Carl,
> >>
> >> Did you this on python 3.5 as well? 3.6 is not an officially supported
> (yet). As a workaround is available I won’t consider this blocking btw.
> >>
> >> Bolke
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
> carl.j.gustavsson@gmail.com> het volgende geschreven:
> >>
> >>> Hi,
> >>>
> >>> First of all, thank you for all the work with the release management.
> >>>
> >>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
> Ubuntu 14.04.5, upgrading from a master build from February.
> >>>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
> (most recent call last):
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
> line 35, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
> import configuration as conf
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
> line 106, in <module>
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> DEFAULT_CONFIG = f.read()
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
> 26, in decode
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
> codecs.ascii_decode(input, self.errors)[0]
> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
> ordinal not in range(128)
> >>>
> >>> Removing the ’ on
>
> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
> solved the issue for me, and digging a bit deeper it seems Airflow now
> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
> Python to read the file as utf-8 and not ascii. (I think this was changed
> in to default to utf-8 in Python 3.7).
> >>>
> >>> I see 3 solutions for this
> >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
> similar.
> >>> 2. Change the default config file to not contain non-ascii characters.
> >>> 3. Always read the file as unicode regardless of the LC_ALL
> environment, by the encoding='utf-8’ parameter to open().
> >>>
> >>> I think 3 is the best solution, and I can prepare a PR for that if
> necessary .
> >>>
> >>> I guess this counts as -1 (non-binding)
> >>>
> >>> All the best
> >>>
> >>> Carl Johan
> >>>
> >>>
> >>>
> >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com) wrote:
> >>>>
> >>>> Hey all,
> >>>>
> >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
> release,
> >>>> which will last for 72 hours. Consider this my (binding) +1.
> >>>>
> >>>> Airflow 1.10.0 RC 1 is available at:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
<
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
> >>>>
> >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
> that
> >>>> comes with INSTALL instructions.
> >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> >>>> release.
> >>>>
> >>>> Public keys are available at:
> >>>>
> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
> >>>>
> >>>> The amount of JIRAs fixed is over 700. Please have a look at the
> changelog.
> >>>>
> >>>> Please note that the version number excludes the `rcX` string as well
> >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
> allow us
> >>>> to rename the artifact without modifying the artifact checksums when
> we
> >>>> actually release.
> >>>>
> >>>>
> >>>> Cheers,
> >>>> Bolke
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message