airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <ash_airflowl...@firemirror.com>
Subject Re: [VOTE] Airflow 1.10.0rc1
Date Thu, 12 Jul 2018 13:14:17 GMT
I wasn't paying enough attention. I can reproduce it.

I'm not sure it was intentional but the bug was introduced in https://github.com/apache/incubator-airflow/pull/3171
<https://github.com/apache/incubator-airflow/pull/3171>

I'd rather not release with a regression since 1.9, so updating my vote to -0.5 (binding),
but not blocking/vetoing release if others are okay with it.

-ash

> On 12 Jul 2018, at 13:46, Ash Berlin-Taylor <ash_airflowlist@firemirror.com> wrote:
> 
> That said I can't reproduce it myself (I will take discussion to the Jira ticket)
> 
> -ash
> 
>> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <ash_airflowlist@firemirror.com>
wrote:
>> 
>> Possible blocking regression here - .airflowignore doesn't seem to be respected anymore
>> 
>> https://issues.apache.org/jira/browse/AIRFLOW-2729 <https://issues.apache.org/jira/browse/AIRFLOW-2729>
>> 
>> -ash
>>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>> 
>>> Hi Jakob,
>>> 
>>> Thanks. We do include an INSTALL document that explains how to well install airflow
and is quite a standard location for install instructions. Or did the install instruction
needed to be included in the Vote?
>>> 
>>> B.
>>> 
>>> 
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
>>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jghoman@gmail.com> het
volgende geschreven:
>>>> 
>>>> +1 (binding)
>>>> 
>>>> * Sigs look good
>>>> * Artifact has incubating in name
>>>> * LICENSE/NOTICE/DISCLAIMER look good
>>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>>> other projects.
>>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>>> NOTICE as being imprecise.  This language remains.
>>>> * Spot check on license headers looks ok.
>>>> * Nit: Last release there was a request for instructions on how to
>>>> build the code.  This isn't included.
>>>> 
>>>> -Jakob
>>>> 
>>>>> On 11 July 2018 at 19:50, Sid Anand <sanand@apache.org> wrote:
>>>>> FYI!
>>>>> I just installed the release candidate. The first thing I noticed is
a
>>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>>> landing page. Since the null globe is new to this UI, users will likely
>>>>> hover over it to inquire what it means... and will be left wanting. Of
>>>>> course, they could click on the globe, which will take them to
>>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>>>>> which will always show an empty list, leaving them a bit more confused.
>>>>> 
>>>>> -s
>>>>> 
>>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>>> <carl.johan@tictail.com.invalid> wrote:
>>>>> 
>>>>>> Hi Bolke,
>>>>>> 
>>>>>> (Switching email to avoid moderation on my emails.)
>>>>>> 
>>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set
to
>>>>>> utf-8.
>>>>>> 
>>>>>> I think it is a proper test though, it is a minimal reproducible
version of
>>>>>> the code that fails. And the only difference in behaviour is at 3.7
which
>>>>>> we don’t support anyway so I’m fairly sure it is broken for all
supported
>>>>>> Python 3 versions.
>>>>>> 
>>>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>>>> unset and I see the same failure.
>>>>>> 
>>>>>> I don’t think this is a big thing though and we could release it
without
>>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>>> apparently we didn’t.
>>>>>> Here’s the log for the test:
>>>>>> 
>>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>>> root@b99b297df111:/# locale
>>>>>> LANG=C.UTF-8
>>>>>> LANGUAGE=
>>>>>> LC_CTYPE="C.UTF-8"
>>>>>> LC_NUMERIC="C.UTF-8"
>>>>>> LC_TIME="C.UTF-8"
>>>>>> LC_COLLATE="C.UTF-8"
>>>>>> LC_MONETARY="C.UTF-8"
>>>>>> LC_MESSAGES="C.UTF-8"
>>>>>> LC_PAPER="C.UTF-8"
>>>>>> LC_NAME="C.UTF-8"
>>>>>> LC_ADDRESS="C.UTF-8"
>>>>>> LC_TELEPHONE="C.UTF-8"
>>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>>> LC_ALL=
>>>>>>> unset LANG
>>>>>> root@b99b297df111:/# locale
>>>>>> LANG=
>>>>>> LANGUAGE=
>>>>>> LC_CTYPE="POSIX"
>>>>>> LC_NUMERIC="POSIX"
>>>>>> LC_TIME="POSIX"
>>>>>> LC_COLLATE="POSIX"
>>>>>> LC_MONETARY="POSIX"
>>>>>> LC_MESSAGES="POSIX"
>>>>>> LC_PAPER="POSIX"
>>>>>> LC_NAME="POSIX"
>>>>>> LC_ADDRESS="POSIX"
>>>>>> LC_TELEPHONE="POSIX"
>>>>>> LC_MEASUREMENT="POSIX"
>>>>>> LC_IDENTIFICATION="POSIX"
>>>>>> LC_ALL=
>>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>>> + AIRFLOW_HOME=/root/airflow
>>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>>> +++ dirname ./run_unit_tests.sh
>>>>>> ++ cd .
>>>>>> ++ pwd
>>>>>> + DIR=/airflow
>>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>>> + nose_args=
>>>>>> + which airflow
>>>>>> + echo 'Initializing the DB'
>>>>>> Initializing the DB
>>>>>> + airflow resetdb
>>>>>> + yes
>>>>>> Traceback (most recent call last):
>>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>>> from airflow import configuration
>>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>>> from airflow import configuration as conf
>>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>>> DEFAULT_CONFIG = f.read()
>>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
21082:
>>>>>> ordinal not in range(128)
>>>>>> + '[' '' ']'
>>>>>> + '[' -z '' ']'
>>>>>> + nose_args='--with-coverage     --cover-erase     --cover-html
>>>>>> --cover-package=airflow     --cover-html-dir=airflow/www/static/coverage
>>>>>> --with-ignore-docstrings     --rednose     --with-timer     -s  
  -v
>>>>>> --logging-level=DEBUG '
>>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>> Starting the unit tests with the following nose arguments: --with-coverage
>>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>>> coverage module
>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
in
>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
in
>>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>>> ======================================================================
>>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
byte 0xe2
>>>>>> in position 21082: ordinal not in range(128))
>>>>>> ----------------------------------------------------------------------
>>>>>> Traceback (most recent call last):
>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>> runTest
>>>>>>   raise self.exc_val.with_traceback(self.tb)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>> loadTestsFromName
>>>>>>   addr.filename, addr.module)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>> importFromPath
>>>>>>   return self.importFromDir(dir_path, fqname)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>> importFromDir
>>>>>>   mod = load_module(part_fqname, fh, filename, desc)
>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>   return load_package(name, filename)
>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>   return _load(spec)
>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>> 
>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>> 
>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>> 
>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>> 
>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>   from airflow import configuration as conf
>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>   DEFAULT_CONFIG = f.read()
>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>   return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>> 21082: ordinal not in range(128)
>>>>>> ======================================================================
>>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode
byte 0xe2
>>>>>> in position 21082: ordinal not in range(128))
>>>>>> ----------------------------------------------------------------------
>>>>>> Traceback (most recent call last):
>>>>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>>> runTest
>>>>>>   raise self.exc_val.with_traceback(self.tb)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>>> loadTestsFromName
>>>>>>   addr.filename, addr.module)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>>> importFromPath
>>>>>>   return self.importFromDir(dir_path, fqname)
>>>>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>>> importFromDir
>>>>>>   mod = load_module(part_fqname, fh, filename, desc)
>>>>>> /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>>   return load_package(name, filename)
>>>>>> /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>>   return _load(spec)
>>>>>> <frozen importlib._bootstrap> line 693 in _load
>>>>>> 
>>>>>> <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>>> 
>>>>>> <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>>> 
>>>>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>>> 
>>>>>> tests/__init__.py line 25 in <module>
>>>>>>   from .configuration import *
>>>>>> tests/configuration.py line 28 in <module>
>>>>>>   from airflow import configuration
>>>>>> airflow/__init__.py line 35 in <module>
>>>>>>   from airflow import configuration as conf
>>>>>> airflow/configuration.py line 106 in <module>
>>>>>>   DEFAULT_CONFIG = f.read()
>>>>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>>   return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>>> 21082: ordinal not in range(128)
>>>>>> 
>>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>>> 
>>>>>> -----------------------------------------------------------------------------
>>>>>> 2 tests run in 0.060 seconds.
>>>>>> 2 errors (0 tests passed)
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Carl Johan Gustavsson
>>>>>> 
>>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com) wrote:
>>>>>> 
>>>>>> Hi Carl,
>>>>>> 
>>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>>> 
>>>>>> B.
>>>>>> 
>>>>>> Verstuurd vanaf mijn iPad
>>>>>> 
>>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>> 
>>>>>>> Hi Bolke,
>>>>>>> 
>>>>>>> 
>>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all
but 3.7
>>>>>> breaks
>>>>>>> 
>>>>>>> Quick repro:
>>>>>>> 
>>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>>> ➜ ~ locale
>>>>>>> LANG=
>>>>>>> LC_COLLATE="C"
>>>>>>> LC_CTYPE="C"
>>>>>>> LC_MESSAGES="C"
>>>>>>> LC_MONETARY="C"
>>>>>>> LC_NUMERIC="C"
>>>>>>> LC_TIME="C"
>>>>>>> LC_ALL=
>>>>>>> ➜ ~ cat testweird.txt
>>>>>>> ’
>>>>>>> ➜ ~ python
>>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on
darwin
>>>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>>>> open('testweird.txt').read()
>>>>>>> Traceback (most recent call last):
>>>>>>> File "<stdin>", line 1, in <module>
>>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>>>>> line 26, in decode
>>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
0:
>>>>>> ordinal not in range(128)
>>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Maybe not a blocking change but it is a breaking change from
1.9 I guess.
>>>>>>> 
>>>>>>> 
>>>>>>> / Carl Johan
>>>>>>> 
>>>>>>> 
>>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com)
wrote:
>>>>>>>> 
>>>>>>>> Hi Carl,
>>>>>>>> 
>>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
supported
>>>>>> (yet). As a workaround is available I won’t consider this blocking
btw.
>>>>>>>> 
>>>>>>>> Bolke
>>>>>>>> 
>>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>>> 
>>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>>> carl.j.gustavsson@gmail.com> het volgende geschreven:
>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> First of all, thank you for all the work with the release
management.
>>>>>>>>> 
>>>>>>>>> I ran in to a weird issue testing the RC1, running under
Python 3.6.0 /
>>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>>> 
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
Traceback
>>>>>> (most recent call last):
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
File
>>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
from airflow
>>>>>> import configuration
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
File
>>>>>> 
>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>>>>> line 35, in <module>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
from airflow
>>>>>> import configuration as conf
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
File
>>>>>> 
>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>>>>> line 106, in <module>
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
File
>>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
line
>>>>>> 26, in decode
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
return
>>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
20770:
>>>>>> ordinal not in range(128)
>>>>>>>>> 
>>>>>>>>> Removing the ’ on
>>>>>> 
>>>>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>>>>> solved the issue for me, and digging a bit deeper it seems Airflow
now
>>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar
to force
>>>>>> Python to read the file as utf-8 and not ascii. (I think this was
changed
>>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>>> 
>>>>>>>>> I see 3 solutions for this
>>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8
or
>>>>>> similar.
>>>>>>>>> 2. Change the default config file to not contain non-ascii
characters.
>>>>>>>>> 3. Always read the file as unicode regardless of the
LC_ALL
>>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>>> 
>>>>>>>>> I think 3 is the best solution, and I can prepare a PR
for that if
>>>>>> necessary .
>>>>>>>>> 
>>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>>> 
>>>>>>>>> All the best
>>>>>>>>> 
>>>>>>>>> Carl Johan
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com)
wrote:
>>>>>>>>>> 
>>>>>>>>>> Hey all,
>>>>>>>>>> 
>>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling
a vote on the
>>>>>> release,
>>>>>>>>>> which will last for 72 hours. Consider this my (binding)
+1.
>>>>>>>>>> 
>>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>>> 
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/
<
>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>>>> 
>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz
is a source release
>>>>>> that
>>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is
the binary Python
>>>>>> "sdist"
>>>>>>>>>> release.
>>>>>>>>>> 
>>>>>>>>>> Public keys are available at:
>>>>>>>>>> 
>>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/
<
>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>>> 
>>>>>>>>>> The amount of JIRAs fixed is over 700. Please have
a look at the
>>>>>> changelog.
>>>>>>>>>> 
>>>>>>>>>> Please note that the version number excludes the
`rcX` string as well
>>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0.
This will
>>>>>> allow us
>>>>>>>>>> to rename the artifact without modifying the artifact
checksums when
>>>>>> we
>>>>>>>>>> actually release.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Bolke
>>>>>> 
>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message