airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: [VOTE] Airflow 1.10.0rc1
Date Fri, 13 Jul 2018 16:36:06 GMT
Example dags are not picked up. If you put a dag in the normal dag folder it works fine. 

Please create a jira for this @fokko. A pr would be appreciated. 

B. 

Sent from my iPhone

> On 13 Jul 2018, at 15:46, Driesprong, Fokko <fokko@driesprong.frl> wrote:
> 
> With the SequentialExecutor the webserver also acts as the scheduler
> (without parallelism)
> 
> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <carl.johan@tictail.com>:
> 
>> Need to run the scheduler also right?
>> 
>> --
>> Carl Johan Gustavsson
>> 
>> On 13 July 2018 at 15:37:22, Driesprong, Fokko (fokko@driesprong.frl)
>> wrote:
>> 
>> I've installed Airflow, but using a clean install and the sequential
>> executor, the tasks are not being picked up.
>> 
>> This is easy to replicate:
>> 
>> docker run -t -i --rm -p 8080:8080 python:3.5 bash
>> pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>> 10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz
>> airflow initdb
>> airflow webserver
>> 
>> I've enabled the http example dag and kicked of a dag, but it isn't being
>> picked up by the SequentialScheduler.
>> 
>> - Fokko
>> 
>> 
>> 
>> 
>> 2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bdbruin@gmail.com>:
>> 
>>> Hi Sid,
>>> 
>>> Do you have a JIRA and a PR to address it? I can then consider it for RC2
>>> 
>>> B.
>>> 
>>>> On 12 Jul 2018, at 04:50, Sid Anand <sanand@apache.org> wrote:
>>>> 
>>>> FYI!
>>>> I just installed the release candidate. The first thing I noticed is a
>>> missing tool tip for the Null State in the Recent Tasks column on the
>>> landing page. Since the null globe is new to this UI, users will likely
>>> hover over it to inquire what it means... and will be left wanting. Of
>>> course, they could click on the globe, which will take them to
>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals
>>> =example_bash_operator&flt2_state_equals=null <
>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equal
>>> s=example_bash_operator&flt2_state_equals=null>, which will always show
>>> an empty list, leaving them a bit more confused.
>>>> 
>>>> -s
>>>> 
>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson <
>>> carl.johan@tictail.com.invalid> wrote:
>>>> Hi Bolke,
>>>> 
>>>> (Switching email to avoid moderation on my emails.)
>>>> 
>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>>> utf-8.
>>>> 
>>>> I think it is a proper test though, it is a minimal reproducible
>>> version of
>>>> the code that fails. And the only difference in behaviour is at 3.7
>>> which
>>>> we don’t support anyway so I’m fairly sure it is broken for all
>>> supported
>>>> Python 3 versions.
>>>> 
>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>> unset and I see the same failure.
>>>> 
>>>> I don’t think this is a big thing though and we could release it without
>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>> apparently we didn’t.
>>>> Here’s the log for the test:
>>>> 
>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>> root@b99b297df111:/# locale
>>>> LANG=C.UTF-8
>>>> LANGUAGE=
>>>> LC_CTYPE="C.UTF-8"
>>>> LC_NUMERIC="C.UTF-8"
>>>> LC_TIME="C.UTF-8"
>>>> LC_COLLATE="C.UTF-8"
>>>> LC_MONETARY="C.UTF-8"
>>>> LC_MESSAGES="C.UTF-8"
>>>> LC_PAPER="C.UTF-8"
>>>> LC_NAME="C.UTF-8"
>>>> LC_ADDRESS="C.UTF-8"
>>>> LC_TELEPHONE="C.UTF-8"
>>>> LC_MEASUREMENT="C.UTF-8"
>>>> LC_IDENTIFICATION="C.UTF-8"
>>>> LC_ALL=
>>>>> unset LANG
>>>> root@b99b297df111:/# locale
>>>> LANG=
>>>> LANGUAGE=
>>>> LC_CTYPE="POSIX"
>>>> LC_NUMERIC="POSIX"
>>>> LC_TIME="POSIX"
>>>> LC_COLLATE="POSIX"
>>>> LC_MONETARY="POSIX"
>>>> LC_MESSAGES="POSIX"
>>>> LC_PAPER="POSIX"
>>>> LC_NAME="POSIX"
>>>> LC_ADDRESS="POSIX"
>>>> LC_TELEPHONE="POSIX"
>>>> LC_MEASUREMENT="POSIX"
>>>> LC_IDENTIFICATION="POSIX"
>>>> LC_ALL=
>>>> root@b99b297df111:/# pip install -e .[devel]
>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>> + export AIRFLOW_HOME=/root/airflow
>>>> + AIRFLOW_HOME=/root/airflow
>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>> +++ dirname ./run_unit_tests.sh
>>>> ++ cd .
>>>> ++ pwd
>>>> + DIR=/airflow
>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>> + nose_args=
>>>> + which airflow
>>>> + echo 'Initializing the DB'
>>>> Initializing the DB
>>>> + airflow resetdb
>>>> + yes
>>>> Traceback (most recent call last):
>>>>  File "/usr/local/bin/airflow", line 6, in <module>
>>>>    exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>  File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>    from airflow import configuration
>>>>  File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>    from airflow import configuration as conf
>>>>  File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>    DEFAULT_CONFIG = f.read()
>>>>  File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in
>>> decode
>>>>    return codecs.ascii_decode(input, self.errors)[0]
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 21082:
>>>> ordinal not in range(128)
>>>> + '[' '' ']'
>>>> + '[' -z '' ']'
>>>> + nose_args='--with-coverage     --cover-erase     --cover-html
>>>> --cover-package=airflow     --cover-html-dir=airflow/www/
>>> static/coverage
>>>>  --with-ignore-docstrings     --rednose     --with-timer     -s     -v
>>>> --logging-level=DEBUG '
>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>> Starting the unit tests with the following nose arguments:
>>> --with-coverage
>>>> --cover-erase --cover-html --cover-package=airflow
>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>> --with-ignore-docstrings --rednose --with-timer -s -v
>>> --logging-level=DEBUG
>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>> coverage module
>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>> ======================================================================
>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>>> 0xe2
>>>> in position 21082: ordinal not in range(128))
>>>> ----------------------------------------------------------------------
>>>>   Traceback (most recent call last):
>>>>    /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>> runTest
>>>>      raise self.exc_val.with_traceback(self.tb)
>>>>    /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>> loadTestsFromName
>>>>      addr.filename, addr.module)
>>>>    /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>> importFromPath
>>>>      return self.importFromDir(dir_path, fqname)
>>>>    /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>> importFromDir
>>>>      mod = load_module(part_fqname, fh, filename, desc)
>>>>    /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>      return load_package(name, filename)
>>>>    /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>      return _load(spec)
>>>>    <frozen importlib._bootstrap> line 693 in _load
>>>> 
>>>>    <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>> 
>>>>    <frozen importlib._bootstrap_external> line 697 in exec_module
>>>> 
>>>>    <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>> 
>>>>    airflow/__init__.py line 35 in <module>
>>>>      from airflow import configuration as conf
>>>>    airflow/configuration.py line 106 in <module>
>>>>      DEFAULT_CONFIG = f.read()
>>>>    /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>      return codecs.ascii_decode(input, self.errors)[0]
>>>>   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>> 21082: ordinal not in range(128)
>>>> ======================================================================
>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte
>>> 0xe2
>>>> in position 21082: ordinal not in range(128))
>>>> ----------------------------------------------------------------------
>>>>   Traceback (most recent call last):
>>>>    /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>> runTest
>>>>      raise self.exc_val.with_traceback(self.tb)
>>>>    /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>> loadTestsFromName
>>>>      addr.filename, addr.module)
>>>>    /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>> importFromPath
>>>>      return self.importFromDir(dir_path, fqname)
>>>>    /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>> importFromDir
>>>>      mod = load_module(part_fqname, fh, filename, desc)
>>>>    /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>      return load_package(name, filename)
>>>>    /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>      return _load(spec)
>>>>    <frozen importlib._bootstrap> line 693 in _load
>>>> 
>>>>    <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>> 
>>>>    <frozen importlib._bootstrap_external> line 697 in exec_module
>>>> 
>>>>    <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>> 
>>>>    tests/__init__.py line 25 in <module>
>>>>      from .configuration import *
>>>>    tests/configuration.py line 28 in <module>
>>>>      from airflow import configuration
>>>>    airflow/__init__.py line 35 in <module>
>>>>      from airflow import configuration as conf
>>>>    airflow/configuration.py line 106 in <module>
>>>>      DEFAULT_CONFIG = f.read()
>>>>    /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>      return codecs.ascii_decode(input, self.errors)[0]
>>>>   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>> 21082: ordinal not in range(128)
>>>> 
>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>> ------------------------------------------------------------
>>> -----------------
>>>> 2 tests run in 0.060 seconds.
>>>> 2 errors (0 tests passed)
>>>> 
>>>> 
>>>> --
>>>> Carl Johan Gustavsson
>>>> 
>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@gmail.com <mailto:
>>> bdbruin@gmail.com>) wrote:
>>>> 
>>>> Hi Carl,
>>>> 
>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>> install on 3.5? Travis’ tests run on 3.5.
>>>> 
>>>> B.
>>>> 
>>>> Verstuurd vanaf mijn iPad
>>>> 
>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>> carl.j.gustavsson@gmail.com <mailto:carl.j.gustavsson@gmail.com>>
het
>>> volgende geschreven:
>>>>> 
>>>>> Hi Bolke,
>>>>> 
>>>>> 
>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>>> breaks
>>>>> 
>>>>> Quick repro:
>>>>> 
>>>>> ➜ ~ pyenv local 3.5.5
>>>>> ➜ ~ locale
>>>>> LANG=
>>>>> LC_COLLATE="C"
>>>>> LC_CTYPE="C"
>>>>> LC_MESSAGES="C"
>>>>> LC_MONETARY="C"
>>>>> LC_NUMERIC="C"
>>>>> LC_TIME="C"
>>>>> LC_ALL=
>>>>> ➜ ~ cat testweird.txt
>>>>> ’
>>>>> ➜ ~ python
>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>> open('testweird.txt').read()
>>>>> Traceback (most recent call last):
>>>>> File "<stdin>", line 1, in <module>
>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/as
>>> cii.py",
>>>> line 26, in decode
>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 0:
>>>> ordinal not in range(128)
>>>>>>>> 
>>>>> 
>>>>> 
>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I
>>> guess.
>>>>> 
>>>>> 
>>>>> / Carl Johan
>>>>> 
>>>>> 
>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@gmail.com
>>> <mailto:bdbruin@gmail.com>) wrote:
>>>>>> 
>>>>>> Hi Carl,
>>>>>> 
>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially
>>> supported
>>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>> 
>>>>>> Bolke
>>>>>> 
>>>>>> Verstuurd vanaf mijn iPad
>>>>>> 
>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>> carl.j.gustavsson@gmail.com <mailto:carl.j.gustavsson@gmail.com>>
het
>>> volgende geschreven:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> First of all, thank you for all the work with the release
>>> management.
>>>>>>> 
>>>>>>> I ran in to a weird issue testing the RC1, running under Python
>>> 3.6.0 /
>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>> 
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>>> (most recent call last):
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>> import configuration
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>>> s/airflow/__init__.py",
>>>> line 35, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from
>>> airflow
>>>> import configuration as conf
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package
>>> s/airflow/configuration.py",
>>>> line 106, in <module>
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>> DEFAULT_CONFIG = f.read()
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py",
>>> line
>>>> 26, in decode
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>> 20770:
>>>> ordinal not in range(128)
>>>>>>> 
>>>>>>> Removing the ’ on
>>>> https://github.com/apache/incubator-airflow/blob/master/airf
>>> low/config_templates/default_airflow.cfg#L613 <
>>> https://github.com/apache/incubator-airflow/blob/master/air
>>> flow/config_templates/default_airflow.cfg#L613>
>>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to
>>> force
>>>> Python to read the file as utf-8 and not ascii. (I think this was
>>> changed
>>>> in to default to utf-8 in Python 3.7).
>>>>>>> 
>>>>>>> I see 3 solutions for this
>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8
or
>>>> similar.
>>>>>>> 2. Change the default config file to not contain non-ascii
>>> characters.
>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>> 
>>>>>>> I think 3 is the best solution, and I can prepare a PR for that
if
>>>> necessary .
>>>>>>> 
>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>> 
>>>>>>> All the best
>>>>>>> 
>>>>>>> Carl Johan
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@gmail.com
>>> <mailto:bdbruin@gmail.com>) wrote:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote
on the
>>>> release,
>>>>>>>> which will last for 72 hours. Consider this my (binding)
+1.
>>>>>>>> 
>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>> 
>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.1
>>> 0.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
>>> 10.0rc1/> <
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>>
>>>>>>>> 
>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source
>>> release
>>>> that
>>>>>>>> comes with INSTALL instructions.
>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary
>>> Python
>>>> "sdist"
>>>>>>>> release.
>>>>>>>> 
>>>>>>>> Public keys are available at:
>>>>>>>> 
>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/
<
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> <
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>>
>>>>>>>> 
>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look
at the
>>>> changelog.
>>>>>>>> 
>>>>>>>> Please note that the version number excludes the `rcX` string
as
>>> well
>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This
will
>>>> allow us
>>>>>>>> to rename the artifact without modifying the artifact checksums
>>> when
>>>> we
>>>>>>>> actually release.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Bolke
>>> 
>>> 
>> 

Mime
View raw message