airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow 1.8.0 alpha 2
Date Thu, 05 Jan 2017 18:39:50 GMT
I have found the email problem:

https://issues.apache.org/jira/browse/AIRFLOW-734

Working on a fix.

On Thu, Jan 5, 2017 at 9:10 AM, Chris Riccomini <criccomini@apache.org>
wrote:

> Also, I'm seeing a second issue:
>
> SMTP doesn't seem to work for us anymore:
>
> [2017-01-05 15:10:13,666] {models.py:1378} ERROR - SMTP AUTH extension not supported
by server.
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1374, in handle_failure
>     self.email_alert(error, is_retry=False)
>   File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1521, in email_alert
>     send_email(task.email, title, body)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line 43, in send_email
>     return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc,
mime_subtype=mime_subtype)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line 84, in send_email_smtp
>     send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun)
>   File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line 100, in send_MIME_email
>     s.login(SMTP_USER, SMTP_PASSWORD)
>   File "/usr/lib64/python2.7/smtplib.py", line 584, in login
>     raise SMTPException("SMTP AUTH extension not supported by server.")
> SMTPException: SMTP AUTH extension not supported by server.
>
> This was working on 1.7.1.2. Having a look now.
>
> On Thu, Jan 5, 2017 at 9:09 AM, Chris Riccomini <criccomini@apache.org>
> wrote:
>
>> Hey Robin,
>>
>> Awesome, thanks! I love open source. :) Let me try your patch out and
>> merge it.
>>
>> Cheers,
>> Chris
>>
>> On Thu, Jan 5, 2017 at 1:34 AM, Miller, Robin <
>> Robin.Miller@affiliate.oliverwyman.com> wrote:
>>
>>> Hi Chris,
>>>
>>>
>>> I think I ran into this issue when setting up LDAP Auth in our
>>> environment (we're using very close to master as we needed some of the
>>> newer features/bugfixes). The problem turned out to be that the search was
>>> finding no results, so the line:
>>>
>>>
>>> groups_list = [regex.search(i).group(1) for i in user_groups]
>>>
>>>
>>> would fail because it had no matching groups to return. This turned out
>>> to be because Windows Active Directory (the LDAP server we're using)
>>> returned capitals "CN=" where the code expected lowercase: regex =
>>> re.compile("cn=([^,]*).*")
>>>
>>>
>>> I haven't looked it up, but Windows Active Directory is case insensitive
>>> when it comes to usernames and groups, so I wouldn't be surprised if the
>>> protocol itself is case insensitive and both "cn=" and "CN=" should be
>>> considered valid. As such I've a PR open for a simple fix to make this
>>> regex case insensitive: https://github.com/apache/incu
>>> bator-airflow/pull/1945
>>>
>>>
>>> Hopefully this helps,
>>>
>>> Robin Miller
>>> OLIVER WYMAN
>>> robin.miller@affiliate.oliverwyman.com<mailto:robin.miller@a
>>> ffiliate.oliverwyman.com>
>>> www.oliverwyman.com<http://www.oliverwyman.com/>
>>>
>>> ________________________________
>>> From: Chris Riccomini <criccomini@apache.org>
>>> Sent: 05 January 2017 00:34:27
>>> To: dev@airflow.incubator.apache.org
>>> Subject: Re: Airflow 1.8.0 alpha 2
>>>
>>> I am now running 1.8.0a2 in our dev environment. It seems to be
>>> functioning
>>> well.
>>>
>>> One issue we've hit is that the LDAP auth plugin isn't working for us
>>> anymore:
>>>
>>> Traceback (most recent call last):
>>>  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1988, in
>>> wsgi_app
>>>    response = self.full_dispatch_request()
>>>  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1641, in
>>> full_dispatch_request
>>>    rv = self.handle_user_exception(e)
>>>  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1544, in
>>> handle_user_exception
>>>    reraise(exc_type, exc_value, tb)
>>>  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1639, in
>>> full_dispatch_request
>>>    rv = self.dispatch_request()
>>>  File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1625, in
>>> dispatch_request
>>>    return self.view_functions[rule.endpoint](**req.view_args)
>>>  File "/usr/lib64/python2.7/site-packages/flask_admin/base.py", line 69,
>>> in inner
>>>    return self._run_view(f, *args, **kwargs)
>>>  File "/usr/lib64/python2.7/site-packages/flask_admin/base.py", line
>>> 368,
>>> in _run_view
>>>    return fn(self, *args, **kwargs)
>>>  File "/usr/lib/python2.7/site-packages/airflow/www/views.py", line 657,
>>> in login
>>>    return airflow.login.login(self, request)
>>>  File
>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe
>>> nds/ldap_auth.py",
>>> line 276, in login
>>>    flask_login.login_user(LdapUser(user))
>>>  File "<string>", line 4, in __init__
>>>  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", line
>>> 306, in _initialize_instance
>>>    manager.dispatch.init_failure(self, args, kwargs)
>>>  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelp
>>> ers.py",
>>> line 60, in __exit__
>>>    compat.reraise(exc_type, exc_value, exc_tb)
>>>  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", line
>>> 303, in _initialize_instance
>>>    return manager.original_init(*mixed[1:], **kwargs)
>>>  File
>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe
>>> nds/ldap_auth.py",
>>> line 148, in __init__
>>>    user.username)
>>>  File
>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe
>>> nds/ldap_auth.py",
>>> line 106, in groups_user
>>>    groups_list = [regex.search(i).group(1) for i in user_groups]
>>> AttributeError: 'NoneType' object has no attribute 'group'
>>>
>>> I believe it's from this patch:
>>>
>>> https://github.com/apache/incubator-airflow/commit/d6d3f5367
>>> 3ba3736d7a858531823933cfef2bb4e
>>>
>>> I haven't dug into it yet. We'll need to fix it, though. In the
>>> meantime, I
>>> just commented out the call to `groups_user`. More to come.
>>>
>>> On Wed, Jan 4, 2017 at 12:32 PM, Alex Van Boxel <alex@vanboxel.be>
>>> wrote:
>>>
>>> > I have another fix that certainly need to be in the final release, but
>>> not
>>> > ready to merge due to failed tests:
>>> >
>>> > https://github.com/apache/incubator-airflow/pull/1961
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Jan 4, 2017 at 8:48 PM Chris Riccomini <criccomini@apache.org>
>>> > wrote:
>>> >
>>> > > Great, I will work toward deploying this in our dev cluster today.
:D
>>> > >
>>> > > On Wed, Jan 4, 2017 at 11:47 AM, Bolke de Bruin <bdbruin@gmail.com>
>>> > wrote:
>>> > >
>>> > > > Some issues remain:
>>> > > >
>>> > > > * one_failed not executed as dag run is marked failed seemingly
>>> > > > prematurely (@chris yes you should see this, see below for an
>>> example
>>> > > that
>>> > > > is not working properly), confirmed regression
>>> > > > * celery instability Alex
>>> > > > * Wrong DAG state after failure inside branch
>>> > > >
>>> > > > So Alpha 2 is definitely not ready for production, but please
do
>>> put in
>>> > > > your canary dags and let them run. I am still quite concerned
>>> about the
>>> > > > scheduler integrity and stability.
>>> > > >
>>> > > > - Bolke
>>> > > >
>>> > > >
>>> > > >
>>> > > > one_failed_not_executed.py
>>> > > > ====
>>> > > > from airflow import DAG
>>> > > > from airflow.operators.bash_operator import BashOperator
>>> > > > from airflow.operators.dummy_operator import DummyOperator
>>> > > > from datetime import datetime, timedelta
>>> > > >
>>> > > > default_args = {
>>> > > >     'owner': 'airflow',
>>> > > >     'depends_on_past': False,
>>> > > >     'start_date': datetime(2016,10,5,19),
>>> > > >     'email': ['airflow@airflow.com'],
>>> > > >     'email_on_failure': False,
>>> > > >     'email_on_retry': False,
>>> > > >     'retries': 1,
>>> > > >     'retry_delay': timedelta(seconds=1),
>>> > > > }
>>> > > >
>>> > > > dag = DAG('tutorial', default_args=default_args,
>>> > > schedule_interval='@once')
>>> > > >
>>> > > > task1 = BashOperator(
>>> > > >     task_id='first_one',
>>> > > >     bash_command='date',
>>> > > >     dag=dag)
>>> > > >
>>> > > > task2 = BashOperator(
>>> > > >     task_id='second_one',
>>> > > >     bash_command='this_should_not_work',
>>> > > >     dag=dag)
>>> > > >
>>> > > > task2.set_upstream(task1)
>>> > > >
>>> > > >
>>> > > >
>>> > > > task3 = BashOperator(
>>> > > >     task_id='third_one',
>>> > > >     bash_command='random_command_third',
>>> > > >     dag=dag)
>>> > > >
>>> > > > task3.set_upstream(task2)
>>> > > >
>>> > > > fail_task = DummyOperator(
>>> > > >     task_id='one_failed',
>>> > > >     trigger_rule='one_failed',
>>> > > >     dag=dag)
>>> > > >
>>> > > > fail_task.set_upstream([task1,task2,task3])
>>> > > >
>>> > > > > On 4 Jan 2017, at 20:35, Chris Riccomini <criccomini@apache.org>
>>> > > wrote:
>>> > > > >
>>> > > > > Bolke, can you describe the current state of the alpha 2
>>> release? I
>>> > saw
>>> > > > > some comments from Alex yesterday about celery instability.
If
>>> I'm
>>> > > > running
>>> > > > > on LocalExecutor, should I be seeing any issues?
>>> > > > >
>>> > > > > On Wed, Jan 4, 2017 at 8:20 AM, Bolke de Bruin <
>>> bdbruin@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > >> Hi All,
>>> > > > >>
>>> > > > >> I have put up Airflow 1.8.0 alpha 2 in
>>> https://people.apache.org/~
>>> > > > bolke/ <
>>> > > > >> https://people.apache.org/~bolke/> .
>>> > > > >>
>>> > > > >> Note: This still cannot be considered an Apache release.
>>> Working on
>>> > > > this.
>>> > > > >>
>>> > > > >> This build is signed (note it is served over https).
>>> > > > >>
>>> > > > >> Changes are in the area of scheduler stability.
>>> > > > >>
>>> > > > >> - Bolke
>>> > > >
>>> > > >
>>> > >
>>> > --
>>> >   _/
>>> > _/ Alex Van Boxel
>>> >
>>>
>>> ________________________________
>>> This e-mail and any attachments may be confidential or legally
>>> privileged. If you received this message in error or are not the intended
>>> recipient, you should destroy the e-mail message and any attachments or
>>> copies, and you are prohibited from retaining, distributing, disclosing or
>>> using any information contained herein. Please inform us of the erroneous
>>> delivery by return e-mail. Thank you for your cooperation.
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message