airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremiah Lowin <jlo...@gmail.com>
Subject Re: Airflow 1.8.0 alpha 2
Date Fri, 06 Jan 2017 20:50:04 GMT
Bolke -- I will see if I can solve it

On Fri, Jan 6, 2017 at 3:46 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:

> For the XCOM issue: https://issues.apache.org/jira/browse/AIRFLOW-738 <
> https://issues.apache.org/jira/browse/AIRFLOW-738>
>
> With example dag that exposes the issue. Would be nice if someone with
> XCOM exposure could take a look.
>
> - Bolke
>
> > On 6 Jan 2017, at 15:31, Bolke de Bruin <bdbruin@gmail.com> wrote:
> >
> > State of to be Alpha 3:
> >
> > Blockers:
> >
> > * XCom throws an duplicate / locking error (new - confirmed)
> > * one_failed task not being run: now seems to pass suddenly (so fixed?)
> -> need to investigate why
> >
> > Fixed issues
> > * Regression in email
> > * LDAP case sensitivity
> >
> > Pending features:
> > * DAG.catchup : minor changes needed, documentation still required,
> integration tests seem to pass flawlessly
> > * Cgroups + impersonation: clean up of patches on going, more tests and
> more elaborate documentation required. Integration tests not executed yet
> > * Schedule all pending DAG runs in a single scheduler loop: no progress
> (**)
> > * Email attachments
> > * Add execution_date to trigger_dag
> >
> > If the pending features are merged within a reasonable time frame
> (except for **, as no progress currently) then I am planning to mark the
> tarball as Beta and only allow bug fixes and (very) minor features.
> Hopefully end of next week.
> >
> > Bolke.
> >
> >
> >
> >> On 5 Jan 2017, at 20:07, Chris Riccomini <criccomini@apache.org> wrote:
> >>
> >> I have merged Robin's LDAP patch.
> >>
> >> On Thu, Jan 5, 2017 at 10:39 AM, Chris Riccomini <criccomini@apache.org
> >
> >> wrote:
> >>
> >>> I have found the email problem:
> >>>
> >>> https://issues.apache.org/jira/browse/AIRFLOW-734
> >>>
> >>> Working on a fix.
> >>>
> >>> On Thu, Jan 5, 2017 at 9:10 AM, Chris Riccomini <criccomini@apache.org
> >
> >>> wrote:
> >>>
> >>>> Also, I'm seeing a second issue:
> >>>>
> >>>> SMTP doesn't seem to work for us anymore:
> >>>>
> >>>> [2017-01-05 15:10:13,666] {models.py:1378} ERROR - SMTP AUTH
> extension not supported by server.
> >>>> Traceback (most recent call last):
> >>>> File "/usr/lib/python2.7/site-packages/airflow/models.py", line
> 1374, in handle_failure
> >>>>   self.email_alert(error, is_retry=False)
> >>>> File "/usr/lib/python2.7/site-packages/airflow/models.py", line
> 1521, in email_alert
> >>>>   send_email(task.email, title, body)
> >>>> File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line
> 43, in send_email
> >>>>   return backend(to, subject, html_content, files=files,
> dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype)
> >>>> File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line
> 84, in send_email_smtp
> >>>>   send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun)
> >>>> File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line
> 100, in send_MIME_email
> >>>>   s.login(SMTP_USER, SMTP_PASSWORD)
> >>>> File "/usr/lib64/python2.7/smtplib.py", line 584, in login
> >>>>   raise SMTPException("SMTP AUTH extension not supported by server.")
> >>>> SMTPException: SMTP AUTH extension not supported by server.
> >>>>
> >>>> This was working on 1.7.1.2. Having a look now.
> >>>>
> >>>> On Thu, Jan 5, 2017 at 9:09 AM, Chris Riccomini <
> criccomini@apache.org>
> >>>> wrote:
> >>>>
> >>>>> Hey Robin,
> >>>>>
> >>>>> Awesome, thanks! I love open source. :) Let me try your patch out
and
> >>>>> merge it.
> >>>>>
> >>>>> Cheers,
> >>>>> Chris
> >>>>>
> >>>>> On Thu, Jan 5, 2017 at 1:34 AM, Miller, Robin <
> >>>>> Robin.Miller@affiliate.oliverwyman.com> wrote:
> >>>>>
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>>
> >>>>>> I think I ran into this issue when setting up LDAP Auth in our
> >>>>>> environment (we're using very close to master as we needed some
of
> the
> >>>>>> newer features/bugfixes). The problem turned out to be that
the
> search was
> >>>>>> finding no results, so the line:
> >>>>>>
> >>>>>>
> >>>>>> groups_list = [regex.search(i).group(1) for i in user_groups]
> >>>>>>
> >>>>>>
> >>>>>> would fail because it had no matching groups to return. This
turned
> out
> >>>>>> to be because Windows Active Directory (the LDAP server we're
using)
> >>>>>> returned capitals "CN=" where the code expected lowercase: regex
=
> >>>>>> re.compile("cn=([^,]*).*")
> >>>>>>
> >>>>>>
> >>>>>> I haven't looked it up, but Windows Active Directory is case
> >>>>>> insensitive when it comes to usernames and groups, so I wouldn't
be
> >>>>>> surprised if the protocol itself is case insensitive and both
"cn="
> and
> >>>>>> "CN=" should be considered valid. As such I've a PR open for
a
> simple fix
> >>>>>> to make this regex case insensitive: https://github.com/apache/incu
> >>>>>> bator-airflow/pull/1945
> >>>>>>
> >>>>>>
> >>>>>> Hopefully this helps,
> >>>>>>
> >>>>>> Robin Miller
> >>>>>> OLIVER WYMAN
> >>>>>> robin.miller@affiliate.oliverwyman.com<mailto:robin.miller@a
> >>>>>> ffiliate.oliverwyman.com>
> >>>>>> www.oliverwyman.com<http://www.oliverwyman.com/>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: Chris Riccomini <criccomini@apache.org>
> >>>>>> Sent: 05 January 2017 00:34:27
> >>>>>> To: dev@airflow.incubator.apache.org
> >>>>>> Subject: Re: Airflow 1.8.0 alpha 2
> >>>>>>
> >>>>>> I am now running 1.8.0a2 in our dev environment. It seems to
be
> >>>>>> functioning
> >>>>>> well.
> >>>>>>
> >>>>>> One issue we've hit is that the LDAP auth plugin isn't working
for
> us
> >>>>>> anymore:
> >>>>>>
> >>>>>> Traceback (most recent call last):
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line
1988,
> in
> >>>>>> wsgi_app
> >>>>>>  response = self.full_dispatch_request()
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line
1641,
> in
> >>>>>> full_dispatch_request
> >>>>>>  rv = self.handle_user_exception(e)
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line
1544,
> in
> >>>>>> handle_user_exception
> >>>>>>  reraise(exc_type, exc_value, tb)
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line
1639,
> in
> >>>>>> full_dispatch_request
> >>>>>>  rv = self.dispatch_request()
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line
1625,
> in
> >>>>>> dispatch_request
> >>>>>>  return self.view_functions[rule.endpoint](**req.view_args)
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask_admin/base.py",
line
> >>>>>> 69,
> >>>>>> in inner
> >>>>>>  return self._run_view(f, *args, **kwargs)
> >>>>>> File "/usr/lib64/python2.7/site-packages/flask_admin/base.py",
line
> >>>>>> 368,
> >>>>>> in _run_view
> >>>>>>  return fn(self, *args, **kwargs)
> >>>>>> File "/usr/lib/python2.7/site-packages/airflow/www/views.py",
line
> >>>>>> 657,
> >>>>>> in login
> >>>>>>  return airflow.login.login(self, request)
> >>>>>> File
> >>>>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe
> >>>>>> nds/ldap_auth.py",
> >>>>>> line 276, in login
> >>>>>>  flask_login.login_user(LdapUser(user))
> >>>>>> File "<string>", line 4, in __init__
> >>>>>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> >>>>>> line
> >>>>>> 306, in _initialize_instance
> >>>>>>  manager.dispatch.init_failure(self, args, kwargs)
> >>>>>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelp
> >>>>>> ers.py",
> >>>>>> line 60, in __exit__
> >>>>>>  compat.reraise(exc_type, exc_value, exc_tb)
> >>>>>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> >>>>>> line
> >>>>>> 303, in _initialize_instance
> >>>>>>  return manager.original_init(*mixed[1:], **kwargs)
> >>>>>> File
> >>>>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe
> >>>>>> nds/ldap_auth.py",
> >>>>>> line 148, in __init__
> >>>>>>  user.username)
> >>>>>> File
> >>>>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe
> >>>>>> nds/ldap_auth.py",
> >>>>>> line 106, in groups_user
> >>>>>>  groups_list = [regex.search(i).group(1) for i in user_groups]
> >>>>>> AttributeError: 'NoneType' object has no attribute 'group'
> >>>>>>
> >>>>>> I believe it's from this patch:
> >>>>>>
> >>>>>> https://github.com/apache/incubator-airflow/commit/d6d3f5367
> >>>>>> 3ba3736d7a858531823933cfef2bb4e
> >>>>>>
> >>>>>> I haven't dug into it yet. We'll need to fix it, though. In
the
> >>>>>> meantime, I
> >>>>>> just commented out the call to `groups_user`. More to come.
> >>>>>>
> >>>>>> On Wed, Jan 4, 2017 at 12:32 PM, Alex Van Boxel <alex@vanboxel.be>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I have another fix that certainly need to be in the final
release,
> >>>>>> but not
> >>>>>>> ready to merge due to failed tests:
> >>>>>>>
> >>>>>>> https://github.com/apache/incubator-airflow/pull/1961
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jan 4, 2017 at 8:48 PM Chris Riccomini <
> criccomini@apache.org
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Great, I will work toward deploying this in our dev
cluster today.
> >>>>>> :D
> >>>>>>>>
> >>>>>>>> On Wed, Jan 4, 2017 at 11:47 AM, Bolke de Bruin <
> bdbruin@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Some issues remain:
> >>>>>>>>>
> >>>>>>>>> * one_failed not executed as dag run is marked failed
seemingly
> >>>>>>>>> prematurely (@chris yes you should see this, see
below for an
> >>>>>> example
> >>>>>>>> that
> >>>>>>>>> is not working properly), confirmed regression
> >>>>>>>>> * celery instability Alex
> >>>>>>>>> * Wrong DAG state after failure inside branch
> >>>>>>>>>
> >>>>>>>>> So Alpha 2 is definitely not ready for production,
but please do
> >>>>>> put in
> >>>>>>>>> your canary dags and let them run. I am still quite
concerned
> >>>>>> about the
> >>>>>>>>> scheduler integrity and stability.
> >>>>>>>>>
> >>>>>>>>> - Bolke
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> one_failed_not_executed.py
> >>>>>>>>> ====
> >>>>>>>>> from airflow import DAG
> >>>>>>>>> from airflow.operators.bash_operator import BashOperator
> >>>>>>>>> from airflow.operators.dummy_operator import DummyOperator
> >>>>>>>>> from datetime import datetime, timedelta
> >>>>>>>>>
> >>>>>>>>> default_args = {
> >>>>>>>>>   'owner': 'airflow',
> >>>>>>>>>   'depends_on_past': False,
> >>>>>>>>>   'start_date': datetime(2016,10,5,19),
> >>>>>>>>>   'email': ['airflow@airflow.com'],
> >>>>>>>>>   'email_on_failure': False,
> >>>>>>>>>   'email_on_retry': False,
> >>>>>>>>>   'retries': 1,
> >>>>>>>>>   'retry_delay': timedelta(seconds=1),
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>> dag = DAG('tutorial', default_args=default_args,
> >>>>>>>> schedule_interval='@once')
> >>>>>>>>>
> >>>>>>>>> task1 = BashOperator(
> >>>>>>>>>   task_id='first_one',
> >>>>>>>>>   bash_command='date',
> >>>>>>>>>   dag=dag)
> >>>>>>>>>
> >>>>>>>>> task2 = BashOperator(
> >>>>>>>>>   task_id='second_one',
> >>>>>>>>>   bash_command='this_should_not_work',
> >>>>>>>>>   dag=dag)
> >>>>>>>>>
> >>>>>>>>> task2.set_upstream(task1)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> task3 = BashOperator(
> >>>>>>>>>   task_id='third_one',
> >>>>>>>>>   bash_command='random_command_third',
> >>>>>>>>>   dag=dag)
> >>>>>>>>>
> >>>>>>>>> task3.set_upstream(task2)
> >>>>>>>>>
> >>>>>>>>> fail_task = DummyOperator(
> >>>>>>>>>   task_id='one_failed',
> >>>>>>>>>   trigger_rule='one_failed',
> >>>>>>>>>   dag=dag)
> >>>>>>>>>
> >>>>>>>>> fail_task.set_upstream([task1,task2,task3])
> >>>>>>>>>
> >>>>>>>>>> On 4 Jan 2017, at 20:35, Chris Riccomini <criccomini@apache.org
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Bolke, can you describe the current state of
the alpha 2
> >>>>>> release? I
> >>>>>>> saw
> >>>>>>>>>> some comments from Alex yesterday about celery
instability. If
> >>>>>> I'm
> >>>>>>>>> running
> >>>>>>>>>> on LocalExecutor, should I be seeing any issues?
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jan 4, 2017 at 8:20 AM, Bolke de Bruin
<
> >>>>>> bdbruin@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi All,
> >>>>>>>>>>>
> >>>>>>>>>>> I have put up Airflow 1.8.0 alpha 2 in
> >>>>>> https://people.apache.org/~
> >>>>>>>>> bolke/ <
> >>>>>>>>>>> https://people.apache.org/~bolke/> .
> >>>>>>>>>>>
> >>>>>>>>>>> Note: This still cannot be considered an
Apache release.
> >>>>>> Working on
> >>>>>>>>> this.
> >>>>>>>>>>>
> >>>>>>>>>>> This build is signed (note it is served
over https).
> >>>>>>>>>>>
> >>>>>>>>>>> Changes are in the area of scheduler stability.
> >>>>>>>>>>>
> >>>>>>>>>>> - Bolke
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>> --
> >>>>>>> _/
> >>>>>>> _/ Alex Van Boxel
> >>>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> This e-mail and any attachments may be confidential or legally
> >>>>>> privileged. If you received this message in error or are not
the
> intended
> >>>>>> recipient, you should destroy the e-mail message and any
> attachments or
> >>>>>> copies, and you are prohibited from retaining, distributing,
> disclosing or
> >>>>>> using any information contained herein. Please inform us of
the
> erroneous
> >>>>>> delivery by return e-mail. Thank you for your cooperation.
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message