airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Best practises for debugging Macros or Plugins?
Date Wed, 17 Jul 2019 11:47:38 GMT
Ah cool Ash. I Did not know about the 'render' command :).

J.

On Wed, Jul 17, 2019 at 12:49 PM Ash Berlin-Taylor <ash@apache.org> wrote:

> And as for ways you can get the macro in a way you debug it - the `airflow
> render` subcommand would help here.
>
> The other option would be to create a unit test file something like what
> I've put below.
>
> (This just tests the default `{{ ds }}` macro, but this could be extended
> to test a custom macro)
>
> -ash
>
> __unittest__
> from datetime import datetime
> from unittest import TestCase
>
> from airflow import DAG
> from airflow.models import DagRun, TaskInstance
> from airflow.utils.db import create_session
> from airflow.utils.timezone import make_aware
> from airflow.utils.state import State
> from airflow.operators.bash_operator import BashOperator
>
>
> class MacroTest(TestCase):
>     def setUp(self):
>         self.dag = DAG(
>             dag_id="unittest",
>             start_date=datetime(2019, 1, 1)
>         )
>
>     def tearDown(self):
>         with create_session() as session:
>             session.query(DagRun).filter(DagRun.dag_id ==
> self.dag.dag_id).delete()
>             session.query(TaskInstance).filter(TaskInstance.dag_id ==
> self.dag.dag_id).delete()
>
>     def test_macro(self):
>         # Use BashOperator as it's got easy macros to render
>         op = BashOperator(
>             task_id="test",
>             dag=self.dag,
>             bash_command='true',
>             env={'KEY': '{{ ds }}'},
>         )
>
>         with create_session() as session:
>             # Set up the object to get a full template context
>             dr = self.dag.create_dagrun(
>                 run_id="test",
>                 execution_date=make_aware(datetime(2019, 1, 1)),
>                 state=State.SCHEDULED,
>                 session=session,
>                 )
>
>             ti = dr.get_task_instance(op.task_id)
>             ti.task = op
>
>             ti.render_templates()
>
>         self.assertEqual(op.env['KEY'], '2019-01-01')
>
>
> > On 17 Jul 2019, at 11:07, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:
> >
> > The best way to debug is to use ipdb debugger or remote debugging feature
> > from PyCharm/IntelliJ.
> >
> > * For the first one ipdb - there is a note in upcoming Docker CI image
> > environment:
> >
> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests
> > .
> > You can set the ipdb trace method in any place of the python code you
> have
> > and you should be able to debug from the console.
> >
> > * For remote debugging it is a bit more difficult and requires paid
> version
> > of PyCharm/IntelliJ - I have another PR that I am going to propose soon -
> > the Breeze environment and you can find description on how to setup
> remote
> > debugging here:
> >
> https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst#debugging-airflow-breeze-tests-in-ide
> >
> >
> > Additionally Pycharm/IntelliJ allow you to debug various templates
> > (including Jinja templates). You can likely combine this with remote
> > debugging but I have never used it this way):
> > https://blog.jetbrains.com/pycharm/2017/06/template-debugging/
> >
> > J.
> >
> > On Tue, Jul 16, 2019 at 9:23 PM Shaw, Damian P. <
> > damian.shaw.2@credit-suisse.com> wrote:
> >
> >> Hi all,
> >>
> >> I've just been working on updating many of my macros to include logic to
> >> handle holiday calendars. There was a small mistake in one of the macros
> >> and I found it very difficult to debug, I'm not sure where a default
> logger
> >> in the plugins would log out to and there was no obvious way to run the
> >> code outside of airflow or somewhere I could add a breakpoint.
> >>
> >> My issue is solved now, but does anyone have an advise on how to debug
> >> Macros or Plugins for future cases?
> >>
> >> Thanks,
> >> Damian
> >>
> >>
> ===============================================================================
> >>
> >> Please access the attached hyperlink for an important electronic
> >> communications disclaimer:
> >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> >>
> ===============================================================================
> >>
> >>
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message