airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <...@apache.org>
Subject Re: Best practises for debugging Macros or Plugins?
Date Wed, 17 Jul 2019 10:49:18 GMT
And as for ways you can get the macro in a way you debug it - the `airflow render` subcommand
would help here.

The other option would be to create a unit test file something like what I've put below.

(This just tests the default `{{ ds }}` macro, but this could be extended to test a custom
macro)

-ash

__unittest__
from datetime import datetime
from unittest import TestCase

from airflow import DAG
from airflow.models import DagRun, TaskInstance
from airflow.utils.db import create_session
from airflow.utils.timezone import make_aware
from airflow.utils.state import State
from airflow.operators.bash_operator import BashOperator


class MacroTest(TestCase):
    def setUp(self):
        self.dag = DAG(
            dag_id="unittest",
            start_date=datetime(2019, 1, 1)
        )

    def tearDown(self):
        with create_session() as session:
            session.query(DagRun).filter(DagRun.dag_id == self.dag.dag_id).delete()
            session.query(TaskInstance).filter(TaskInstance.dag_id == self.dag.dag_id).delete()

    def test_macro(self):
        # Use BashOperator as it's got easy macros to render
        op = BashOperator(
            task_id="test",
            dag=self.dag,
            bash_command='true',
            env={'KEY': '{{ ds }}'},
        )

        with create_session() as session:
            # Set up the object to get a full template context
            dr = self.dag.create_dagrun(
                run_id="test",
                execution_date=make_aware(datetime(2019, 1, 1)),
                state=State.SCHEDULED,
                session=session,
                )

            ti = dr.get_task_instance(op.task_id)
            ti.task = op

            ti.render_templates()

        self.assertEqual(op.env['KEY'], '2019-01-01')


> On 17 Jul 2019, at 11:07, Jarek Potiuk <Jarek.Potiuk@polidea.com> wrote:
> 
> The best way to debug is to use ipdb debugger or remote debugging feature
> from PyCharm/IntelliJ.
> 
> * For the first one ipdb - there is a note in upcoming Docker CI image
> environment:
> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests
> .
> You can set the ipdb trace method in any place of the python code you have
> and you should be able to debug from the console.
> 
> * For remote debugging it is a bit more difficult and requires paid version
> of PyCharm/IntelliJ - I have another PR that I am going to propose soon -
> the Breeze environment and you can find description on how to setup remote
> debugging here:
> https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst#debugging-airflow-breeze-tests-in-ide
> 
> 
> Additionally Pycharm/IntelliJ allow you to debug various templates
> (including Jinja templates). You can likely combine this with remote
> debugging but I have never used it this way):
> https://blog.jetbrains.com/pycharm/2017/06/template-debugging/
> 
> J.
> 
> On Tue, Jul 16, 2019 at 9:23 PM Shaw, Damian P. <
> damian.shaw.2@credit-suisse.com> wrote:
> 
>> Hi all,
>> 
>> I've just been working on updating many of my macros to include logic to
>> handle holiday calendars. There was a small mistake in one of the macros
>> and I found it very difficult to debug, I'm not sure where a default logger
>> in the plugins would log out to and there was no obvious way to run the
>> code outside of airflow or somewhere I could add a breakpoint.
>> 
>> My issue is solved now, but does anyone have an advise on how to debug
>> Macros or Plugins for future cases?
>> 
>> Thanks,
>> Damian
>> 
>> ===============================================================================
>> 
>> Please access the attached hyperlink for an important electronic
>> communications disclaimer:
>> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
>> ===============================================================================
>> 
>> 
> 
> 
> -- 
> 
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
> 
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>


Mime
View raw message