Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
MIME-Version: 1.0
From: Aaron Polhamus <aaron@credijusto.com>
Date: Fri, 28 Jul 2017 11:00:38 -0500
Message-ID: <CACP1=KmgfnAKdqP=MoxXnYCpy617OmSvKMERGT8RfCuO7saRww@mail.gmail.com>
Subject: Possible bug in bitshift operator?
To: dev@airflow.incubator.apache.org
Content-Type: multipart/alternative; boundary="f403043ecd5c494a6d055562c83f"
archived-at: Fri, 28 Jul 2017 16:00:51 -0000

--f403043ecd5c494a6d055562c83f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I've run into an issue using bitshift operators to define task dependencies
in my DAG (this question is here
<https://stackoverflow.com/questions/45362880/reversed-upstream-downstream-=
relationships-when-generating-multiple-tasks-in-air>
on StackOverflow as well).

I've pasted code in, below. What this looks like it *should* do is that
preproc_task will run first, followed by clear_tables. What it actually
*does* do, which is the behavior that I want to it have, is run
clear_tables first, followed by parallel execution of a bunch of
preproc_task operations defined by different user IDs.

The dependency relationship seems like it's reversed. I'm choosing between
proper syntax and the wrong behavior and improper syntax and the right
behavior. How can I debug this?

from datetime import datetimeimport osimport sys
from airflow.models import DAGfrom airflow.operators.python_operator
import PythonOperator
import ds_dependencies

SCRIPT_PATH =3D os.getenv('DASH_PREPROC_PATH')if SCRIPT_PATH:
    sys.path.insert(0, SCRIPT_PATH)
    import dash_workerselse:
    print('Define DASH_PREPROC_PATH value in environmental variables')
    sys.exit(1)

ENV =3D os.environ

default_args =3D {
  'start_date': datetime.now(),}

DAG =3D DAG(
  dag_id=3D'dash_preproc',
  default_args=3Ddefault_args)

clear_tables =3D PythonOperator(
  task_id=3D'clear_tables',
  python_callable=3Ddash_workers.clear_db,
  dag=3DDAG)
def id_worker(uid):
    return PythonOperator(
        task_id=3Did,
        python_callable=3Ddash_workers.main_preprocess,
        op_args=3D[uid],
        dag=3DDAG)
for uid in dash_workers.get_id_creds():
    preproc_task =3D id_worker(uid)
    clear_tables << preproc_task


--=20


*Aaron Polhamus*
*Director of Data Science *

Cel (M=C3=A9xico): +52 (55) 1951-5612
Cell (USA): +1 (206) 380-3948
Tel: +52 (55) 1168 9757 - Ext. 181

--=20
***Por favor referirse a nuestra p=C3=A1gina web=20
<https://www.credijusto.com/aviso-de-privacidad/> para m=C3=A1s informaci=
=C3=B3n=20
acerca de nuestras pol=C3=ADticas de privacidad.*


--f403043ecd5c494a6d055562c83f--