airflow-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li, Richard" <rl...@allstate.com>
Subject Re: Re: How to access external arguments when passing them using REST API in airflow?
Date Mon, 06 Jul 2020 18:20:47 GMT
Define this way in “val” as a variable reference and pass value from command line, for
example,
bash_command= “/path/scriptname” + keyname,



Richard Li
Big Data Engineer @ Product/Service Innovation Development
222 W. Merchandise Mart Plaza | Suite 850 | Chicago, Illinois 60654
6308632669


From: Ayush Goel <ayushgoel1594@gmail.com>
Date: Monday, July 6, 2020 at 12:59 PM
To: Richard Li <rliaa@allstate.com>
Cc: "users@airflow.apache.org" <users@airflow.apache.org>
Subject: [External] Re: How to access external arguments when passing them using REST API
in airflow?

 Hi Richard,

The arguments which we are passing are dynamic and the user does not have access to UI, so
he cannot change the variable values each time.

We want to give the rest api interface to the user through which he can trigger the dag by
himself while passing some user specific arguments.

Thanks and Regards,
Ayush Goel


On Mon, Jul 6, 2020 at 11:14 PM Li, Richard <rliaa@allstate.com<mailto:rliaa@allstate.com>>
wrote:
From “admin” menu, chose “variable”, input your “key”and “val” values. In
python scripts add
keyname = Variable.get("key"), use keyname in your dags


Richard Li
Big Data Engineer @ Product/Service Innovation Development
222 W. Merchandise Mart Plaza | Suite 850 | Chicago, Illinois 60654
6308632669


From: Ayush Goel <ayushgoel1594@gmail.com<mailto:ayushgoel1594@gmail.com>>
Reply-To: "users@airflow.apache.org<mailto:users@airflow.apache.org>" <users@airflow.apache.org<mailto:users@airflow.apache.org>>
Date: Monday, July 6, 2020 at 12:36 PM
To: "users@airflow.apache.org<mailto:users@airflow.apache.org>" <users@airflow.apache.org<mailto:users@airflow.apache.org>>
Subject: [External] How to access external arguments when passing them using REST API in airflow?

Hi,


 I tried triggering a airflow dag using the curl: http://localhost:8080/api/experimental/dags/api_test/dag_runs<https://urldefense.com/v3/__http:/localhost:8080/api/experimental/dags/api_test/dag_runs__;!!IIU9BLNPZ2ob!bzFlFU4TxDdurM-OBcl6Fpj1f_AHAfkulbMVFtUPEX2sxn3leOZOmRvP6U6aSw$>
-H 'Cache-Control: no-cache'
-H 'Content-Type: application/json'
-d '{"conf":"{"input_path":"value", "output_path":"value" }"}'

I am able to trigger the workflow successfully but now I want to access this input_path and
output_path inside my PythonOperator code in the airflow dag.

Can someone help in this regard?

My Dag Code:

from datetime import timedelta

import logging



# The DAG object; we'll need this to instantiate a DAG

from airflow import DAG

# Operators; we need this to operate!

from airflow.operators.python_operator import PythonOperator

from airflow.utils.dates import days_ago



# These args will get passed on to each operator

# You can override them on a per-task basis during operator initialization

default_args = {

    'owner': 'someone',

    'depends_on_past': False,

    'start_date': days_ago(2),

    'email': ['xyz@xyz.com<mailto:xyz@xyz.com>'],

    'email_on_failure': False,

    'email_on_retry': False,

    'retries': 1,

    'retry_delay': timedelta(minutes=2)

}

dag = DAG(

    'api_test',

    default_args=default_args,

    description='testing rest api calls',

    schedule_interval=None,

)

def run_this_func(**kwargs):

    logging.info("Trying to print the logs")

    logging.info(kwargs['conf'])

    for key,value in kwargs['conf'].items():

        logging.info(key)

        logging.info(value)

    #logging.info<https://urldefense.com/v3/__http:/logging.info__;!!IIU9BLNPZ2ob!bzFlFU4TxDdurM-OBcl6Fpj1f_AHAfkulbMVFtUPEX2sxn3leOZOmRtQNTdTJw$>(kwargs['dag_run'].conf['input_path'])

    #logging.info<https://urldefense.com/v3/__http:/logging.info__;!!IIU9BLNPZ2ob!bzFlFU4TxDdurM-OBcl6Fpj1f_AHAfkulbMVFtUPEX2sxn3leOZOmRtQNTdTJw$>(kwargs['conf']['output_path'])



run_this = PythonOperator(

    task_id='run_this',

    python_callable=run_this_func,

    dag=dag,

    provide_context=True,

)



run_this

I tried printing the arguments by logging.info<https://urldefense.com/v3/__http:/logging.info__;!!IIU9BLNPZ2ob!bzFlFU4TxDdurM-OBcl6Fpj1f_AHAfkulbMVFtUPEX2sxn3leOZOmRtQNTdTJw$>(kwargs['dag_run'].conf['input_path'])
logging.info<https://urldefense.com/v3/__http:/logging.info__;!!IIU9BLNPZ2ob!bzFlFU4TxDdurM-OBcl6Fpj1f_AHAfkulbMVFtUPEX2sxn3leOZOmRtQNTdTJw$>(kwargs['conf']['output_path'])

But both of these lines are giving me errors. Is there any way of accessing the input_path
and output_path?

Thanks and Regards,
Ayush Goel
Mime
View raw message