airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <chetan.opensou...@gmail.com>
Subject Re: Deadlock at DAG SubDAG Execution at LocalExecutor
Date Sat, 04 May 2019 16:24:22 GMT
Thanks airflowuser for response.

Hello Bolke,

Do you any observations...

On Sat, May 4, 2019 at 1:01 AM Chetan Khatri <chetan.opensource@gmail.com>
wrote:

> What I am guessing is,
>
> 1) Increase the concurrency to 56 at airflow.cfg and don't overwrite
> attribute in DAG Constructor.
> 2) Change subdag's LocalExecutor to sequentialexecutor.
>
> On Sat, May 4, 2019 at 12:55 AM Chetan Khatri <chetan.opensource@gmail.com>
> wrote:
>
>> Hello Airflow Dev,
>>
>> I am using Airflow to schedule, orchestrate and monitor Data pipelines.
>> My airflow.cfg is default, I haven't change any attribute value yet.
>>
>> *Main Dag:*
>>
>> dag = DAG(
>>     dag_id=DAG_NAME,
>>     default_args=args,
>>     schedule_interval=None,
>>     concurrency=8
>> )
>>
>> I have 8 another sub-dags with similar setting:
>>
>> group_one_parquet = SubDagOperator(
>>     executor=LocalExecutor(),
>>     task_id='group_one_parquet',
>>     subdag=group_one_parquet_subdag(DAG_NAME, 'group_one_parquet' , args,
>> network_id,schema_name, env_name),
>>     default_args=args,
>>     dag=dag,
>> )
>>
>> Now when I am running this same DAG for let's say 5 times parallely with
>> different data points passed explicitly with **kwargs.
>>
>> I am getting an error as Deadlock!,
>> 1) The maximum number of running tasks (8) for this task's DAG 'xya' has
>> reached
>> 2) BackfillJob is deadlocked at log.
>>
>> [image: image (3).png]
>>
>> I would like to Scale LocalExecutor as vertically only because of certain
>> limitation.
>>
>> Can someone please throw light from experience.
>>
>> Thanks
>>
>>
>>
>>
>>
>>
>>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message