airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
Date Tue, 31 Dec 2019 07:16:00 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17005959#comment-17005959
] 

ASF GitHub Bot commented on AIRFLOW-6359:
-----------------------------------------

nuclearpinguin commented on pull request #6909: [AIRFLOW-6359] spark_submit_hook.py status
polling interval config
URL: https://github.com/apache/airflow/pull/6909
 
 
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> spark_submit_hook.py polling too frequent
> -----------------------------------------
>
>                 Key: AIRFLOW-6359
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: hooks
>    Affects Versions: 1.10.6
>            Reporter: t oo
>            Assignee: t oo
>            Priority: Major
>
> spark polling every second is too frequent, use config to change to desired interval,
ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval instead of
every second
>  
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>  
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message