airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerard Toonstra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1147) airflow scheduler not working
Date Thu, 27 Apr 2017 09:34:04 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986270#comment-15986270
] 

Gerard Toonstra commented on AIRFLOW-1147:
------------------------------------------

The example uses an interval of 1440 minutes, or 24 hours. How long did you wait until you
come to the conclusion that the scheduler doesn't work?  

This was raised 2 days ago and the scheduler would only pick this up every 24 hours. 

To be able to test this, you could copy this dag, change the dag-id to something different
(you should never change intervals of dags that have been deployed/run), and then change the
interval to something like 2 minutes. With those settings, do you still see that the dag doesn't
get processed every 2 minutes?   What's the output of the scheduler in that case?

When you have a start_date of today, the first time the dag runs is on start_date+interval,
so the day after, because it obviously doesn't have any data prior to that.

Let us know!

> airflow scheduler not working
> -----------------------------
>
>                 Key: AIRFLOW-1147
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1147
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: Airflow 1.8
>         Environment: CentOS running on 128 GB ram
>            Reporter: Mubin Khalid
>            Priority: Critical
>              Labels: documentation, newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> I've created some `DAG`s, and I tried to put it on scheduler. I want to run all the tasks
in the DAG after exact 24 hours.
> I tried to do something like this.
> {code}
>     DEFAULT_ARGS        = {
>     'owner'           : 'mubin',
>     'depends_on_past' : False,
>     'start_date'      : datetime(2017, 4, 24, 14, 30),
>     'retries'         : 5,
>     'retry_delay'     : timedetla(1),
>     }
>     SCHEDULE_INTERVAL      = timedelta(minutes=1440)
>     # SCHEDULE_INTERVAL    = timedelta(hours=24)
>     # SCHEDULE_INTERVAL    = timedelta(days=1)
>     dag = DAG('StandardizeDataDag',
>         default_args       = DEFAULT_ARGS,
>         schedule_interval  = SCHEDULE_INTERVAL
>     )
>  {code}   
> I tried to put different intervals, but not any working. However if I try to reset db
 {code} airflow resetdb -y {code}  and then run  {code} airflow initdb {code} , it works for
once. then after that, scheduler isn't able to run it.
> PS.  {code} airflow scheduler {code}  executed from  {code} root {code} 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message