airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jack (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-2975) Add support for Amazon cloudwatch (computing power at will)
Date Wed, 29 Aug 2018 07:18:00 GMT
jack created AIRFLOW-2975:
-----------------------------

             Summary: Add support for Amazon cloudwatch (computing power at will)
                 Key: AIRFLOW-2975
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2975
             Project: Apache Airflow
          Issue Type: New Feature
    Affects Versions: 1.10.0
            Reporter: jack
             Fix For: 2.0.0, 1.10.1


Some of have one machine that runs airfow…

While we can scale up the executor to have many resources over different servers this is consider
to be expensive.

There is another solution.. using Amazon Cloudwatch:

[https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/]

 

This enable the user to create and close EC2 machines on specific intervals for specific tasks.

Basically if I have 50 DAGS to run on 1PM-3PM and few dags to run on other hours there is
no point in paying for 2nd server 24/7. 

This could be an enhancement to one of the executors that know to work with more than one
server.

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message