Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
User-Agent: Microsoft-MacOutlook/f.18.0.160707
Date: Tue, 17 Jan 2017 14:58:23 +0530
Subject: Re: Programatically generating DAGs
From: Arvind Nedumaran <arvindamirtaa@gmail.com>
To: <dev@airflow.incubator.apache.org>
Message-ID: <817D137C-6DAF-4EDD-AB74-5BAD959276B9@gmail.com>
Thread-Topic: Programatically generating DAGs
References: <CAF_6RoML8yXexkg=_360g=0Li9D9s+3XzCEzSbO0uVRrXfrYkg@mail.gmail.com>
 <CAKOr0jtjCN+YTBqkOzvjQPBO=VDPaTsTzS2NzrRkmNv5TLpCfw@mail.gmail.com>
In-Reply-To: <CAKOr0jtjCN+YTBqkOzvjQPBO=VDPaTsTzS2NzrRkmNv5TLpCfw@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="UTF-8"
Content-transfer-encoding: 7bit
archived-at: Tue, 17 Jan 2017 09:28:35 -0000

Hello Ludovic, 

Thank you so much. Things are a lot clearer now. Looks like there is a way to make this work for us. 

Have a nice day,
Arvind

On 16/01/17, 22:30, "Ludovic Claude" <ludovic.claude54@gmail.com> wrote:

    Hello Arvind,
    
    DAGs in Airflow are simple Python objects that only need to be registered
    in the globals namespace. It's easy to generate DAGs from some
    configuration or database, I'm using that for example to generate several
    image processing pipelines based on configuration.
    
    Here is an example:
    https://github.com/LREN-CHUV/airflow-mri-preprocessing-dags/blob/master/mri_pipelines_init.py
    
    Ludovic
    
    
    2017-01-16 12:28 GMT+01:00 Arvind Nedumaran <arvindamirtaa@gmail.com>:
    
    > Hello everyone,
    >
    > I'm new to Airflow and I'm evaluating it for implementation at our org. I
    > find that there are a lot of use cases where workflows are not known during
    > development. Is there a way to programatically generate, and manage DAGs.
    >
    > Better yet, is there a way to manage them directly from a DB instead of
    > generating them from templates and managing them as files. We're about to
    > spend considerable time implementing this, so I just want to know about the
    > various existing options.
    >
    > Does it help if the tasks are pre-defined and the DAGs only happen to be
    > some permutation of the pre-existing tasks?
    >
    > Anybody who's tried something like before, I'd be grateful for any input on
    > this issue.
    >
    > Thanks,
    > Arvind
    >