airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Imberman (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-2714) Kubernetes Deployment Operator
Date Wed, 04 Jul 2018 15:44:00 GMT
Daniel Imberman created AIRFLOW-2714:

             Summary: Kubernetes Deployment Operator
                 Key: AIRFLOW-2714
             Project: Apache Airflow
          Issue Type: New Feature
            Reporter: Daniel Imberman
            Assignee: Daniel Imberman

{color:#212121} Add an operator that monitors a k8s deployment, declaring the task{color}
{color:#212121}complete on proper deployment/accessibility of endpoint{color}

{color:#212121} Not all tasks are single pods, sometimes you would want to run one task{color}
{color:#212121}that launches a service, and then a second task that smoke tests/stress{color}
{color:#212121} gives state to an application deployment. This would give airflow extra{color}
{color:#212121}functionality as a CI/CD tool in the k8s ecosystem.{color}

{color:#212121} Create a modification (or extension) of the k8sPodOperator that can handle{color}
{color:#212121}entire deployments (possibly using the k8s model API to ensure{color}
{color:#212121}full flexibility of users).

An example of creating a deployment using the k8s model architecture can be
found here:
def create_deployment_object():
    # Configureate Pod template container
    container = client.V1Container(
    # Create and configurate a spec section
    template = client.V1PodTemplateSpec(
        metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
    # Create the specification of deployment
    spec = client.ExtensionsV1beta1DeploymentSpec(
    # Instantiate the deployment object
    deployment = client.ExtensionsV1beta1Deployment(

return deployment

This would involve a more k8s knowledge from the user, but would have the
massive benefit that we would not have to maintain new features as the k8s
API updates (Would simply update version). A user would have to supply is a
deployment object and possibly a "success criteria" (i.e. an endpoint to

Conversely, we could make the API a bit easier by only requiring a spec and
an optional metadata, after which we would handle a lot of the boilerplate.

This message was sent by Atlassian JIRA

View raw message