airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Imberman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-2714) Kubernetes Deployment Operator
Date Wed, 04 Jul 2018 15:44:00 GMT
Daniel Imberman created AIRFLOW-2714:
----------------------------------------

             Summary: Kubernetes Deployment Operator
                 Key: AIRFLOW-2714
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2714
             Project: Apache Airflow
          Issue Type: New Feature
            Reporter: Daniel Imberman
            Assignee: Daniel Imberman


 *{color:#212121}What?{color}*
{color:#212121} Add an operator that monitors a k8s deployment, declaring the task{color}
{color:#212121}complete on proper deployment/accessibility of endpoint{color}

*{color:#212121}Why?{color}*
{color:#212121} Not all tasks are single pods, sometimes you would want to run one task{color}
{color:#212121}that launches a service, and then a second task that smoke tests/stress{color}
{color:#212121}tests/{color}
{color:#212121} gives state to an application deployment. This would give airflow extra{color}
{color:#212121}functionality as a CI/CD tool in the k8s ecosystem.{color}

{color:#212121}*Fix*:{color}
{color:#212121} Create a modification (or extension) of the k8sPodOperator that can handle{color}
{color:#212121}entire deployments (possibly using the k8s model API to ensure{color}
{color:#212121}full flexibility of users).

An example of creating a deployment using the k8s model architecture can be
found here:
[https://github.com/kubernetes-client/python/blob/master/examples/deployment_examples.py]
{color}
{code:java}
def create_deployment_object():
    # Configureate Pod template container
    container = client.V1Container(
        name="nginx",
        image="nginx:1.7.9",
        ports=[client.V1ContainerPort(container_port=80)])
    # Create and configurate a spec section
    template = client.V1PodTemplateSpec(
        metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
        spec=client.V1PodSpec(containers=[container]))
    # Create the specification of deployment
    spec = client.ExtensionsV1beta1DeploymentSpec(
        replicas=3,
        template=template)
    # Instantiate the deployment object
    deployment = client.ExtensionsV1beta1Deployment(
        api_version="extensions/v1beta1",
        kind="Deployment",
        metadata=client.V1ObjectMeta(name=DEPLOYMENT_NAME),
        spec=spec)
{code}
{color:#212121}


return deployment

This would involve a more k8s knowledge from the user, but would have the
massive benefit that we would not have to maintain new features as the k8s
API updates (Would simply update version). A user would have to supply is a
deployment object and possibly a "success criteria" (i.e. an endpoint to
test).

Conversely, we could make the API a bit easier by only requiring a spec and
an optional metadata, after which we would handle a lot of the boilerplate.
{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message