airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Imberman <daniel.imber...@gmail.com>
Subject Re: K8S deployment operator proposal
Date Mon, 09 Jul 2018 05:50:05 GMT
It's a really tough balancing act.

On the one hand I agree that ideally we don't want to force people to user
k8s's (ridiculously) verbose model. On the other hand the other solution is
to hand-craft requests like this
<https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py>
which
I think will be really hard to maintain in the long-term (and I actually
want to propose refactoring out in the next few days).

Perhaps what we can do is offer a "PodBuilder" and "DeploymentBuilder"
class that allows users to enter in basic info, gives them back an official
model to plug in to the operator, and if they want to customize it they
would have the option to take the V1Pod or the V1Deployment object and
customize as necessary. This would also future-proof both operators, as the
system would be much easier to update as k8s starts offering new features.

I also agree that a helm operator sounds like a pretty good idea, though no
reason these two efforts can't be done in parallel :)

On Thu, Jul 5, 2018 at 5:48 AM James Meickle
<jmeickle@quantopian.com.invalid> wrote:

> I think it's going to be an antipattern to write Python configuration in
> Airflow to configure a Kubernetes deployment since even a "simple"
> deployment would likely be more classes/objects than the DAG itself. I do
> like the idea of having a more featured operator than the PodOperator, but
> if I were to choose a second operator to implement, it would definitely be
> a Helm operator instead of a Kube API operator.
>
> On Tue, Jul 3, 2018 at 12:47 PM, Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> > An example of creating a deployment using the k8s model architecture can
> be
> > found here:
> > https://github.com/kubernetes-client/python/blob/master/
> > examples/deployment_examples.py
> >
> > def create_deployment_object():
> >     # Configureate Pod template container
> >     container = client.V1Container(
> >         name="nginx",
> >         image="nginx:1.7.9",
> >         ports=[client.V1ContainerPort(container_port=80)])
> >     # Create and configurate a spec section
> >     template = client.V1PodTemplateSpec(
> >         metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
> >         spec=client.V1PodSpec(containers=[container]))
> >     # Create the specification of deployment
> >     spec = client.ExtensionsV1beta1DeploymentSpec(
> >         replicas=3,
> >         template=template)
> >     # Instantiate the deployment object
> >     deployment = client.ExtensionsV1beta1Deployment(
> >         api_version="extensions/v1beta1",
> >         kind="Deployment",
> >         metadata=client.V1ObjectMeta(name=DEPLOYMENT_NAME),
> >         spec=spec)
> >
> > return deployment
> >
> > This would involve a more k8s knowledge from the user, but would have the
> > massive benefit that we would not have to maintain new features as the
> k8s
> > API updates (Would simply update version). A user would have to supply
> is a
> > deployment object and possibly a "success criteria" (i.e. an endpoint to
> > test).
> >
> > Conversely, we could make the API a bit easier by only requiring a spec
> and
> > an optional metadata, after which we would handle a lot of the
> boilerplate.
> >
> > On Tue, Jul 3, 2018 at 9:20 AM Daniel Imberman <
> daniel.imberman@gmail.com>
> > wrote:
> >
> > >  Hi all,
> > >
> > > Enclosed is a proposal for a kubernetes deployment management
> operator. I
> > > think this would be a good addition to current k8s offerings s.t. users
> > can
> > > actually launch persistent applications from airflow DAGs.
> > >
> > > * What?*
> > >  Add an operator that monitors a k8s deployment, declaring the task
> > > complete on proper deployment/accessibility of endpoint
> > >
> > > * Why?*
> > >  Not all tasks are single pods, sometimes you would want to run one
> task
> > > that launches a service, and then a second task that smoke tests/stress
> > > tests/
> > >  gives state to an application deployment. This would give airflow
> extra
> > > functionality as a CI/CD tool in the k8s ecosystem.
> > >
> > > * Fix:*
> > >  Create a modification (or extension) of the k8sPodOperator that can
> > > handle entire deployments (possibly using the k8s model API to ensure
> > > full flexibility of users).
> > >
> > >  Thank you.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message