airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefano Baghino <stefano.bagh...@teralytics.ch>
Subject Re: Per-task resources with Mesos
Date Thu, 17 Aug 2017 07:17:06 GMT
Hi everybody,

I'd like to come back to this to give you an update on how we've made the
thing work. And of course, thanks for your kind responses.

We found out about Eremetic, a Mesos framework to run one-off jobs in
Docker containers (let's say it's a one-off jobs counterpart to Marathon,
in some respect).

I've written a small Python wrapper around their HTTP API and then an
EremeticOperator that uses this client (POSTs a job creation request and,
if successful, polls from the task status). We then use the LocalExecutor
for the whole thing. It's still a small toy and think there's a lot to be
improved to avoid undesirable situations in terms of handling partial
failures, but so far it works.

Here is an excerpt from a small test pipeline I wrote to give you a sense
of how the whole thing looks like:

dag = DAG(
    dag_id='some_dag',
    start_date=datetime(2017, 7, 1),
    schedule_interval=timedelta(days=1),
    default_args={
        'eremetic_url': 'https://url_to_eremetic/',
        'image': 'some_registry/some_image_with_spark',
        'volumes': [
            {
                'container_path': '/path/to/relevant/container/stuff',
                'host_path': '/path/to/relevant/host/stuff'
            }
        ]
    })

...

some_spark_job = EremeticOperator(
    task_id='some_spark_job',
    command='/path/to/relevant/container/stuff/bin/spark-submit run-example
SparkPi 10',
    cpu=4,
    mem=4096,
    dag=dag)

On Fri, Aug 4, 2017 at 11:48 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> At Airbnb using the Celery executor we use queues to wire tasks to machines
> provisioned in specific ways and we use the cgroup feature to constrain
> resource utilization as we fire up tasks. That requires running the worker
> service as root as its a requirement to impersonate and use cgroups.
>
> In the context of Mesos things may be different as you may want to do that
> on a different layer. I'd read through the MesosExecutor to see if it does
> any of this already, or to figure out where you may be able to hook things
> up.
>
> Note that (from memory) the MesosExecutor relies on pickling to get
> serialized DAGs [through the database] to Mesos slots, and that chances are
> high that we may deprecate that feature in the future. By that time we'll
> probably have a "DagFetcher" abstraction, allowing to get the DAG
> definition in another way on the fly.
>
> Max
>
> On Thu, Aug 3, 2017 at 10:24 AM, Victor Monteiro <victor.monteiro@ubee.in>
> wrote:
>
> > Hi Stefano, have you read about queues? Airflow has this concept and I
> > think you can decide for which queue a task should go. By doing this and
> > integrating it with mesos, I believe you can make a mesos cluster with
> more
> > resources to get tasks from a certain queue specific for heavy
> > computations.
> >
> > Maybe this can solve your problem (not sure) :D
> >
> > 2017-08-03 4:34 GMT-03:00 Stefano Baghino <stefano.baghino@teralytics.ch
> >:
> >
> > > Hi everyone,
> > >
> > > I'm investigating the possibility for our organization to use Airflow
> for
> > > workflow management.
> > >
> > > Some requirements on our side regard resource management, and in
> > particular
> > > the possibility for the system to run tasks on top of Apache Mesos.
> > Airflow
> > > partially satisfies our requirements in that regard, meaning that after
> > > having a look at the docs and code, it appears to me (correct me if I'm
> > > wrong) that resources are determined for the whole system (via
> > > configuration) and cannot be determined on a per-task basis. We'd need
> > this
> > > because some of our jobs are quite lightweight while others may
> require a
> > > lot of resources, making it a "one-size-fits-all" configuration quite
> > > wasteful.
> > >
> > > I had a look at the AirflowMesosScheduler and MesosExecutor and thought
> > it
> > > would be nice to add this feature and perhaps I can add it myself.
> What I
> > > would need is some guidance on how to make this fit into the overall
> > system
> > > design: is there an established way to explicitly ask for resources
> for a
> > > specific task in the DAG? If not, what could be a possible way to
> > introduce
> > > it? And if this reveals itself to be outside of the scope of Airflow,
> how
> > > do you think I can make it meet our requirement?
> > >
> > > Thanks in advance.
> > >
> > > P.S.: if by any chance some of you are on the Mesos mailing list as
> well,
> > > you may know that I'm having issues in making Airflow run successfully
> > > using Mesos due to missing Python packages. I'm not sure whether this
> > > mailing list is an appropriate place for users to get help. If so, I
> > could
> > > probably share that post here as well. Thanks!
> > >
> > > --
> > > Stefano Baghino | TERALYTICS
> > > *software engineer*
> > >
> > > Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> > > phone: +41 43 508 24 57
> > > email: stefano.baghino@teralytics.ch
> > > www.teralytics.net
> > >
> > > Company registration number: CH-020.3.037.709-7 | Trade register Canton
> > > Zurich
> > > Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
> > Yann
> > > de Vries
> > >
> > > This e-mail message contains confidential information which is for the
> > sole
> > > attention and use of the intended recipient. Please notify us at once
> if
> > > you think that it may not be intended for you and delete it
> immediately.
> > >
> >
>



-- 
Stefano Baghino | TERALYTICS
*software engineer*

Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
phone: +41 43 508 24 57
email: stefano.baghino@teralytics.ch
www.teralytics.net

Company registration number: CH-020.3.037.709-7 | Trade register Canton
Zurich
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
de Vries

This e-mail message contains confidential information which is for the sole
attention and use of the intended recipient. Please notify us at once if
you think that it may not be intended for you and delete it immediately.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message