aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "benley@gmail.com" <ben...@gmail.com>
Subject Re: Parameterize each Job Instance.
Date Tue, 12 Jan 2016 05:07:40 GMT
As a starting point, you might be able to cook up something involving
{{mesos.instance}} as a lookup key to a pystachio list.  You do have a
unique integer task number per instance to work with.

cf.
http://aurora.apache.org/documentation/latest/configuration-reference/#template-namespaces

On Mon, Jan 11, 2016 at 8:05 PM Bill Farner <wfarner@apache.org> wrote:

> I agree that this appears necessary when parameters are needed to define
> the runtime environment of the task (in this case, setting up the docker
> container).
>
> What's particularly interesting here is that this would call for the
> scheduler to fill in the parameter values prior to launching each task.
> Using pystachio variables for this is certainly the most natural in the
> DSL, but becomes a paradigm shift since the scheduler is currently ignorant
> of pystachio.
>
> Possibly only worth mentioning for shock value, but in the DSL this starts
> to look like lambdas pretty quickly.
>
> On Mon, Jan 11, 2016 at 7:46 PM, Mauricio Garavaglia <
> mauriciogaravaglia@gmail.com> wrote:
>
> > Hi guys,
> >
> > We are using the docker rbd volume plugin
> > <
> https://ceph.com/planet/getting-started-with-the-docker-rbd-volume-plugin>
> > to
> > provide persistent storage to the aurora jobs that runs in the
> containers.
> > Something like:
> >
> > p = [Parameter(name='volume', value='my-ceph-volume:/foo'), ...]
> > jobs = [ Service(..., container = Container(docker = Docker(...,
> parameters
> > = p)))]
> >
> > But in the case of jobs with multiple instances it's required to start
> each
> > container using different volumes, in our case different ceph images.
> This
> > could be achieved by deploying, for example, 10 instances and then update
> > each one independently to use the appropiate volume. Of course this is
> > quite inconvenient, error prone, and adds a lot of logic and state
> outside
> > aurora.
> >
> > We where thinking if it would make sense to have a way to parameterize
> the
> > task instances, in a similar way that is done with portmapping for
> example.
> > In the job definition have something like
> >
> > params = [
> >   Parameter( name='volume',
> > value='service-{{instanceParameters.volume}}:/foo' )
> > ]
> > ...
> > jobs = [
> >   Service(
> >     name = 'logstash',
> >     ...
> >     instanceParameters = { "volume" : ["foo", "bar", "zaa"]},
> >     instances = 3,
> >     container = Container(
> >       docker = Docker(
> >         image = 'image',
> >         parameters = params
> >       )
> >     )
> >   )
> > ]
> >
> >
> > Something like that, it would create 3 instances of the tasks, each one
> > running in a container that uses the volumes foo, bar, and zaa.
> >
> > Does it make sense? I'd be glad to work on it but I want to validate the
> > idea with you first and hear comments about the api/implementation.
> >
> > Thanks
> >
> >
> > Mauricio
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message