aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Farner <wfar...@apache.org>
Subject Re: Parameterize each Job Instance.
Date Tue, 12 Jan 2016 04:05:15 GMT
I agree that this appears necessary when parameters are needed to define
the runtime environment of the task (in this case, setting up the docker
container).

What's particularly interesting here is that this would call for the
scheduler to fill in the parameter values prior to launching each task.
Using pystachio variables for this is certainly the most natural in the
DSL, but becomes a paradigm shift since the scheduler is currently ignorant
of pystachio.

Possibly only worth mentioning for shock value, but in the DSL this starts
to look like lambdas pretty quickly.

On Mon, Jan 11, 2016 at 7:46 PM, Mauricio Garavaglia <
mauriciogaravaglia@gmail.com> wrote:

> Hi guys,
>
> We are using the docker rbd volume plugin
> <https://ceph.com/planet/getting-started-with-the-docker-rbd-volume-plugin>
> to
> provide persistent storage to the aurora jobs that runs in the containers.
> Something like:
>
> p = [Parameter(name='volume', value='my-ceph-volume:/foo'), ...]
> jobs = [ Service(..., container = Container(docker = Docker(..., parameters
> = p)))]
>
> But in the case of jobs with multiple instances it's required to start each
> container using different volumes, in our case different ceph images. This
> could be achieved by deploying, for example, 10 instances and then update
> each one independently to use the appropiate volume. Of course this is
> quite inconvenient, error prone, and adds a lot of logic and state outside
> aurora.
>
> We where thinking if it would make sense to have a way to parameterize the
> task instances, in a similar way that is done with portmapping for example.
> In the job definition have something like
>
> params = [
>   Parameter( name='volume',
> value='service-{{instanceParameters.volume}}:/foo' )
> ]
> ...
> jobs = [
>   Service(
>     name = 'logstash',
>     ...
>     instanceParameters = { "volume" : ["foo", "bar", "zaa"]},
>     instances = 3,
>     container = Container(
>       docker = Docker(
>         image = 'image',
>         parameters = params
>       )
>     )
>   )
> ]
>
>
> Something like that, it would create 3 instances of the tasks, each one
> running in a container that uses the volumes foo, bar, and zaa.
>
> Does it make sense? I'd be glad to work on it but I want to validate the
> idea with you first and hear comments about the api/implementation.
>
> Thanks
>
>
> Mauricio
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message