mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Chen <>
Subject Re: Mesos 0.20.0 with Docker registry availability
Date Fri, 05 Sep 2014 18:27:20 GMT
Hi Tom,

It's definitely a tradeoff between able to automatically pick up latest
changes, and able to have more control of what your cluster is running.

The downside of just pulling latest with the latest is that you might not
know what's exactly running since someone might override latest and there
is no more versioning that you can rollback to.

I thought a good solution could be having the frameworks to have a way to
configure loading a default setting for image that it puts into DockerInfo
vs having it baked into the slave, then you're much more flexible of how
you want to deal with default images/tags and not need to rely on Mesos


On Fri, Sep 5, 2014 at 3:29 AM, Tom Arnfeld <> wrote:

> > You can tag each image with your commit hash that way Mesos will always
> have to do a docker pull and you don't lose the fast iteration cycle in
> development.
> I mentioned this on one of the review requests the other day. The problem
> here is that, say I want to iterate quickly on installing things for our
> Hadoop on Mesos cluster, I need to now change all the hadoop configuration
> on my Job Tracker's to point to the new image, which means a restart of the
> JT and jobs will die. This goes for pretty much every mesos framework that
> isn't for launching long running tasks.
> On 5 September 2014 11:14, Steve Domin <> wrote:
>> Hi Ryan,
>> You can tag each image with your commit hash that way Mesos will always
>> have to do a docker pull and you don't lose the fast iteration cycle in
>> development.
>> Steve
>> On Friday, September 5, 2014, craig mcmillan <>
>> wrote:
>>> hey ryan,
>>> there are two deployment use-cases i generally have :
>>> - production : i want to consider carefully what i deploy, and refer to
>>>                a specific image. a versioned tag works well here
>>> - development : i want to iterate quickly and something like a branch
>>>                 (movable tag) works really well here, à la heroku :
>>>                 git-push => commit-hook => build docker-image =>
>>> curl-to-marathon
>>> it's the development use-case that pull on every launch supports best
>>> would an option in the ContainerInfo to pull on every launch be
>>> reasonable ?
>>> i'm happy to do a PR if that would be helpful !
>>> :craig
>>> On 5 Sep 2014, at 9:07, Ryan Thomas wrote:
>>>  Whilst this is somewhat unrelated to the mesos implementation, I think
>>>> it
>>>> is generally good practice to have immutable tags on the images, this is
>>>> something I dislike about docker :)
>>>> Whist the gc of old images will eventually become a problem, it will
>>>> really
>>>> only be the layer delta that is consumed with each new tag. But I think
>>>> yes, there would need to be some mechanism to clear out the images in
>>>> the
>>>> local registry.
>>>> ryan
>>>> On 5 Sep 2014 18:03, "mccraig mccraig" <>
>>>> wrote:
>>>>  ah, so i will have to use a different tag to update an app
>>>>> one immediate problem i can see is that it makes garbage collecting old
>>>>> docker images from slaves harder : currently i update the image
>>>>> associated
>>>>> with a tag and restart tasks to update the running app, then
>>>>> occasionally a
>>>>> cron job to remove all docker images with no tag
>>>>> if every updated image has a new tag it will be harder to figure out
>>>>> which
>>>>> images to remove... perhaps any with no running container, though that
>>>>> could lead to unnecessary pulls and slower restarts of failed tasks
>>>>> :craig
>>>>> On 5 Sep 2014, at 08:43, Ryan Thomas <> wrote:
>>>>> Hey Craig,
>>>>> docker run will attempt a pull of the image if it cannot find a
>>>>> matching
>>>>> image and tag in its local repository.
>>>>> So it should only pull on the first run of a given tag.
>>>>> ryan
>>>>> On 5 Sep 2014 17:41, "mccraig mccraig" <>
>>>>> wrote:
>>>>>  hi tim,
>>>>>> if it doesn't pull on every run, when will it pull ?
>>>>>> :craig
>>>>>> On 5 Sep 2014, at 07:05, Tim Chen <> wrote:
>>>>>> Hi Maxime,
>>>>>> It is a very valid concern and that's why I've added a patch that
>>>>>> should
>>>>>> go out in 0.20.1 to not do a docker pull on every run anymore.
>>>>>> Mesos will still try to docker pull when the image isn't available
>>>>>> locally (via docker inspect), but only once.
>>>>>> The downside ofcourse is that you're not able to automatically get
>>>>>> latest tagged image, but I think it's worth while price to may to
>>>>>> gain the
>>>>>> benefits of not depending on registry, able to run local images and
>>>>>> more.
>>>>>> Tim
>>>>>> On Thu, Sep 4, 2014 at 10:50 PM, Maxime Brugidou <
>>>>>>> wrote:
>>>>>>  Hi,
>>>>>>> The current Docker integration in 0.20 does a "docker pull" from
>>>>>>> registry before running any task. This means that your entire
>>>>>>> cluster
>>>>>>> becomes unusable if the registry goes down.
>>>>>>> The docs allow you to configure a custom .dockercfg for your
tasks to
>>>>>>> point to a private docker registry.
>>>>>>> However it is not easy to run an HA docker registry. The
>>>>>>> docker-registry
>>>>>>> project recommend using S3 storage buy this is definitely not
>>>>>>> option for
>>>>>>> some people.
>>>>>>> I know that for regular artifacts, Mesos can use HDFS storage
and you
>>>>>>> can run your HDFS datanodes as Mesos tasks.
>>>>>>> So even if I attempt to have a docker registry storage in HDFS
>>>>>>> (which is
>>>>>>> not supported by docker-registry at the moment), I am stuck on
>>>>>>> chicken
>>>>>>> and egg problem. I want to have as little services outside of
>>>>>>> as
>>>>>>> possible and it is hard to maintain HA services (especially outside
>>>>>>> of
>>>>>>> Mesos).
>>>>>>> Is there anyone running Mesos with Docker in production without
>>>>>>> I am
>>>>>>> trying to make all the services outside of Mesos (the "infra"
>>>>>>> services that
>>>>>>> are necessary to run Mesos like DNS, Haproxy, Chef server...
>>>>>>> either HA
>>>>>>> or not critical for the cluster to run. The docker registry is
a new
>>>>>>> piece
>>>>>>> of infra outside of Mesos that is critical...
>>>>>>> Best,
>>>>>>> Maxime

View raw message