mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Brugidou <>
Subject Re: HDFS on Mesos
Date Thu, 26 Jun 2014 11:03:10 GMT
This has been discussed before apparently

I think that this topic will become more important now that external
containerization is out. The write-outside-sandbox pattern won't work in a
chroot or docker AFAIK.

In addition the docker pattern for persistent data storage is to use a
data-only docker image. Not sure if this is appropriate here.
On Jun 26, 2014 12:42 PM, "Maxime Brugidou" <>

> There is clearly a need for persistent storage management in Mesos from
> what I can observe.
> The current sandbox is what I consider ephemeral storage since it gets
> lost when task exits. It can recover after a slave failure using the
> recovery mechanism but for example it won't survive a slave reboot.
> Other frameworks I know of that seem to use or need persistent storage are
> Cassandra and Kafka. I wonder what has been done in the framework to
> survive a DC power outage for example. Is all data lost?
> As Vinod said if we want to implement persistent storage by ourselves we
> need to track the resource "manually" using attributes or zk. This "trick"
> will be reimplemented over and over by frameworks and will be outside
> Mesos' control (I don't even know if this trick is feasible with docker
> containerization).
> The proper way would be to have a persistent disk resource type (or
> something else equivalent) that let you keep data on disk. The resource
> will belong to a user/framework and we can have quotas. I have no idea how
> to implement that since I'm not familiar with the details but it could be
> using simple FS quotas and directories in the mesos directory itself (so we
> mutualize ephemeral and persistent storage), it could also be on the form
> of raw storage using LVM volumes to enable other sort of applications... Or
> it could be both actually, mesos could have a raw volume group to use for
> any sort of temporary/ephemeral and persistent volumes.
> This is probably very complex since you will need tools to report the
> storage usage and do some cleanup (or have a TTL/expiry mechanism). But I
> believe that every storage framework will reinvent this every time outside
> Mesos.
> On Jun 26, 2014 1:01 AM, "Vinod Kone" <> wrote:
>> Thanks for listing this out Adam.
>> Data Residency:
>>> - Should we destroy the sandbox/hdfs-data when shutting down a DN?
>>> - If starting DN on node that was previously running a DN, can/should we
>>> try to revive the existing data?
>> I think this is one of the key challenges for a production quality HDFS
>> on Mesos. Currently, since sandbox is deleted after a task exits, if all
>> the data nodes that hold a block (and its replicas) get lost/killed for
>> whatever reason there would be data loss. A short terms solution would be
>> to write outside sandbox and use slave attributes to track where to
>> re-launch data node tasks.

View raw message