mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Chen <...@mesosphere.io>
Subject Re: implementing data locality via mesos resource offers
Date Fri, 16 Jan 2015 17:53:31 GMT
Hi Sharma,

You're correct and that's how most schedulers handle this, which is to
handle the locality information itself.

We've considering and finding primitives to help in this front though, so
if you have any input let us know how to help manage locality that fits at
the level of Mesos.

Tim

On Fri, Jan 16, 2015 at 9:34 AM, Sharma Podila <spodila@netflix.com> wrote:

> Using the attributes would be the simplest way, if the slave were to
> support dynamic updates of the attributes. The JIRA that Tim references
> would be nice! Otherwise one would have to resort to something like a
> wrapper script of the mesos-slave process that detects new data
> availability and restarts mesos-slave with new attributes in cmdline.
> Restarts may be OK when slaves are run to checkpoint state and recover
> state upon restart.
>
> Another possibility in the interim would be for the framework scheduler to
> launch the task that does the download of the file(s) to the small subset
> of nodes. Then, the scheduler can maintain this state information and
> assign the tasks based on that. This has the additional advantage of
> maintaining the list of that subset of nodes in a more dynamic way, if that
> is useful to you.
>
> In general, I am a fan of achieving data locality via the scheduler's
> state info. In a more generic scenario, the data would be created
> dynamically by tasks previously run (instead of just an initial download)
> and therefore locality for such data is easier done via the scheduler.
>
>
>
> On Fri, Jan 16, 2015 at 12:15 AM, Tim Chen <tim@mesosphere.io> wrote:
>
>> Hi Douglas,
>>
>> The simplest way that Mesos can support is to add attributes via cli
>> flags when you launch a mesos slave. And when this slave's resources is
>> being offered, it will also include all the attributes you've tagged.
>>
>> This currently is static information on launch, and I believe there is
>> JIRA tickets to make this dynamic (updatable at runtime).
>>
>> Tim
>>
>> On Thu, Jan 15, 2015 at 7:23 PM, Douglas Voet <dvoet@broadinstitute.org>
>> wrote:
>>
>>> Hello,
>>>
>>> I am evaluating mesos in the context of running analyses of many large
>>> files. I only want to download a file to a small subset of my nodes and
>>> route the related processing there. The mesos paper talks about using
>>> resource offers as a mechanism to achieve data locality but I can't find
>>> any reference to how one might do this in the documentation. How would a
>>> mesos slave know what data is available keeping in mind that that might
>>> change over time? How can I configure a slave to include this information
>>> in resource offers?
>>>
>>> Thanks in advance for any pointers.
>>>
>>> -Doug
>>>
>>
>>
>

Mime
View raw message