mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharma Podila <spod...@netflix.com>
Subject Re: implementing data locality via mesos resource offers
Date Fri, 16 Jan 2015 17:34:15 GMT
Using the attributes would be the simplest way, if the slave were to
support dynamic updates of the attributes. The JIRA that Tim references
would be nice! Otherwise one would have to resort to something like a
wrapper script of the mesos-slave process that detects new data
availability and restarts mesos-slave with new attributes in cmdline.
Restarts may be OK when slaves are run to checkpoint state and recover
state upon restart.

Another possibility in the interim would be for the framework scheduler to
launch the task that does the download of the file(s) to the small subset
of nodes. Then, the scheduler can maintain this state information and
assign the tasks based on that. This has the additional advantage of
maintaining the list of that subset of nodes in a more dynamic way, if that
is useful to you.

In general, I am a fan of achieving data locality via the scheduler's state
info. In a more generic scenario, the data would be created dynamically by
tasks previously run (instead of just an initial download) and therefore
locality for such data is easier done via the scheduler.



On Fri, Jan 16, 2015 at 12:15 AM, Tim Chen <tim@mesosphere.io> wrote:

> Hi Douglas,
>
> The simplest way that Mesos can support is to add attributes via cli flags
> when you launch a mesos slave. And when this slave's resources is being
> offered, it will also include all the attributes you've tagged.
>
> This currently is static information on launch, and I believe there is
> JIRA tickets to make this dynamic (updatable at runtime).
>
> Tim
>
> On Thu, Jan 15, 2015 at 7:23 PM, Douglas Voet <dvoet@broadinstitute.org>
> wrote:
>
>> Hello,
>>
>> I am evaluating mesos in the context of running analyses of many large
>> files. I only want to download a file to a small subset of my nodes and
>> route the related processing there. The mesos paper talks about using
>> resource offers as a mechanism to achieve data locality but I can't find
>> any reference to how one might do this in the documentation. How would a
>> mesos slave know what data is available keeping in mind that that might
>> change over time? How can I configure a slave to include this information
>> in resource offers?
>>
>> Thanks in advance for any pointers.
>>
>> -Doug
>>
>
>

Mime
View raw message