mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Mathiske (JIRA)" <>
Subject [jira] [Updated] (MESOS-2057) Concurrency control for fetcher cache
Date Fri, 14 Nov 2014 15:49:33 GMT


Bernd Mathiske updated MESOS-2057:
    Summary: Concurrency control for fetcher cache  (was: Add cache functionality with concurrent
downloading to fetcher)

> Concurrency control for fetcher cache
> -------------------------------------
>                 Key: MESOS-2057
>                 URL:
>             Project: Mesos
>          Issue Type: Improvement
>          Components: fetcher, slave
>            Reporter: Bernd Mathiske
>            Assignee: Bernd Mathiske
> Having added a URI flag to CommandInfo messages (in MESOS-2069) that indicates caching,
caching files downloaded by the fetcher in a repository, now ensure that when a URI is "cached",
it is only ever downloaded once for the same user on the same slave as long as the slave keeps
> This even holds if multiple tasks request the same URI concurrently. If multiple requests
for the same URI occur, perform only one of them and reuse the result. Make concurrent requests
for the same URI wait for the one download. 
> Different URIs from different CommandInfos can be downloaded concurrently.
> No cache eviction, cleanup or failover will be handled for now. Additional tickets will
be filed for these enhancements. (So don't use this feature in production until the whole
epic is complete.)
> Note that implementing this does not suffice for production use. This ticket contains
the main part of the fetcher logic, though. See the epic MESOS-336 for the rest of the features
that lead to a fully functional fetcher cache.
> The proposed general approach is to keep all bookkeeping about what is in which stage
of being fetched and where it resides in the slave's MesosContainerizerProcess, so that all
concurrent access is disambiguated and controlled by an "actor" (aka libprocess "process").
> Depends on MESOS-2056 and MESOS-2069.

This message was sent by Atlassian JIRA

View raw message