hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Trezzo (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1492) truly shared cache for jars (jobjar/libjar)
Date Mon, 16 Jun 2014 18:02:02 GMT

     [ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Trezzo updated YARN-1492:
-------------------------------

    Attachment: YARN-1492-all-trunk-v1.patch

Attached is an all inclusive trunk patch for the shared cache code. I will break it up into
smaller more reviewable patches and attach them to subtasks. There are three major components
to the yarn shared cache (these are in-line with the posted design document):
1. The shared cache manager - this is a new service responsible for maintaining the shared
cache metadata (i.e. what is in the cache and which applications are currently using each
entry). The manager currently leverages an in-memory store. In the case of a manager restart,
the cache entries are recreated from HDFS.
2. Client side changes - These include changes at the yarn and MapReduce layers. There is
a new yarn shared cache api and changes to the MapReduce JobSubmitter to leverage the new
api.
3. Upload Service - There is a new service in the NodeManager that is responsible for uploading
resources to the shared cache.

> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>
>                 Key: YARN-1492
>                 URL: https://issues.apache.org/jira/browse/YARN-1492
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-1492-all-trunk-v1.patch, shared_cache_design.pdf, shared_cache_design_v2.pdf,
shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, shared_cache_design_v5.pdf
>
>
> Currently there is the distributed cache that enables you to cache jars and files so
that attempts from the same job can reuse them. However, sharing is limited with the distributed
cache because it is normally on a per-job basis. On a large cluster, sometimes copying of
jobjars and libjars becomes so prevalent that it consumes a large portion of the network bandwidth,
not to speak of defeating the purpose of "bringing compute to where data is". This is wasteful
because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared cache so that
multiple jobs from multiple users can share and cache jars. This JIRA is to open the discussion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message