hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Trezzo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1492) truly shared cache for jars (jobjar/libjar)
Date Wed, 10 Sep 2014 23:23:37 GMT

    [ https://issues.apache.org/jira/browse/YARN-1492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129342#comment-14129342

Chris Trezzo commented on YARN-1492:

Thanks [~kasha]! A couple of questions:

bq. 2. The choice of SCM store should be transparent to the rest of SCM code. It would be
better to define an interface for the SCMStore similar to the RMStateStore today.

To clarify the above point. An interface does exist in the current implementation (see SCMStore.java
in YARN-2180), and all SCMStore implementations should be based off of that. Unfortunately
some implementation details from the in-memory store have leaked through via the SCMContext
object. I am working on an update to improve the interface so that an SCMContext object is
no longer needed and all implementation details are hidden behind SCMStore.java. Does your
above point mean that you are looking for a state machine-based interface like RMStateStore,
or do you see additional issues with the SCMStore interface outside of the SCMContext fix?

bq. 3. Defaulting to the in-memory store requires providing a way to initialize the store
with currently running applications and cached jars, which is quite involved and not so elegant
either. I propose implementing leveldb and zk stores. We could default to leveldb on non-HA
clusters, and ZK store for HA clusters if we choose to embed the SCM in the RM.

Do you see the leveldb and zk stores as blockers to merging into trunk/2.6 or would an in-memory
store with the interface fixes mentioned above be enough initially? Leveldb and ZK stores
could be easily added post-merge in an incremental way as additional SCMStore implementations.

> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>                 Key: YARN-1492
>                 URL: https://issues.apache.org/jira/browse/YARN-1492
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sangjin Lee
>            Assignee: Chris Trezzo
>         Attachments: YARN-1492-all-trunk-v1.patch, YARN-1492-all-trunk-v2.patch, YARN-1492-all-trunk-v3.patch,
YARN-1492-all-trunk-v4.patch, YARN-1492-all-trunk-v5.patch, shared_cache_design.pdf, shared_cache_design_v2.pdf,
shared_cache_design_v3.pdf, shared_cache_design_v4.pdf, shared_cache_design_v5.pdf, shared_cache_design_v6.pdf
> Currently there is the distributed cache that enables you to cache jars and files so
that attempts from the same job can reuse them. However, sharing is limited with the distributed
cache because it is normally on a per-job basis. On a large cluster, sometimes copying of
jobjars and libjars becomes so prevalent that it consumes a large portion of the network bandwidth,
not to speak of defeating the purpose of "bringing compute to where data is". This is wasteful
because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared cache so that
multiple jobs from multiple users can share and cache jars. This JIRA is to open the discussion.

This message was sent by Atlassian JIRA

View raw message