ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-4426) Hadoop: allow optional classloader reuse between tasks.
Date Wed, 14 Dec 2016 10:40:58 GMT
Vladimir Ozerov created IGNITE-4426:
---------------------------------------

             Summary: Hadoop: allow optional classloader reuse between tasks.
                 Key: IGNITE-4426
                 URL: https://issues.apache.org/jira/browse/IGNITE-4426
             Project: Ignite
          Issue Type: Sub-task
          Components: hadoop
    Affects Versions: 1.8
            Reporter: Vladimir Ozerov
            Assignee: Vladimir Ozerov
             Fix For: 2.0


*Problem*
Currently we create separate {{HadoopClassLoader}} for each and every MR task. It has serious
performance implications:
1) Every task have to reload classes form disk;
2) Every task starts from "cold" state with not-compiled classes;
3) Very high permgen/metaspace and code cache regions usage.

We can observe, that every task classloader works with the same JARs. It means that the only
case when we really need to isolate tasks through classloaders is when they depend on some
static data, which should not be shared.

*Proposal*
- Share {{HadoopClassLoader}} between tasks by default;
- Allow to optionally disable this opto.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message