hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5605) Memory-centric MapReduce aiming to solve the I/O bottleneck
Date Wed, 23 Apr 2014 14:19:25 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ming Chen updated MAPREDUCE-5605:
---------------------------------

    Attachment: TR-mammoth-HUST.pdf

The design, evaluation and discoveries are included in the paper.

> Memory-centric MapReduce aiming to solve the I/O bottleneck
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-5605
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5605
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 1.0.1
>         Environment: x86-64 Linux/Unix
> 64-bit jdk7 preferred
>            Reporter: Ming Chen
>            Assignee: Ming Chen
>             Fix For: 1.0.1
>
>         Attachments: MAPREDUCE-5605-v1.patch, TR-mammoth-HUST.pdf, hadoop-core-1.0.1-mammoth-0.9.0.jar
>
>
> Memory is a very important resource to bridge the gap between CPUs and I/O devices. So
the idea is to maximize the usage of memory to solve the problem of I/O bottleneck. We developed
a multi-threaded task execution engine, which runs in a single JVM on a node. In the execution
engine, we have implemented the algorithm of memory scheduling to realize global memory management,
based on which we further developed the techniques such as sequential disk accessing, multi-cache
and solved the problem of full garbage collection in the JVM. The benchmark results shows
that it can get impressive improvement in typical cases. When the a system is relatively short
of memory (eg, HPC, small- and medium-size enterprises), the improvement will be even more
impressive.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message