Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Mon, 4 Nov 2013 06:49:22 +0000 (UTC)
From: "Ming Chen (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12677229.1383401907229.14901.1383547762712@arcas>
In-Reply-To: <JIRA.12677229.1383401907229@arcas>
References: <JIRA.12677229.1383401907229@arcas>
Subject: [jira] [Updated] (MAPREDUCE-5605) Memory-centric MapReduce aiming
 to solve the I/O bottleneck
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/MAPREDUCE-5605?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ming Chen updated MAPREDUCE-5605:
---------------------------------

    Attachment:     (was: CacheOutputStream.java)

> Memory-centric MapReduce aiming to solve the I/O bottleneck
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-5605
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5605
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 1.0.1
>         Environment: x86-64 Linux/Unix
> jdk7 preferred
>            Reporter: Ming Chen
>            Assignee: Ming Chen
>         Attachments: JobTaskRunner.java, JvmManager.java, JvmTask.java, M=
apOutputFile.java, MapRamManager.java, MapRunner.java, MapTask.java, MapTas=
kCompletionEventsUpdate.java, MapTaskRunner.java, MapTaskStatus.java, Memor=
yElement.java, MergeSorter.java, Merger.java, Operation.java, OutputCollect=
or.java, OutputCommitter.java, OutputFormat.java, OutputLogFilter.java, Par=
titioner.java, RamManager.java, RawBufferedOutputStream.java, RawHistoryFil=
eServlet.java, RawKeyValueIterator.java, RecordReader.java, ReduceRamManage=
r.java, ReduceTask.java, ReduceTaskRunner.java, ReduceTaskStatus.java, Rein=
itTrackerAction.java, RoundQueue.java, RunningJob.java, SequenceFileOutputF=
ormat.java, SpillScheduler.java, Task.java, TaskInProgress.java, TaskLog.ja=
va, TaskLogAppender.java, TaskLogServlet.java, TaskLogsTruncater.java, Task=
MemoryManagerThread.java, TaskReport.java, TaskRunner.java, TaskScheduler.j=
ava, TaskStatus.java, TaskTracker.java, TaskTrackerAction.java, TaskTracker=
Instrumentation.java, TaskTrackerStatus.java, TextOutputFormat.java
>
>
> Memory is a very important resource to bridge the gap between CPUs and I/=
O devices. So the idea is to maximize the usage of memory to solve the prob=
lem of I/O bottleneck. We developed a multi-threaded task execution engine,=
 which runs in a single JVM on a node. In the execution engine, we have imp=
lemented the algorithm of memory scheduling to realize global memory manage=
ment, based on which we further developed the techniques such as sequential=
 disk accessing, multi-cache and solved the problem of full garbage collect=
ion in the JVM. We have conducted extensive experiments with comparison aga=
inst the native Hadoop platform. The results show that the Mammoth system c=
an reduce the job execution time by more than 40% in typical cases, without=
 requiring any modifications of the Hadoop programs. When a system is short=
 of memory, Mammoth can improve the performance by up to 4 times, as observ=
ed for I/O intensive applications, such as PageRank.=20


--
This message was sent by Atlassian JIRA
(v6.1#6144)