crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ioannis Kerkinos (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-505) Store intermediate data in memory only using Tachyon
Date Tue, 31 Mar 2015 11:26:52 GMT
Ioannis Kerkinos created CRUNCH-505:
---------------------------------------

             Summary: Store intermediate data in memory only using Tachyon
                 Key: CRUNCH-505
                 URL: https://issues.apache.org/jira/browse/CRUNCH-505
             Project: Crunch
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.12.0
            Reporter: Ioannis Kerkinos
            Assignee: Josh Wills


Tachyon is a memory-centric distributed storage system that enables reliable data sharing
at memory-speed. If used as the storage for intermediate data (between MR jobs) it should
improve performance as you won't have to go to HDFS. In order to do so, the MUST_CACHE write
type of Tachyon can be used. This will enable data to be persisted in memory only without
going to HDFS. So the intermediate data will be read/written at memory-speed and only the
final result will be written in HDFS.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message