crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ioannis Kerkinos (JIRA)" <>
Subject [jira] [Created] (CRUNCH-505) Store intermediate data in memory only using Tachyon
Date Tue, 31 Mar 2015 11:26:52 GMT
Ioannis Kerkinos created CRUNCH-505:

             Summary: Store intermediate data in memory only using Tachyon
                 Key: CRUNCH-505
             Project: Crunch
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.12.0
            Reporter: Ioannis Kerkinos
            Assignee: Josh Wills

Tachyon is a memory-centric distributed storage system that enables reliable data sharing
at memory-speed. If used as the storage for intermediate data (between MR jobs) it should
improve performance as you won't have to go to HDFS. In order to do so, the MUST_CACHE write
type of Tachyon can be used. This will enable data to be persisted in memory only without
going to HDFS. So the intermediate data will be read/written at memory-speed and only the
final result will be written in HDFS.

This message was sent by Atlassian JIRA

View raw message