crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Sasvari (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-636) Make replication factor for temporary files configurable
Date Mon, 13 Feb 2017 20:44:41 GMT
Attila Sasvari created CRUNCH-636:
-------------------------------------

             Summary: Make replication factor for temporary files configurable
                 Key: CRUNCH-636
                 URL: https://issues.apache.org/jira/browse/CRUNCH-636
             Project: Crunch
          Issue Type: New Feature
            Reporter: Attila Sasvari


As of now, Crunch does not allow having different replication factor for temporary files and
non-temporary files (e.g. final output data of leaf nodes) at the same time. If a user has
a large amount of data (say hundreds a of gigabytes) to process, they might want to have lower
replication factor for large temporary files between Crunch jobs. 

We could make this configurable via a new setting (e.g. {{crunch.tmp.dir.replication}}).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message