crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Sasvari (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-636) Make replication factor for temporary files configurable
Date Thu, 23 Mar 2017 21:06:42 GMT


Attila Sasvari updated CRUNCH-636:
    Attachment: CRUNCH-636.04-amendment.patch

- Fixed test failure in {{JobPrototypeTest#initialReplicationFactorUsedFromFileSystem}}
- Fixed 1 JLint warning in JobPrototype (Local variable 'job' shadows component of class -
Configuration argument of {{setInitialJobReplicationConfig()}} was called job) 

> Make replication factor for temporary files configurable
> --------------------------------------------------------
>                 Key: CRUNCH-636
>                 URL:
>             Project: Crunch
>          Issue Type: New Feature
>            Reporter: Attila Sasvari
>            Assignee: Attila Sasvari
>             Fix For: 1.0.0
>         Attachments: CRUNCH-636.01.patch, CRUNCH-636.02.patch, CRUNCH-636.03.patch, CRUNCH-636.04-amendment.patch,
CRUNCH-636.04.patch,, test.WordCount_2017-03-08_16.31.55.737.log
> As of now, Crunch does not allow having different replication factor for temporary files
and non-temporary files (e.g. final output data of leaf nodes) at the same time. If a user
has a large amount of data (say hundreds a of gigabytes) to process, they might want to have
lower replication factor for large temporary files between Crunch jobs. 
> We could make this configurable via a new setting (e.g. {{crunch.tmp.dir.replication}}).

This message was sent by Atlassian JIRA

View raw message