crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Baugher <bjb...@gmail.com>
Subject Plan Dotfile in Configuration
Date Wed, 01 Jul 2015 19:29:04 GMT
We recently ran into an issue where our code to serialize a pipeline's
configuration was throwing an exception because one of the key/values in
the config was too big (65k characters). We found this key/value was
'crunch.planner.dotfile' which is included in the pipelines config from
Crunch.

My question is why does Crunch provide this value into the config object?

Crunch saves the dotfile string in the MRExecutor context[1] and I don't
think any pipeline would need this at runtime. It also seems like there are
no references to this config value anywhere within Crunch other then to
write the value into the config object.

[1] -
https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/DotfileUtills.java#L139-L140

Mime
View raw message