crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Tzolov <christian.tzo...@gmail.com>
Subject Re: Plan Dotfile in Configuration
Date Wed, 01 Jul 2015 22:05:03 GMT
Hi Bryan, Josh,

IIRC this comes from the original dotfile jobplan implementation. I kept it
for backward compatible. You can see that only the "jobplan" (e.g. the
original/main plan) is stored in the Configuration.

+Gabriel i am not sure I remember the original intent to have the jobplan
stored in the Configuration?





On Wed, Jul 1, 2015 at 11:02 PM, Josh Wills <josh.wills@gmail.com> wrote:

> +Christian
>
> I'm not sure what the intent was there-- Christian?
>
> J
>
> On Wed, Jul 1, 2015 at 12:29 PM, Bryan Baugher <bjbq4d@gmail.com> wrote:
>
>> We recently ran into an issue where our code to serialize a pipeline's
>> configuration was throwing an exception because one of the key/values in
>> the config was too big (65k characters). We found this key/value was
>> 'crunch.planner.dotfile' which is included in the pipelines config from
>> Crunch.
>>
>> My question is why does Crunch provide this value into the config object?
>>
>> Crunch saves the dotfile string in the MRExecutor context[1] and I don't
>> think any pipeline would need this at runtime. It also seems like there are
>> no references to this config value anywhere within Crunch other then to
>> write the value into the config object.
>>
>> [1] -
>> https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/DotfileUtills.java#L139-L140
>>
>>
>

Mime
View raw message