crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-519) Plan dot file can display more infromation
Date Wed, 20 May 2015 16:41:00 GMT


Josh Wills commented on CRUNCH-519:

Hey [~ronhash], thanks for this! I had a couple of q's based on looking it over: first, why
Kbs for the size of the PCollection vs. Mbs? Also, do we only want to include the numReducers
in the plan file when it's not specified by the developer, or should we always include it
(maybe w/some indication as to whether it was hard-coded or determined by Crunch?)

> Plan dot file can display more infromation
> ------------------------------------------
>                 Key: CRUNCH-519
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ron Hashimshony
>            Assignee: Josh Wills
>         Attachments: CRUNCH-519.diff
> The current plan dot file display nicely the jobs, with nice names and arrows.
> However it does not explain how the planner decided on the reducers number, which is
based on the input data size, scale factor and desired size per reducer.
> I suggest adding this information to the dot file.
> An addition to the DotfileWriter class can do this easily.

This message was sent by Atlassian JIRA

View raw message