hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: setNumBspTask not always honored
Date Wed, 25 Sep 2013 22:14:58 GMT

If there are input data files, the number of tasks is determined by
the number of HDFS blocks in the input files. You can force the number
of tasks using the partition function (it means that creating N
partitions), but it's not recommended.

On Thu, Sep 26, 2013 at 1:10 AM, Anveshi Charuvaka
<anvesh.subs@gmail.com> wrote:
> Hello Everyone,
>         What is the impact of setting the number of bsp tasks using BSPJob.setNumBspTasks(N)
? Is the number of bsp tasks spawned ℗ by Hama equal to the set parameter value (N) or is
it just a recommendation to the framework. Does the number of input (key,value) pairs provided
have any influence over the number of tasks spawned?
>         Context: Let N = parameter of setNumBspTasks, P = # of bsp tasks spawned by hama
(inferred from console output). I have created a small BSP program and I am  running it in
local mode with different inputs and different values of number of jobs (provided through
BSPJob.setNumBspTasks). When I set the input format as NullInputFormat then, P == N. If I
use TextInputFormat or KeyValueInputFormat the P is not always equal to N, usually P <
N and never P > N.
> Thanks
> Anvesh

Best Regards, Edward J. Yoon

View raw message