giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Lagang <manuellag...@gmail.com>
Subject Re: setWorkerConfiguration parameters clarification
Date Wed, 16 Oct 2013 18:24:56 GMT
Does anyone have any thoughts on how to have Giraph not require a fixed
amount of workers, but rather be able to start a superstep with a possibly
smaller number of workers?


On Thu, Oct 10, 2013 at 4:12 PM, Manuel Lagang <manuellagang@gmail.com>wrote:

> I'm trying to understand the meaning of the 3 parameters to
> GiraphConfiguration.setWorkerConfiguration: minWorkers, maxWorkers, and
> minPercentResponded. I want my Giraph jobs to co-exist nicely with other
> jobs in the cluster, and it's not always the case that I can get a fixed
> number of map slots for my job before the job times out. Thus, I would like
> the job to be able to start with a possibly smaller set of workers, ideally
> being able to pick up workers in later supersteps.
>
> So I tried <minWorkers=10,maxWorkers=50,minPercentResponded=100>,
> expecting this to mean that it can start with 10 workers provided that 100%
> of those workers respond. But this setting ends up again waiting for all 50
> workers.
>
> Then I tried <minWorkers=10,maxWorkers=50,minPercentResponded=20>,
> expecting that minPercentResponded was just a redundant expression of
> minWorkers/maxWorkers. But this setting leads to null pointer exceptions in
> org.apache.giraph.comm.SendCache.removeWorkerData(SendCache.java:199).
>
> So I must be confused about the meaning of these variables, and what the
> legal values are. Can anyone enlighten me on how (if possible) I can get
> the behavior I want?
>
> Thanks,
> Manuel Lagang
>

Mime
View raw message