giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: setWorkerConfiguration parameters clarification
Date Sun, 10 Nov 2013 20:15:31 GMT
Properly redistributing the data partitions shared by 20 workers if the
next superstep  will run on only 15 workers (for example) would be
prohibitive performance wise for most use cases I think?

In the "Pure YARN" Giraph profile this functionality and many similar
things will be fairly easy to implement, but will have the same
limitations/practical constraints in practice, so may not be a great idea.
Adding more workers as they become available or are required for load
balancing in the middle of a calculation might be more worthwhile, but also
has not been implemented yet in Giraph.



On Wed, Oct 16, 2013 at 11:24 AM, Manuel Lagang <manuellagang@gmail.com>wrote:

> Does anyone have any thoughts on how to have Giraph not require a fixed
> amount of workers, but rather be able to start a superstep with a possibly
> smaller number of workers?
>
>
> On Thu, Oct 10, 2013 at 4:12 PM, Manuel Lagang <manuellagang@gmail.com>wrote:
>
>> I'm trying to understand the meaning of the 3 parameters to
>> GiraphConfiguration.setWorkerConfiguration: minWorkers, maxWorkers, and
>> minPercentResponded. I want my Giraph jobs to co-exist nicely with other
>> jobs in the cluster, and it's not always the case that I can get a fixed
>> number of map slots for my job before the job times out. Thus, I would like
>> the job to be able to start with a possibly smaller set of workers, ideally
>> being able to pick up workers in later supersteps.
>>
>> So I tried <minWorkers=10,maxWorkers=50,minPercentResponded=100>,
>> expecting this to mean that it can start with 10 workers provided that 100%
>> of those workers respond. But this setting ends up again waiting for all 50
>> workers.
>>
>> Then I tried <minWorkers=10,maxWorkers=50,minPercentResponded=20>,
>> expecting that minPercentResponded was just a redundant expression of
>> minWorkers/maxWorkers. But this setting leads to null pointer exceptions in
>> org.apache.giraph.comm.SendCache.removeWorkerData(SendCache.java:199).
>>
>> So I must be confused about the meaning of these variables, and what the
>> legal values are. Can anyone enlighten me on how (if possible) I can get
>> the behavior I want?
>>
>> Thanks,
>> Manuel Lagang
>>
>
>

Mime
View raw message