giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Giraph and Fair Scheduler
Date Tue, 07 May 2013 21:19:16 GMT
Oh, I see.  You can change the timeout of how long the giraph job waits 
for tasks before giving up.  Try setting 
giraph.maxMasterSuperstepWaitMsecs to a higher number. The default is 10 
minutes.

Avery

On 5/7/13 2:10 PM, Ramani, Arun wrote:
> Hi Avery,
>
> I am not preempting tasks out of the giraph pool. I have configured 
> pre-emption so that any job submitted to giraph pool will get its min 
> share. Any suggestion on how to make this work?
>
> Thanks so much in advance.
>
> Arun Ramani
>
> From: Avery Ching <aching@apache.org <mailto:aching@apache.org>>
> Reply-To: "user@giraph.apache.org <mailto:user@giraph.apache.org>" 
> <user@giraph.apache.org <mailto:user@giraph.apache.org>>
> Date: Tuesday, May 7, 2013 7:25 AM
> To: "user@giraph.apache.org <mailto:user@giraph.apache.org>" 
> <user@giraph.apache.org <mailto:user@giraph.apache.org>>
> Subject: Re: Giraph and Fair Scheduler
>
> Can you disable the preemption for the giraph pool?  It's not great to 
> preempt those tasks.
>
> Avery
>
> On 5/6/13 6:37 PM, Ramani, Arun wrote:
>> Hi,
>>
>> I am running Fair scheduler with many applications in hadoop stack in 
>> my cluster (like pig, hive, hbase etc). I have dedicated a pool for 
>> Giraph and want to run giraph along with those other applications. I 
>> have configured pre-emption and and set the 
>> "minsharepreemptiontimeout=5" (sec – for the jobs submitted to this 
>> pool to wait to get the min share).
>>
>> I am trying to run giraph in this mode. I see that jobs from other 
>> pools are getting pre-empted to give the giraph job's pool its 
>> configured min share but my job fails with "Unable to create native 
>> thread" error. This same job passes if the slots are available 
>> immediately without having to wait for the tasks from other queues to 
>> be pre-empted. I also tried to tweak the 
>> "giraph.minPercentResponded=50.0f". My Giraph job still fails. Please 
>> help in this scenario.
>>
>> Basically, I wanted to know how to configure giraph to wait for a 
>> threshold for the slots to be available for it through pre-emption.
>>
>> Thanks
>> Arun Ramani
>


Mime
View raw message