giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Giraph and Fair Scheduler
Date Tue, 07 May 2013 21:29:36 GMT
Can you check the logs of the failed task and report what the error is?

Avery

On 5/7/13 2:26 PM, Ramani, Arun wrote:
> Hi Avery,
>
> I am setting "minsharepreemptiontimeout" to 5 sec and my Giraph job 
> could not even wait for 5 secs to get its slots. Let me explain the 
> scenario below:
>
> Assume, Cluster capacity is 150
> Queue A (min share –10 maps) - I submit a sleep job with 100 map 
> tasks. Cluster is empty, and hence the first job submitted to Queue A 
> will take the entire 100 map tasks.
> Queue B (Giraph pool with min share – 140 maps) - Now my job 1 is 
> running with 100 tasks occupied. I submit a giraph shortestpathfirst 
> example job with 100 workers to Queue B. Queue B has 
> "minsharepreemptiontimeout" to 5 sec". So, it will first schedule 50 
> tasks since first job only took 100 tasks and cluster's capacity is 
> 150. Meanwhile, in 5 sec, 50 more tasks would be preempted from Queue 
> A and would be given to Giraph Job. I see this happening, however, the 
> job fails with "Unable to create native thread error"
>
> Please let me know if "giraph.maxMasterSuperstepWaitMsecs" will help 
> in this scenario.
>
> Thanks so much
> Arun Ramani
>
> From: Avery Ching <aching@apache.org <mailto:aching@apache.org>>
> Date: Tuesday, May 7, 2013 2:19 PM
> To: "user@giraph.apache.org <mailto:user@giraph.apache.org>" 
> <user@giraph.apache.org <mailto:user@giraph.apache.org>>
> Cc: "Ramani, Arun(aramani)" <aramani@paypal.com 
> <mailto:aramani@paypal.com>>
> Subject: Re: Giraph and Fair Scheduler
>
> Oh, I see.  You can change the timeout of how long the giraph job 
> waits for tasks before giving up.  Try setting 
> giraph.maxMasterSuperstepWaitMsecs to a higher number.  The default is 
> 10 minutes.
>
> Avery
>
> On 5/7/13 2:10 PM, Ramani, Arun wrote:
>> Hi Avery,
>>
>> I am not preempting tasks out of the giraph pool. I have configured 
>> pre-emption so that any job submitted to giraph pool will get its min 
>> share. Any suggestion on how to make this work?
>>
>> Thanks so much in advance.
>>
>> Arun Ramani
>>
>> From: Avery Ching <aching@apache.org <mailto:aching@apache.org>>
>> Reply-To: "user@giraph.apache.org <mailto:user@giraph.apache.org>" 
>> <user@giraph.apache.org <mailto:user@giraph.apache.org>>
>> Date: Tuesday, May 7, 2013 7:25 AM
>> To: "user@giraph.apache.org <mailto:user@giraph.apache.org>" 
>> <user@giraph.apache.org <mailto:user@giraph.apache.org>>
>> Subject: Re: Giraph and Fair Scheduler
>>
>> Can you disable the preemption for the giraph pool?  It's not great 
>> to preempt those tasks.
>>
>> Avery
>>
>> On 5/6/13 6:37 PM, Ramani, Arun wrote:
>>> Hi,
>>>
>>> I am running Fair scheduler with many applications in hadoop stack 
>>> in my cluster (like pig, hive, hbase etc). I have dedicated a pool 
>>> for Giraph and want to run giraph along with those other 
>>> applications. I have configured pre-emption and and set the 
>>> "minsharepreemptiontimeout=5" (sec – for the jobs submitted to this 
>>> pool to wait to get the min share).
>>>
>>> I am trying to run giraph in this mode. I see that jobs from other 
>>> pools are getting pre-empted to give the giraph job's pool its 
>>> configured min share but my job fails with "Unable to create native 
>>> thread" error. This same job passes if the slots are available 
>>> immediately without having to wait for the tasks from other queues 
>>> to be pre-empted. I also tried to tweak the 
>>> "giraph.minPercentResponded=50.0f". My Giraph job still fails. 
>>> Please help in this scenario.
>>>
>>> Basically, I wanted to know how to configure giraph to wait for a 
>>> threshold for the slots to be available for it through pre-emption.
>>>
>>> Thanks
>>> Arun Ramani
>>
>


Mime
View raw message