giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Islam <misla...@yahoo.com>
Subject Re: yarn container accounting errors.
Date Sat, 04 Jan 2014 01:19:24 GMT
I agree with Eric and Rafal's assessment. You can send a patch about this.

Regards,
Mohammad



On Friday, January 3, 2014 1:13 PM, Rafal Wojdyla <ravwojdyla@gmail.com> wrote:
 
I had the same problem, and I have created this ticket: https://issues.apache.org/jira/browse/GIRAPH-819
First I thought it's just operator problem, then I got to the same conclusion as you, and
removed the if check.
Btw - what was the purpose of that check, in the same function we check whether there is enough
memory available. Memory check still doesn't guaranty that the job will run though, imagine
10 really small node manager (512MB) in total 5120MB, and giraph application that requires
2 containers 1GB each.



On Fri, Jan 3, 2014 at 9:47 PM, Eric Kimbrel <lekimbrel@gmail.com> wrote:

I am running on hadoop 2.2.0-cdh5.0.0-beta-1 with pure yarn mode.   I =
>have noticed two issues in org.apache.giraph.yarn.GiraphYarnClient
>
>giraph code base is 1.1.0-SNAPSHOT, downloaded on Jan 2, 2014
>
>looking at the checkPerNodeResourcesAvailable Method.
>
>The first issue is that the number of containers available is calculated =
>using node.getNumContainers().  Looking at the yarn documentation this =
>is the number of containers currently running on the node.  So with a =
>yarn cluster with no jobs running all nodes report 0 containers.
>
>
>The second issue (in the same method) is this if block:
>
>if (workers < numContainers) {
>     throw new RuntimeException("Giraph job requires " + workers +
>       " containers to run; cluster only hosts " + numContainers);
>
>}
>
>So with the current set up if a cluster has 4 containers currently =
>running and a graph job is submitted that requires 2 containers the job =
>will fail saying =93Giraph job requires 2 containers to run; cluster =
>only hosts 4=94.
>
>the if statement should be =93 workers > numContainers=94  and =
>numContainers should reflect the total number of containers available, =
>not the number of containers currently running.  I don=92t know yarn =
>well so i don=92t know if such a number is available at all.
>
>For the time being i plan on getting this working by bypassing the check =
>all together.
>
Mime
View raw message