giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafal Wojdyla <ravwojd...@gmail.com>
Subject Re: yarn container accounting errors.
Date Fri, 03 Jan 2014 21:12:49 GMT
I had the same problem, and I have created this ticket:
https://issues.apache.org/jira/browse/GIRAPH-819
First I thought it's just operator problem, then I got to the same
conclusion as you, and removed the if check.
Btw - what was the purpose of that check, in the same function we check
whether there is enough memory available. Memory check still doesn't
guaranty that the job will run though, imagine 10 really small node manager
(512MB) in total 5120MB, and giraph application that requires 2 containers
1GB each.


On Fri, Jan 3, 2014 at 9:47 PM, Eric Kimbrel <lekimbrel@gmail.com> wrote:

> I am running on hadoop 2.2.0-cdh5.0.0-beta-1 with pure yarn mode.   I =
> have noticed two issues in org.apache.giraph.yarn.GiraphYarnClient
>
> giraph code base is 1.1.0-SNAPSHOT, downloaded on Jan 2, 2014
>
> looking at the checkPerNodeResourcesAvailable Method.
>
> The first issue is that the number of containers available is calculated =
> using node.getNumContainers().  Looking at the yarn documentation this =
> is the number of containers currently running on the node.  So with a =
> yarn cluster with no jobs running all nodes report 0 containers.
>
>
> The second issue (in the same method) is this if block:
>
> if (workers < numContainers) {
>      throw new RuntimeException("Giraph job requires " + workers +
>        " containers to run; cluster only hosts " + numContainers);
>
> }
>
> So with the current set up if a cluster has 4 containers currently =
> running and a graph job is submitted that requires 2 containers the job =
> will fail saying =93Giraph job requires 2 containers to run; cluster =
> only hosts 4=94.
>
> the if statement should be =93 workers > numContainers=94  and =
> numContainers should reflect the total number of containers available, =
> not the number of containers currently running.  I don=92t know yarn =
> well so i don=92t know if such a number is available at all.
>
> For the time being i plan on getting this working by bypassing the check =
> all together.
>

Mime
View raw message