giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafal Wojdyla <ravwojd...@gmail.com>
Subject Re: yarn container accounting errors.
Date Sat, 04 Jan 2014 11:17:12 GMT
I updated patch (remove check) in
https://issues.apache.org/jira/browse/GIRAPH-819. Could you please take a
look.
Should I create a separate ticket for correct memory check?



On Sat, Jan 4, 2014 at 2:19 AM, Mohammad Islam <mislam77@yahoo.com> wrote:

> I agree with Eric and Rafal's assessment. You can send a patch about this.
>
> Regards,
> Mohammad
>
>
>   On Friday, January 3, 2014 1:13 PM, Rafal Wojdyla <ravwojdyla@gmail.com>
> wrote:
>  I had the same problem, and I have created this ticket:
> https://issues.apache.org/jira/browse/GIRAPH-819
> First I thought it's just operator problem, then I got to the same
> conclusion as you, and removed the if check.
> Btw - what was the purpose of that check, in the same function we check
> whether there is enough memory available. Memory check still doesn't
> guaranty that the job will run though, imagine 10 really small node manager
> (512MB) in total 5120MB, and giraph application that requires 2 containers
> 1GB each.
>
>
> On Fri, Jan 3, 2014 at 9:47 PM, Eric Kimbrel <lekimbrel@gmail.com> wrote:
>
> I am running on hadoop 2.2.0-cdh5.0.0-beta-1 with pure yarn mode.   I =
> have noticed two issues in org.apache.giraph.yarn.GiraphYarnClient
>
> giraph code base is 1.1.0-SNAPSHOT, downloaded on Jan 2, 2014
>
> looking at the checkPerNodeResourcesAvailable Method.
>
> The first issue is that the number of containers available is calculated =
> using node.getNumContainers().  Looking at the yarn documentation this =
> is the number of containers currently running on the node.  So with a =
> yarn cluster with no jobs running all nodes report 0 containers.
>
>
> The second issue (in the same method) is this if block:
>
> if (workers < numContainers) {
>      throw new RuntimeException("Giraph job requires " + workers +
>        " containers to run; cluster only hosts " + numContainers);
>
> }
>
> So with the current set up if a cluster has 4 containers currently =
> running and a graph job is submitted that requires 2 containers the job =
> will fail saying =93Giraph job requires 2 containers to run; cluster =
> only hosts 4=94.
>
> the if statement should be =93 workers > numContainers=94  and =
> numContainers should reflect the total number of containers available, =
> not the number of containers currently running.  I don=92t know yarn =
> well so i don=92t know if such a number is available at all.
>
> For the time being i plan on getting this working by bypassing the check =
> all together.
>
>
>
>
>

Mime
View raw message