Mailing-List: contact dev-help@samza.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of
 prvs=349f31606=criccomini@linkedin.com designates 69.28.149.81 as permitted
 sender)
From: Chris Riccomini <criccomini@linkedin.com.INVALID>
To: "dev@samza.incubator.apache.org" <dev@samza.incubator.apache.org>
Subject: Re: Problems running new jobs in hello-samza
Thread-Topic: Problems running new jobs in hello-samza
Thread-Index: AQHP4WD38+sK+sWrRUiUu0FvxCCNvZwjNEAA
Date: Mon, 6 Oct 2014 15:37:54 +0000
Message-ID: <D0580465.343F1%criccomini@linkedin.com>
References: 
 <CANjo42zQit-WhvAOAjUe4AULpKrU9gFJGGw2zp3yQvxSFQ2_dg@mail.gmail.com>
In-Reply-To: 
 <CANjo42zQit-WhvAOAjUe4AULpKrU9gFJGGw2zp3yQvxSFQ2_dg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.4.4.140807
Content-Type: text/plain; charset="us-ascii"
Content-ID: <108D6E748A097845BDD2EE88B8DD6CB8@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hey Zach,

The Vagrant box is configured to have 2048MB of memory:

 =20
https://github.com/apache/incubator-samza-hello-samza/blob/master/Vagrantfi
le


The YARN NM by default is configured to have 8GB of memory allotted to it.
This is just an oversight on our part. I'll open a JIRA for that.

Now, your NM has 8GB allotted to it, and all 8GB are being used. Once this
happens, any new containers that need to be started aren't going to be
able to start because there's no space to start them. If the container
that needs to be started is a Samza AM (ApplicationMaster), then the job
will sit in the ACCEPTED state.

You'll need to do one of the following:

1. Run fewer jobs
2. Lower the yarn.container.memory.mb (and probably heap usage if you
customized task.opts).
3. Increase the NM's allotted GB space (yarn-site.xml) and bump up the
Vagrant box's memory footprint as well.

Cheers,
Chris

On 10/6/14 5:27 AM, "Zach Cox" <zcox522@gmail.com> wrote:

>Hi - I'm just getting started with Samza. I got the hello-samza example
>working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
>everything and submitted them to yarn using run-job.sh. These 2 new jobs
>show up in the yarn web ui, however only one of them has State=3DRUNNING,
>the
>other just sits forever at State=3DACCEPTED.
>
>The Cluster Metrics section shows some interesting things:
> - Apps Pending =3D 1
> - Apps Running =3D 4
> - Containers Running =3D 8
> - Memory Used =3D 8 GB
> - Memory Total =3D 8 GB
> - Memory Reserved =3D 0 B
>
>Again I'm really new to samza & yarn, but does this mean that the node on
>this vagrant box has 8 GB memory available but all 8 GB is being used, so
>it can't run the 5th samza job?
>
>Are there 8 containers running because each Samza job has an
>ApplicationMaster and a SamzaContainer? Are each of those containers using
>1 GB memory, and that's why all the available memory is used up? Do these
>containers really need 1 GB memory each? Can this be adjusted somehow?
>
>Just trying to better understand what's going on here, and see if there's
>a
>simple way to get both of my new tasks running in hello-samza.
>
>Thanks,
>Zach