samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@linkedin.com.INVALID>
Subject Re: Problems running new jobs in hello-samza
Date Mon, 06 Oct 2014 15:37:54 GMT
Hey Zach,

The Vagrant box is configured to have 2048MB of memory:

  
https://github.com/apache/incubator-samza-hello-samza/blob/master/Vagrantfi
le


The YARN NM by default is configured to have 8GB of memory allotted to it.
This is just an oversight on our part. I'll open a JIRA for that.

Now, your NM has 8GB allotted to it, and all 8GB are being used. Once this
happens, any new containers that need to be started aren't going to be
able to start because there's no space to start them. If the container
that needs to be started is a Samza AM (ApplicationMaster), then the job
will sit in the ACCEPTED state.

You'll need to do one of the following:

1. Run fewer jobs
2. Lower the yarn.container.memory.mb (and probably heap usage if you
customized task.opts).
3. Increase the NM's allotted GB space (yarn-site.xml) and bump up the
Vagrant box's memory footprint as well.

Cheers,
Chris

On 10/6/14 5:27 AM, "Zach Cox" <zcox522@gmail.com> wrote:

>Hi - I'm just getting started with Samza. I got the hello-samza example
>working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
>everything and submitted them to yarn using run-job.sh. These 2 new jobs
>show up in the yarn web ui, however only one of them has State=RUNNING,
>the
>other just sits forever at State=ACCEPTED.
>
>The Cluster Metrics section shows some interesting things:
> - Apps Pending = 1
> - Apps Running = 4
> - Containers Running = 8
> - Memory Used = 8 GB
> - Memory Total = 8 GB
> - Memory Reserved = 0 B
>
>Again I'm really new to samza & yarn, but does this mean that the node on
>this vagrant box has 8 GB memory available but all 8 GB is being used, so
>it can't run the 5th samza job?
>
>Are there 8 containers running because each Samza job has an
>ApplicationMaster and a SamzaContainer? Are each of those containers using
>1 GB memory, and that's why all the available memory is used up? Do these
>containers really need 1 GB memory each? Can this be adjusted somehow?
>
>Just trying to better understand what's going on here, and see if there's
>a
>simple way to get both of my new tasks running in hello-samza.
>
>Thanks,
>Zach


Mime
View raw message