samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zach Cox <zcox...@gmail.com>
Subject Re: Problems running new jobs in hello-samza
Date Mon, 06 Oct 2014 18:27:38 GMT
Thanks for the replies everyone! I did the 3 things that Mark mentioned,
re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid
start all`. But when I re-submitted the 3 hello-samza jobs along with my 2
new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory
Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
ACCEPTED.

When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
see this repeated continuously:
https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt

I notice it says "available=<memory:4096, vCores:0>" - is my job not being
run now because vCores=0?

I also updated Vagrantfile to use:

samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end

Thanks,
Zach


On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <lukas@doubledutch.me>
wrote:

> I'll add that if you check the YARN node application master container log
> and see that the job is constantly restarting, you might need to increase
> the container memory limit to 1024MB at least. Also, a good parameter to
> play with in YARN is  yarn.nodemanager.vmem-pmem-ratio.
>
> Lukas
>
> -----Original Message----- From: Mark Mindenhall
> Sent: Monday, October 6, 2014 8:44 AM
> To: dev@samza.incubator.apache.org
> Subject: Re: Problems running new jobs in hello-samza
>
>
> Hi Zach,
>
> I’m also a relative newbie, but I did run into this same issue.  You are
> correct, in that your 5th job isn’t starting due to not enough resources
> available in the cluster, so you need to reduce the resources required.
>
> First, in yarn-site.xml I switched over to the FairScheduler<
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
> FairScheduler.html>:
>
>  <property>
>    <name>yarn.resourcemanager.scheduler.class</name>
>    <value>org.apache.hadoop.yarn.server.resourcemanager.
> scheduler.fair.FairScheduler</value>
>  </property>
>
> I also added these two properties (yarn-site.xml) to control the amount of
> memory allocated to each job:
>
>  <property>
>    <name>yarn.scheduler.minimum-allocation-mb</name>
>    <value>256</value>
>    <description>Minimum limit of memory to allocate to each container
> request at the Resource Manager.</description>
>  </property>
>  <property>
>    <name>yarn.scheduler.maximum-allocation-mb</name>
>    <value>512</value>
>    <description>Maximum limit of memory to allocate to each container
> request at the Resource Manager.</description>
>  </property>
>
> Then, in each of my Samza properties files describing my jobs, I added the
> following two settings:
>
>    yarn.container.memory.mb=512
>    yarn.am.container.memory.mb=256
>
> Hope that helps!
>
> Best,
> Mark
>
>
> On Oct 6, 2014, at 6:27 AM, Zach Cox <zcox522@gmail.com<mailto:zcox
> 522@gmail.com>> wrote:
>
> Hi - I'm just getting started with Samza. I got the hello-samza example
> working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
> everything and submitted them to yarn using run-job.sh. These 2 new jobs
> show up in the yarn web ui, however only one of them has State=RUNNING, the
> other just sits forever at State=ACCEPTED.
>
> The Cluster Metrics section shows some interesting things:
> - Apps Pending = 1
> - Apps Running = 4
> - Containers Running = 8
> - Memory Used = 8 GB
> - Memory Total = 8 GB
> - Memory Reserved = 0 B
>
> Again I'm really new to samza & yarn, but does this mean that the node on
> this vagrant box has 8 GB memory available but all 8 GB is being used, so
> it can't run the 5th samza job?
>
> Are there 8 containers running because each Samza job has an
> ApplicationMaster and a SamzaContainer? Are each of those containers using
> 1 GB memory, and that's why all the available memory is used up? Do these
> containers really need 1 GB memory each? Can this be adjusted somehow?
>
> Just trying to better understand what's going on here, and see if there's a
> simple way to get both of my new tasks running in hello-samza.
>
> Thanks,
> Zach
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message