samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Mindenhall <mark.mindenh...@machineshop.io>
Subject Re: Problems running new jobs in hello-samza
Date Mon, 06 Oct 2014 18:53:28 GMT
Yes, looks like you need to increase the number of vCores to at least 10 in order to run 5
jobs (yarn-site.xml):

  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>10</value>
    <description>Number of CPU cores that can be allocated for containers.</description>
  </property>


On Oct 6, 2014, at 12:27 PM, Zach Cox <zcox522@gmail.com> wrote:

> Thanks for the replies everyone! I did the 3 things that Mark mentioned,
> re-built & deployed the .tar.gz, then did `bin/grid stop all` and `bin/grid
> start all`. But when I re-submitted the 3 hello-samza jobs along with my 2
> new jobs, yarn still won't run my 5th job. The yarn web ui now shows Memory
> Used = 4 GB and Memory Total = 8 GB, but my job still sits at State =
> ACCEPTED.
> 
> When I tail deploy/yarn/logs/yarn-vagrant-resourcemanager-precise64.log I
> see this repeated continuously:
> https://gist.githubusercontent.com/zcox/0f2b260d29e18d40d038/raw/0d805bcd7d8fec5332756efc9c990679480df117/gistfile1.txt
> 
> I notice it says "available=<memory:4096, vCores:0>" - is my job not being
> run now because vCores=0?
> 
> I also updated Vagrantfile to use:
> 
> samza.vm.provider :virtualbox do |vb| vb.memory = 4096 vb.cpus = 8 end
> 
> Thanks,
> Zach
> 
> 
> On Mon, Oct 6, 2014 at 12:40 PM, Lukas Steiblys <lukas@doubledutch.me>
> wrote:
> 
>> I'll add that if you check the YARN node application master container log
>> and see that the job is constantly restarting, you might need to increase
>> the container memory limit to 1024MB at least. Also, a good parameter to
>> play with in YARN is  yarn.nodemanager.vmem-pmem-ratio.
>> 
>> Lukas
>> 
>> -----Original Message----- From: Mark Mindenhall
>> Sent: Monday, October 6, 2014 8:44 AM
>> To: dev@samza.incubator.apache.org
>> Subject: Re: Problems running new jobs in hello-samza
>> 
>> 
>> Hi Zach,
>> 
>> I’m also a relative newbie, but I did run into this same issue.  You are
>> correct, in that your 5th job isn’t starting due to not enough resources
>> available in the cluster, so you need to reduce the resources required.
>> 
>> First, in yarn-site.xml I switched over to the FairScheduler<
>> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
>> FairScheduler.html>:
>> 
>> <property>
>>   <name>yarn.resourcemanager.scheduler.class</name>
>>   <value>org.apache.hadoop.yarn.server.resourcemanager.
>> scheduler.fair.FairScheduler</value>
>> </property>
>> 
>> I also added these two properties (yarn-site.xml) to control the amount of
>> memory allocated to each job:
>> 
>> <property>
>>   <name>yarn.scheduler.minimum-allocation-mb</name>
>>   <value>256</value>
>>   <description>Minimum limit of memory to allocate to each container
>> request at the Resource Manager.</description>
>> </property>
>> <property>
>>   <name>yarn.scheduler.maximum-allocation-mb</name>
>>   <value>512</value>
>>   <description>Maximum limit of memory to allocate to each container
>> request at the Resource Manager.</description>
>> </property>
>> 
>> Then, in each of my Samza properties files describing my jobs, I added the
>> following two settings:
>> 
>>   yarn.container.memory.mb=512
>>   yarn.am.container.memory.mb=256
>> 
>> Hope that helps!
>> 
>> Best,
>> Mark
>> 
>> 
>> On Oct 6, 2014, at 6:27 AM, Zach Cox <zcox522@gmail.com<mailto:zcox
>> 522@gmail.com>> wrote:
>> 
>> Hi - I'm just getting started with Samza. I got the hello-samza example
>> working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt
>> everything and submitted them to yarn using run-job.sh. These 2 new jobs
>> show up in the yarn web ui, however only one of them has State=RUNNING, the
>> other just sits forever at State=ACCEPTED.
>> 
>> The Cluster Metrics section shows some interesting things:
>> - Apps Pending = 1
>> - Apps Running = 4
>> - Containers Running = 8
>> - Memory Used = 8 GB
>> - Memory Total = 8 GB
>> - Memory Reserved = 0 B
>> 
>> Again I'm really new to samza & yarn, but does this mean that the node on
>> this vagrant box has 8 GB memory available but all 8 GB is being used, so
>> it can't run the 5th samza job?
>> 
>> Are there 8 containers running because each Samza job has an
>> ApplicationMaster and a SamzaContainer? Are each of those containers using
>> 1 GB memory, and that's why all the available memory is used up? Do these
>> containers really need 1 GB memory each? Can this be adjusted somehow?
>> 
>> Just trying to better understand what's going on here, and see if there's a
>> simple way to get both of my new tasks running in hello-samza.
>> 
>> Thanks,
>> Zach
>> 
>> 


Mime
View raw message