hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject Container is running beyond physical memory limits
Date Tue, 13 Oct 2015 16:31:14 GMT
Hi,

 

I have been having some issues with loading data into hive from one table to
another for 1,767,886 rows. I was getting the following error

 

Task with the most failures(4):

-----

Task ID:

  task_1444731612741_0001_r_000000

 

URL:

 
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1444731612741_0001&tipid=task_
1444731612741_0001_r_000000

-----

Diagnostic Messages for this Task:

Container [pid=16238,containerID=container_1444731612741_0001_01_000019] is
running beyond physical memory limits. Current usage: 2.0 GB of 2 GB
physical memory used; 6.6 GB of 8 GB virtual memory used. Killing container.

 

 

Changed parameters in yarn-site.xml  and mapred-site.xml files few times but
no joy.

 

Finally the following changes in mapred-site.xml worked for me

 

<property>

<name>mapreduce.job.tracker.reserved.physicalmemory.mb</name>

<value>1024</value>

</property>

 

<property>

<name>mapreduce.map.memory.mb</name>

<value>4096</value>

</property>

 

<property>

<name>mapreduce.reduce.memory.mb</name>

<value>4096</value>

</property>

 

<property>

<name>mapreduce.map.java.opts</name>

<value>-Xmx3072m</value>

</property>

 

<property>

<name>mapreduce.reduce.java.opts</name>

<value>-Xmx6144m</value>

</property>

 

And the following changes to yarn-site.xml

 

<property>

   <name>yarn.nodemanager.vmem-check-enabled</name>

   <value>false</value>

</property>

<property>

  <name>yarn.nodemanager.resource.memory-mb</name>

  <value>8192</value>

 <description>Amount of physical memory, in MB, that can be allocated for
containers.</description>

</property>

<property>

   <name>yarn.nodemanager.vmem-pmem-ratio</name>

    <value>4</value>

    <description>Ratio between virtual memory to physical memory when
setting memory limits for containers</description>

  </property>

 

I did a lot of web search but most resolution to this issue seems to be
cryptic or anecdotal. Anyone has better explanation I would be interested.

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

 
<http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908
.pdf>
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.
pdf

Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN:
978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly

 

 <http://talebzadehmich.wordpress.com/> http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.

 


Mime
View raw message