hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neemesh (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6707) Running a mapreduce Job for updating Hbase table. data we are getting from txt/csv file 1.1 gb from hdfs location
Date Tue, 31 May 2016 12:21:12 GMT
Neemesh created MAPREDUCE-6707:
----------------------------------

             Summary: Running a mapreduce Job for updating Hbase table. data we are getting
from txt/csv file 1.1 gb from hdfs location
                 Key: MAPREDUCE-6707
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6707
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Neemesh


I am try to read 1.1 gb file which has data and i am reading this file and based on the data
doing a put to Hbase table , mapper is working fine but reducer is doing till 85% and fails
with following error

java.lang.OutOfMemoryError: Java heap space
        at java.io.DataOutputStream.<init>(DataOutputStream.java:204)
        at org.apache.jute.BinaryOutputArchive.getArchive(BinaryOutputArchive.java:38)
        at org.apache.zookeeper.ClientCnxn$Packet.createBB(ClientCnxn.java:282)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:115)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
16/05/31 03:37:50 WARN mapred.LocalJobRunner: job_local1952932043_0001
java.lang.OutOfMemoryError: Java heap space

following are the configuration i have given for the running this mapreduce job

configuration.set("mapreduce.child.java.opts", "-Xmx16g -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit");

configuration.set("mapreduce.reduce.memory.mb", "8192m");

configuration.set("mapred.reduce.child.java.opts", "-Xmx10g");

I try reducing the file size by using some split logic. what i have notice if the file size
is 350 Mb Mapreduce work file without issue, but if it is more than 350 Mb we will be getting
OOM exception.

can some one please let us know what configuration is am missing or changes is required to
run Mapreduce Job 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message