avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Hodges <jhod...@twitter.com>
Subject OOMs in java hadoop tests?
Date Sun, 04 Apr 2010 01:09:40 GMT
Anyone else seeing OutOfMemory errors when trying to run the hadoop tests?

Specifically:


    [junit] Running org.apache.avro.mapred.TestWordCountGeneric
    [junit] Apr 3, 2010 5:57:56 PM org.apache.hadoop.metrics.jvm.JvmMetrics init
    [junit] INFO: Initializing JVM Metrics with
processName=JobTracker, sessionId=
    [junit] Apr 3, 2010 5:57:56 PM org.apache.hadoop.mapred.JobClient
configureCommandLineOptions
    [junit] WARNING: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same.
    [junit] Apr 3, 2010 5:57:57 PM org.apache.hadoop.mapred.JobClient
configureCommandLineOptions
    [junit] WARNING: No job jar file set.  User classes may not be
found. See JobConf(Class) or JobConf#setJar(String).
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.FileInputFormat listStatus
    [junit] INFO: Total input paths to process : 1
    [junit] Apr 3, 2010 5:57:57 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO: Running job: job_local_0001
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.FileInputFormat listStatus
    [junit] INFO: Total input paths to process : 1
    [junit] Apr 3, 2010 5:57:57 PM org.apache.hadoop.mapred.MapTask runOldMapper
    [junit] INFO: numReduceTasks: 1
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
    [junit] INFO: io.sort.mb = 100
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.LocalJobRunner$Job run
    [junit] WARNING: job_local_0001
    [junit] java.lang.OutOfMemoryError: Java heap space
    [junit] 	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
    [junit] 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
    [junit] 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    [junit] 	at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO:  map 0% reduce 0%
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO: Job complete: job_local_0001
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.Counters log
    [junit] INFO: Counters: 0
    [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 2.29 sec
    [junit] Error:
    [junit] Apr 3, 2010 5:57:56 PM org.apache.hadoop.metrics.jvm.JvmMetrics init
    [junit] INFO: Initializing JVM Metrics with
processName=JobTracker, sessionId=
    [junit] Apr 3, 2010 5:57:56 PM org.apache.hadoop.mapred.JobClient
configureCommandLineOptions
    [junit] WARNING: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same.
    [junit] Apr 3, 2010 5:57:57 PM org.apache.hadoop.mapred.JobClient
configureCommandLineOptions
    [junit] WARNING: No job jar file set.  User classes may not be
found. See JobConf(Class) or JobConf#setJar(String).
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.FileInputFormat listStatus
    [junit] INFO: Total input paths to process : 1
    [junit] Apr 3, 2010 5:57:57 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO: Running job: job_local_0001
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.FileInputFormat listStatus
    [junit] INFO: Total input paths to process : 1
    [junit] Apr 3, 2010 5:57:57 PM org.apache.hadoop.mapred.MapTask runOldMapper
    [junit] INFO: numReduceTasks: 1
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
    [junit] INFO: io.sort.mb = 100
    [junit] Apr 3, 2010 5:57:57 PM
org.apache.hadoop.mapred.LocalJobRunner$Job run
    [junit] WARNING: job_local_0001
    [junit] java.lang.OutOfMemoryError: Java heap space
    [junit] 	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
    [junit] 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
    [junit] 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    [junit] 	at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO:  map 0% reduce 0%
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO: Job complete: job_local_0001
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.Counters log
    [junit] INFO: Counters: 0
    [junit]
    [junit] TEST org.apache.avro.mapred.TestWordCountGeneric FAILED
    [junit] Running org.apache.avro.mapred.TestWordCountSpecific
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.metrics.jvm.JvmMetrics init
    [junit] INFO: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.JobClient
configureCommandLineOptions
    [junit] WARNING: Use GenericOptionsParser for parsing the
arguments. Applications should implement Tool for the same.
    [junit] Apr 3, 2010 5:57:58 PM org.apache.hadoop.mapred.JobClient
configureCommandLineOptions
    [junit] WARNING: No job jar file set.  User classes may not be
found. See JobConf(Class) or JobConf#setJar(String).
    [junit] Apr 3, 2010 5:57:58 PM
org.apache.hadoop.mapred.FileInputFormat listStatus
    [junit] INFO: Total input paths to process : 1
    [junit] Apr 3, 2010 5:57:59 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO: Running job: job_local_0002
    [junit] Apr 3, 2010 5:57:59 PM
org.apache.hadoop.mapred.FileInputFormat listStatus
    [junit] INFO: Total input paths to process : 1
    [junit] Apr 3, 2010 5:57:59 PM org.apache.hadoop.mapred.MapTask runOldMapper
    [junit] INFO: numReduceTasks: 1
    [junit] Apr 3, 2010 5:57:59 PM
org.apache.hadoop.mapred.MapTask$MapOutputBuffer <init>
    [junit] INFO: io.sort.mb = 100
    [junit] Apr 3, 2010 5:57:59 PM
org.apache.hadoop.mapred.LocalJobRunner$Job run
    [junit] WARNING: job_local_0002
    [junit] java.lang.OutOfMemoryError: Java heap space
    [junit] 	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
    [junit] 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
    [junit] 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    [junit] 	at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
    [junit] Apr 3, 2010 5:58:00 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO:  map 0% reduce 0%
    [junit] Apr 3, 2010 5:58:00 PM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
    [junit] INFO: Job complete: job_local_0002
    [junit] Apr 3, 2010 5:58:00 PM org.apache.hadoop.mapred.Counters log
    [junit] INFO: Counters: 0
    [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.58 sec
    [junit] TEST org.apache.avro.mapred.TestWordCountSpecific FAILED

Mime
View raw message