hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: out of memory running examples
Date Tue, 07 Feb 2012 12:45:29 GMT
What is your io.sort.mb set to? Given the heap size, you may have to
lower it down.

And did you intend to run the program in 'local' mode (i.e.
non-distributed mode, no JobTracker but LocalJobRunner)?

On Tue, Feb 7, 2012 at 3:20 PM, Tim Broberg <Tim.Broberg@exar.com> wrote:
> I'm trying to run the basic example from hadoop/hadoop-1.0.0/docs/single_node_setup.html.
>
> I'm getting java.lang.OutOfMemoryError's when I run the grep example from that page.
>
> Stackoverflow suggests various tweaks to the command line, mapred-site.xml, or hadoop-env.sh,
none of which seem to be helping in my case.
>
> When I tweak hadoop-env.sh to echo text to a file, that file doesn't show up, which suggests
that hadoop-env.sh isn't even getting executed.
>
> Any hints on debugging this?
>
>    - Tim.
>
> [tbroberg@san-mothra hadoop-1.0.0]$ bin/hadoop jar hadoop-examples-1.0.0.jar grep input
output 'dfs[a-z.]+'
> Warning: $HADOOP_HOME is deprecated.
> 12/02/07 01:39:35 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/07 01:39:35 INFO mapred.FileInputFormat: Total input paths to process : 7
> 12/02/07 01:39:35 INFO mapred.JobClient: Running job: job_local_0001
> 12/02/07 01:39:35 INFO util.ProcessTree: setsid exited with exit code 0
> 12/02/07 01:39:35 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4c349471<mailto:org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4c349471>
> 12/02/07 01:39:35 INFO mapred.MapTask: numReduceTasks: 1
> 12/02/07 01:39:35 INFO mapred.MapTask: io.sort.mb = 100
> 12/02/07 01:39:35 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.OutOfMemoryError: Java heap space
>        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
>        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> 12/02/07 01:39:36 INFO mapred.JobClient:  map 0% reduce 0%
> 12/02/07 01:39:36 INFO mapred.JobClient: Job complete: job_local_0001
> 12/02/07 01:39:36 INFO mapred.JobClient: Counters: 0
> 12/02/07 01:39:36 INFO mapred.JobClient: Job Failed: NA
> java.io.IOException: Job failed!
>        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
>        at org.apache.hadoop.examples.Grep.run(Grep.java:69)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.examples.Grep.main(Grep.java:93)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> [tbroberg@san-mothra hadoop-1.0.0]$
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.



-- 
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

Mime
View raw message