hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shahab mehmandoust" <shaha...@gmail.com>
Subject map/reduce driver as daemon
Date Thu, 06 Nov 2008 01:10:08 GMT
I'm try to write a daemon that periodically wakes up and runs map/reduce
jobs, but I've have little luck.  I've tried different ways (including using
cascading) and I keep arriving at the below exception:

java.lang.OutOfMemoryError: Java heap space
    at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:359)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
    at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
    at
com.txing.mapred.watcher.DirWatcherImpl2.runMapReduce(DirWatcherImpl2.java:29)
    at com.txing.mapred.watcher.DirWatcher.run(DirWatcher.java:52)
    at java.lang.Thread.run(Thread.java:637)

I've set this propertyl: mapred.child.java.opts  larger and larger makes no
difference.

Furthermore, I get working like this:

WARN | No job jar file set.  User classes may not be found. See
JobConf(Class) or JobConf#setJar(String). | JobClient.java:637 |
org.apache.hadoop.mapred.JobClient | Thread-0 |
WARN | job_local_1 | LocalJobRunner.java:234 |
org.apache.hadoop.mapred.LocalJobRunner | Thread-15 |

Do I have to submit jar files to hadoop?  Can't I daemonize this?

Thanks,
Shahab

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message