hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mahajan, Neeraj" <nemaha...@ebay.com>
Subject RE: Please help with running hadoop from IDE
Date Mon, 23 Jul 2007 16:46:55 GMT
Search for the posting titled "How to Start Hadoop Cluster from source
code in Eclipse".
The idea should apply to any IDE.

~ Neeraj

-----Original Message-----
From: Jeroen Verhagen [mailto:jeroenverhagen@gmail.com] 
Sent: Monday, July 23, 2007 6:50 AM
To: hadoop-user@lucene.apache.org
Subject: Please help with running hadoop from IDE

Hi all,

I'm trying to get Hadoop 0.13 running from Intellij so I can test my own
Map and Reduce classes on the local job runner. To do that I'm first
trying to get the WordCount sample to work and I'm already running into
trouble:

I copied the WordCount class into my own package from my own project,
created a directory 'in' containing the file from which I want to count
the words. I copied the 'conf' dir from the hadoop dir that I know
contains a configuration that allows me to run the WordCount sample.

This is the output from my console:

******************************
C:\java\jdk1.5.0\bin\java
	-Xmx1000m
	-Dhadoop.log.dir=logs
	-Dhadoop.log.file=hadoop.log
	-Dhadoop.home.dir=.
	-Dhadoop.id.str=
	-Dhadoop.root.logger=DEBUG,console
	-Didea.launcher.port=7534
	"-Didea.launcher.bin.path=C:\java\IntelliJ IDEA 5.1\bin"
	-Dfile.encoding=windows-1252
	-classpath "
	C:\java\jdk1.5.0\jre\lib\charsets.jar;
	C:\java\jdk1.5.0\jre\lib\deploy.jar;
	C:\java\jdk1.5.0\jre\lib\javaws.jar;
	C:\java\jdk1.5.0\jre\lib\jce.jar;
	C:\java\jdk1.5.0\jre\lib\jsse.jar;
	C:\java\jdk1.5.0\jre\lib\plugin.jar;
	C:\java\jdk1.5.0\jre\lib\rt.jar;
	C:\java\jdk1.5.0\jre\lib\ext\dnsns.jar;
	C:\java\jdk1.5.0\jre\lib\ext\localedata.jar;
	C:\java\jdk1.5.0\jre\lib\ext\sunjce_provider.jar;
	C:\java\jdk1.5.0\jre\lib\ext\sunpkcs11.jar;
	C:\mycomp\myproject\core\target\test-classes;
	C:\mycomp\myproject\core\target\classes;
	C:\mycomp\myproject\core\lib\hadoop-0.13.0-core.jar;
	C:\java\IntelliJ IDEA 5.1\lib\idea_rt.jar"
com.intellij.rt.execution.application.AppMain nl.mycomp.TestHadoop in
target/out

	Exception in thread "main" java.io.IOException: Job failed!
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
	at nl.mycomp.TestHadoop.main(TestHadoop.java:106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

Process finished with exit code 1
******************************

And this is the content of the log4j log file:

******************************
DEBUG 15:36:11,813 main org.apache.hadoop.conf.Configuration -
java.io.IOException: config()
	at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:66)
	at nl.mycomp.TestHadoop.main(TestHadoop.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

DEBUG 15:36:12,313 main org.apache.hadoop.mapred.JobClient - default
FileSystem: file:///
INFO 15:36:12,344 main org.apache.hadoop.mapred.FileInputFormat - Total
input paths to process : 1 DEBUG 15:36:12,344 main
org.apache.hadoop.mapred.JobClient - Creating splits at
file:/tmp/hadoop-jeroen/mapred/system/submit_em16mi/job.split
DEBUG 15:36:12,360 main org.apache.hadoop.mapred.FileInputFormat - Total
# of splits: 2 DEBUG 15:36:12,579 main
org.apache.hadoop.conf.Configuration -
java.io.IOException: config(config)
	at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:103)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
	at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:8
3)
	at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:26
0)
	at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:376)
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
	at nl.mycomp.TestHadoop.main(TestHadoop.java:106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

DEBUG 15:36:12,657 main org.apache.hadoop.conf.Configuration -
java.io.IOException: config()
	at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:93)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:107)
	at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:8
7)
	at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:26
0)
	at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:376)
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
	at nl.mycomp.TestHadoop.main(TestHadoop.java:106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at
com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)

INFO 15:36:12,719 main org.apache.hadoop.mapred.JobClient - Running
job: job_8wasq7
DEBUG 15:36:12,719 Thread-0 org.apache.hadoop.mapred.FileInputFormat -
Total # of splits: 1 DEBUG 15:36:12,735 Thread-0
org.apache.hadoop.conf.Configuration -
java.io.IOException: config(config)
	at
org.apache.hadoop.conf.Configuration.<init>(Configuration.java:103)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:77)
	at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)

DEBUG 15:36:12,782 Thread-0 org.apache.hadoop.mapred.MapTask - Writing
local split to /tmp/hadoop-jeroen/mapred/system/submit_em16mi/split.dta
INFO 15:36:12,782 Thread-0 org.apache.hadoop.mapred.MapTask -
numReduceTasks: 1 INFO 15:36:13,719 main
org.apache.hadoop.mapred.JobClient -  map 0% reduce 0% INFO 15:36:13,751
Thread-0 org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:14,735 main org.apache.hadoop.mapred.JobClient -  map 9%
reduce 0% INFO 15:36:14,766 Thread-0
org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:15,735 main org.apache.hadoop.mapred.JobClient -  map 23%
reduce 0% INFO 15:36:15,969 Thread-0
org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:16,735 main org.apache.hadoop.mapred.JobClient -  map 36%
reduce 0% INFO 15:36:17,063 Thread-0
org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:17,735 main org.apache.hadoop.mapred.JobClient -  map 54%
reduce 0% INFO 15:36:18,079 Thread-0
org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
INFO 15:36:18,798 main org.apache.hadoop.mapred.JobClient -  map 73%
reduce 0% INFO 15:36:19,094 Thread-0
org.apache.hadoop.mapred.LocalJobRunner -
file:/C:/mycomp/myproject/core/in/kjv12.txt:0+4834757
DEBUG 15:36:19,751 Sort progress reporter for task map_0000
org.apache.hadoop.mapred.MapTask - Started thread: Sort progress
reporter for task map_0000 WARN 15:36:19,766 Thread-0
org.apache.hadoop.mapred.LocalJobRunner - job_8wasq7
java.io.IOException: CreateProcess: df -k
C:\tmp\hadoop-jeroen\mapred\local error=2
	at java.lang.ProcessImpl.create(Native Method)
	at java.lang.ProcessImpl.<init>(ProcessImpl.java:81)
	at java.lang.ProcessImpl.start(ProcessImpl.java:30)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
	at java.lang.Runtime.exec(Runtime.java:591)
	at java.lang.Runtime.exec(Runtime.java:464)
	at org.apache.hadoop.fs.DF.doDF(DF.java:60)
	at org.apache.hadoop.fs.DF.<init>(DF.java:53)
	at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(L
ocalDirAllocator.java:181)
	at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathF
orWrite(LocalDirAllocator.java:218)
	at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllo
cator.java:124)
	at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFil
e.java:88)
	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapT
ask.java:405)
	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:620)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:187)
	at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:131)
******************************
	
Could anyone please help me and tell me what's causing this problem?

On a sidenote: Why is hadoop creating tmp files in C:\tmp? I did not
configure C:\tmp to be my tempdir, that's
C:\DOCUME~1\jeroen\LOCALS~1\Temp which is set through an environment
variable.

Thanks and regards,

Jeroen

Mime
View raw message