hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Sripati <praveensrip...@gmail.com>
Subject Re: MRv2 Configuration in Eclipse
Date Mon, 15 Aug 2011 14:29:19 GMT
Hi,

I did spend some time on trying to run RM, NM and HS from within Eclipse,
but got partial success. Here are the steps I followed (just in case someone
is interested)

- Got the code from SVN.

- Ran the ant and mvn command for compiling Hadoop Common, HDFS and
MapReduce.

- Ran the 'mvn eclipse:eclipse' and 'ant eclipse' for the projects.

- Included the following in /etc/environment (a reboot was required for it
to take effect)

HADOOP_MAPRED_HOME="/home/praveensripati/Hadoop/trunk/mapreduce"
HADOOP_COMMON_HOME="/home/praveensripati/Hadoop/trunk/hadoop-common"
HADOOP_HDFS_HOME="/home/praveensripati/Hadoop/trunk/hdfs"
YARN_HOME="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
HADOOP_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_INSTALL="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
JAVA_HOME="/usr/lib/jvm/java-6-sun"

- Imported the java projects in Eclipse.

- In the 'Java Build Path' removed the jar dependencies in the Libraries Tab
and included the corresponding projects in the Sources Tab. Made sure that
the projects got compiled properly.

- Started the resourcemanager, nodemanager and historyserver from command
prompt using the yarn-daemon.sh command. This dumps out the java command
with the arguments. Stopped the resourcemanager, nodemanager and
historyserver again from the command propmpt.

- Ran the ResourceManager, NodeManager and JobHistoryServer from Eclipse,
passing the parameters got from the previous step.

- Created a Java project with Eclipse and copied
org.apache.hadoop.examples.RandomWriter.java and ran with the following
parameters

    -Dmapreduce.job.user.name=praveensripati
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912
/home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin/output

and got the following exceptions for each of the mapper invocation. Could
not figure out the cause of this exception.

2011-08-13 21:44:35,785 INFO  mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
    at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
    ... 8 more

- But was able to run the RandomWriter from command propmpt using the below
command

./hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output

Thanks,
Praveen

On Sat, Aug 13, 2011 at 10:40 PM, Praveen Sripati
<praveensripati@gmail.com>wrote:

>
> Hi,
>
> I am able to run the RM, NM and HS from Eclipse with a couple of settings
> and run the org.apache.hadoop.examples.RandomWriter from command promptusing the 'hadoop
jar' command. Would be nice for easier development and
> debugging of these components.
>
> But, when I run the org.apache.hadoop.examples.RandomWriter from Eclipse, I
> get the below exceptions. I created a new Java project in Eclipse and copied
> the RandomWriter.java to it. This exception is getting repeated map number
> of times, looks like the error is thrown for every map launch.
>
> Not sure what this error means, any help is appreciated.
>
> 2011-08-13 21:44:35,785 INFO  mapreduce.Job
> (Job.java:printTaskEvents(1213)) - Task Id :
> attempt_1313250752016_0003_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.RandomWriter$RandomMapper
>     at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
>     at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.examples.RandomWriter$RandomMapper
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:247)
>     at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
>     at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
>     ... 8 more
>
> Thanks,
> Praveen
>
> On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <acm@hortonworks.com>wrote:
>
>> Praveen,
>>
>> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
>> myself.
>>
>>  OTOH, if you do get to we are happy to follow your lead, please feel free
>> to provide us instructions and we can update the README... thanks.
>>
>> Arun
>>
>> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>>
>> > Hi,
>> >
>> > Are there any instructions to start the ResourceManager, NodeManager,
>> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
>> the
>> > sample program. The projects have also been exported in Eclipse. I want
>> to
>> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
>> > problems in the code.
>> >
>> > Thanks,
>> > Praveen
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message