hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Subramanian <Sanjay.Subraman...@wizecommerce.com>
Subject Re: Eclipse plugin
Date Thu, 23 May 2013 01:08:41 GMT
Excerpts from a Development Arch doc I had created
2 Architecture

We will use Core Java , MySQL, Hadoop, SVN, Junit, Maven.

For webservices will use Spring MVC & JBOSS.

2.1 Hardware Requirements

Either Mac/Snow Leopard(or higher)  or PC/Laptop/Windows

8GB RAM recommended

2 X 500GB disc recommended

2.2 Software Requirements

For Windows 64bit

Java

Java, java version "1.6.0_29"

Java(TM) SE Runtime Environment (build 1.6.0_29-b11)

Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)



Eclipse - version eclipse-SDK-3.6.2-win32-x86_64 http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-win32-x86_64.zip

Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.

You can, for example, use the Sun 64-bit 1.5 JVM for AMD64.  Note

that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be

used to run this build.



apache-maven-3.0.3 : http://archive.apache.org/dist/maven/binaries/apache-maven-3.0.3-bin.zip

add maven bin to the path



q4e maven plugin  - http://q4e.googlecode.com/svn/trunk/updatesite-iam/

(if q4e does not work then u can use http://www.sonatype.org/m2eclipse/



subclipe 1.8x plugin - http://subclipse.tigris.org/update_1.8.x<http://subclipse.tigris.org/update_1.8.x>



Cygwin : required to run Hadoop on a Windows Box. Add cygwin executable to path



For Mac

Java

same version as mentioned for Windows



Eclipse - version eclipse-SDK-3.6.2

http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-macosx-cocoa-x86_64.tar.gz

Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.



From: Bharati Adkar <bharati.adkar@mparallelo.com<mailto:bharati.adkar@mparallelo.com>>
Date: Wednesday, May 22, 2013 4:37 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>,
Sanjay Subramanian <sanjay.subramanian@wizecommerce.com<mailto:sanjay.subramanian@wizecommerce.com>>
Subject: Re: Eclipse plugin

Hi,
I am using a mac.

I have not used maven before,  I am new to hadoop and eclipse.

Any directions to start a project as map reuce as per all the videos on youtube.

Thanks,
Bharati


On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sanjay.Subramanian@wizecommerce.com<mailto:Sanjay.Subramanian@wizecommerce.com>>
wrote:

Hi

I don't use any need any special plugin to walk thru the code

All my map reduce jobs have a

JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)

I create a new maven project in eclipse (easier to manage dependencies) ….the elements are
in the order as they should appear in the POM

Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging
! That’s it…..


MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>

<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>

WordCountNew (please modify as needed)
======================================

publicclass WordCountNew {



   public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text,
Text, IntWritable> {
     private final static IntWritable one = new IntWritable(1);
     private Text word = new Text();



     public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException
{
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName =  fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
       }
     }
   }



   public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable,
Text, IntWritable> {
     public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws
IOException, InterruptedException {
       int sum = 0;
       for (IntWritable value : values) {
         sum += value.get();
       }
       ctxt.write(key, new IntWritable(sum));
     }
   }



   public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");

     Job job = new Job(hadoopConf);
     job.setJobName("wordcountNEW");
     job.setJarByClass(WordCountNew.class);
     job.setOutputKeyClass(Text.class);
     job.setOutputValueClass(IntWritable.class);
     job.setMapOutputKeyClass(Text.class);
     job.setMapOutputValueClass(IntWritable.class);



     job.setMapperClass(WordCountNew.Map.class);
     job.setCombinerClass(WordCountNew.Reduce.class);
     job.setReducerClass(Reduce.class);



//      job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
     job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);

     job.setOutputFormatClass(TextOutputFormat.class);



     if (FileUtils.doesFileOrDirectoryExist(args[1])){
     org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
     }
     org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
    org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));



     job.waitForCompletion(true);
     System.out.println();
   }
}





From: Bharati <bharati.adkar@mparallelo.com<mailto:bharati.adkar@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Re: Eclipse plugin

Hi Jing,

I want to be able to open a project as map reduce project in eclipse instead of java project
as per some of the videos on youtube.

For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.

The goal here is to setup the development env to start project as mad reduce right in eclipse
or netbeans which ever works better. The idea is to be able to step through the code.

Thanks,
Bharati

Sent from my iPad

On May 22, 2013, at 2:42 PM, Jing Zhao <jing@hortonworks.com<mailto:jing@hortonworks.com>>
wrote:

> Hi Bharati,
>
>    Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bharati.adkar@mparallelo.com<mailto:bharati.adkar@mparallelo.com>>
wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods  to
build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered

CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please contact the sender
by reply email and destroy all copies of the original message along with any attachments,
from your computer system. If you are the intended recipient, please be advised that the content
of this message is subject to access, review and disclosure by the sender's Email System Administrator.


CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient, please contact the sender
by reply email and destroy all copies of the original message along with any attachments,
from your computer system. If you are the intended recipient, please be advised that the content
of this message is subject to access, review and disclosure by the sender's Email System Administrator.

Mime
View raw message