Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 902F0D06F for ; Wed, 22 May 2013 23:24:39 +0000 (UTC) Received: (qmail 38454 invoked by uid 500); 22 May 2013 23:24:34 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 38233 invoked by uid 500); 22 May 2013 23:24:33 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 38226 invoked by uid 99); 22 May 2013 23:24:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 May 2013 23:24:33 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MANY_SPAN_IN_TEXT,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Sanjay.Subramanian@wizecommerce.com designates 213.199.154.188 as permitted sender) Received: from [213.199.154.188] (HELO db8outboundpool.messaging.microsoft.com) (213.199.154.188) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 May 2013 23:24:29 +0000 Received: from mail9-db8-R.bigfish.com (10.174.8.248) by DB8EHSOBE033.bigfish.com (10.174.4.96) with Microsoft SMTP Server id 14.1.225.23; Wed, 22 May 2013 23:24:06 +0000 Received: from mail9-db8 (localhost [127.0.0.1]) by mail9-db8-R.bigfish.com (Postfix) with ESMTP id 7B6E48023E for ; Wed, 22 May 2013 23:24:06 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.232.197;KIP:(null);UIP:(null);IPV:NLI;H:BLUPRD0411HT003.namprd04.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: -3 X-BigFish: PS-3(z569dhz98dI9371I181fMc85eh1432I4015Idd85k9a6kzz1f42h1ee6h1de0h1fdah1202h1e76h1d1ah1d2ah1fc6hzz18c673h8275bh8275dhz2fh2a8h668h839hbe3he5bhf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d0ch1d2eh1d3fh1dc1h1155h) Received-SPF: pass (mail9-db8: domain of wizecommerce.com designates 157.56.232.197 as permitted sender) client-ip=157.56.232.197; envelope-from=Sanjay.Subramanian@wizecommerce.com; helo=BLUPRD0411HT003.namprd04.prod.outlook.com ;.outlook.com ; Received: from mail9-db8 (localhost.localdomain [127.0.0.1]) by mail9-db8 (MessageSwitch) id 136926504382833_5496; Wed, 22 May 2013 23:24:03 +0000 (UTC) Received: from DB8EHSMHS011.bigfish.com (unknown [10.174.8.236]) by mail9-db8.bigfish.com (Postfix) with ESMTP id 108CB380093 for ; Wed, 22 May 2013 23:24:03 +0000 (UTC) Received: from BLUPRD0411HT003.namprd04.prod.outlook.com (157.56.232.197) by DB8EHSMHS011.bigfish.com (10.174.4.21) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 22 May 2013 23:24:02 +0000 Received: from BLUPRD0411MB426.namprd04.prod.outlook.com ([169.254.10.213]) by BLUPRD0411HT003.namprd04.prod.outlook.com ([10.255.127.38]) with mapi id 14.16.0311.000; Wed, 22 May 2013 23:23:58 +0000 From: Sanjay Subramanian To: "user@hadoop.apache.org" Subject: Re: Eclipse plugin Thread-Topic: Eclipse plugin Thread-Index: AQHOVzMPZMshpyZtKkOFXmP/RL7yU5kRvBGAgAAP+4D//5cOAA== Date: Wed, 22 May 2013 23:23:56 +0000 Message-ID: In-Reply-To: <8E697649-7AFE-4CE7-9692-15794C96FCFE@mparallelo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.255.127.4] Content-Type: multipart/alternative; boundary="_000_CDC2A0187AC1sanjaysubramanianwizecommercecom_" MIME-Version: 1.0 X-OriginatorOrg: wizecommerce.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_CDC2A0187AC1sanjaysubramanianwizecommercecom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi I don't use any need any special plugin to walk thru the code All my map reduce jobs have a JobMapper.java JobReducer.java JobProcessor.java (set any configs u like) I create a new maven project in eclipse (easier to manage dependencies) =85= .the elements are in the order as they should appear in the POM Then In Eclipse Debug Configurations I create a new JAVA application and th= en I start debugging ! That=92s it=85.. MAVEN REPO INFO =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Cloudera repository https://repository.cloudera.com/artifactory/cloudera-repos/ 2.0.0-cdh4.1.2 org.apache.hadoop hadoop-mapreduce-client-core ${cloudera_version} compile org.apache.hadoop hadoop-common ${cloudera_version} compile org.apache.hadoop hadoop-client ${cloudera_version} compile org.apache.hadoop hadoop-client ${cloudera_version} compile WordCountNew (please modify as needed) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D public class WordCountNew { public static class Map extends org.apache.hadoop.mapreduce.Mapper { private final static IntWritable one =3D new IntWritable(1); private Text word =3D new Text(); public void map(LongWritable key, Text value, Context ctxt) throws IO= Exception, InterruptedException { FileSplit fileSplit =3D (FileSplit)ctxt.getInputSplit(); // System.out.println(value.toString()); String fileName =3D fileSplit.getPath().toString(); String line =3D value.toString(); StringTokenizer tokenizer =3D new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); ctxt.write(word, one); } } } public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<= Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable values, Context ct= xt) throws IOException, InterruptedException { int sum =3D 0; for (IntWritable value : values) { sum +=3D value.get(); } ctxt.write(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { org.apache.hadoop.conf.Configuration hadoopConf =3D new org.apache.hadoop.c= onf.Configuration(); hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), Mapre= dConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal()); hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal()= , MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal()); hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.Gzip= Codec"); Job job =3D new Job(hadoopConf); job.setJobName("wordcountNEW"); job.setJarByClass(WordCountNew.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setMapperClass(WordCountNew.Map.class); job.setCombinerClass(WordCountNew.Reduce.class); job.setReducerClass(Reduce.class); // job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class); job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInp= utFormat.class); job.setOutputFormatClass(TextOutputFormat.class); if (FileUtils.doesFileOrDirectoryExist(args[1])){ org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1])); } org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(j= ob, new Path(args[0])); org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(j= ob, new Path(args[1])); job.waitForCompletion(true); System.out.println(); } } From: Bharati > Reply-To: "user@hadoop.apache.org" > Date: Wednesday, May 22, 2013 3:39 PM To: "user@hadoop.apache.org" > Subject: Re: Eclipse plugin Hi Jing, I want to be able to open a project as map reduce project in eclipse instea= d of java project as per some of the videos on youtube. For now let us say I want to write a wordcount program and step through it = with hadoop 1.2.0 How can I use eclipse to rewrite the code. The goal here is to setup the development env to start project as mad reduc= e right in eclipse or netbeans which ever works better. The idea is to be a= ble to step through the code. Thanks, Bharati Sent from my iPad On May 22, 2013, at 2:42 PM, Jing Zhao > wrote: > Hi Bharati, > > Usually you only need to run "ant clean jar jar-test" and "ant > eclipse" on your code base, and then import the project into your > eclipse. Can you provide some more detailed description about the > problem you met? > > Thanks, > -Jing > > On Wed, May 22, 2013 at 2:25 PM, Bharati > wrote: >> Hi, >> >> I am trying to get or build eclipse plugin for 1.2.0 >> >> All the methods I found on the web did not work for me. Any tutorial, me= thods to build the plugin will help. >> >> I need to build a hadoop map reduce project and be able to debug in ecli= pse. >> >> Thanks, >> Bharati >> Sent from my iPad >> Fortigate Filtered >> Fortigate Filtered CONFIDENTIALITY NOTICE =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply ema= il and destroy all copies of the original message along with any attachment= s, from your computer system. If you are the intended recipient, please be = advised that the content of this message is subject to access, review and d= isclosure by the sender's Email System Administrator. --_000_CDC2A0187AC1sanjaysubramanianwizecommercecom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <269826D8502F044AA3D3C99C814A9A36@namprd04.prod.outlook.com> Content-Transfer-Encoding: quoted-printable
Hi 

I don't use any need any special plugin to = walk thru the code 

All my map reduce jobs have a 

JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)<= /div>

I create a new maven project in eclipse (ea= sier to manage dependencies) =85.the elements are in the order as they shou= ld appear in the POM

Then In Eclipse Debug Configurations I crea= te a new JAVA application and then I start debugging ! That=92s it=85..


MAVEN REPO INFO
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D

<reposito= ries>

<reposito= ry>

<id>Cloudera repository</id>=

<url>https://repository.cloudera.com/artifac= tory/cloudera-repos/</url>

</reposit= ory>

</reposit= ories>


<properti= es>

<cloudera= _version>2.0.0-cdh4.1.2</cloudera= _version>

</propert= ies>


<de= pendency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-mapreduce-c= lient-core</artifactId>

<version>${cloudera_version}</version>

<scope>compile</scope>

</d= ependency>

<de= pendency>

<groupId>org.apache.hadoop</groupId>

<ar= tifactId>hadoop-= common</artifactId>

<version>${cloudera_version}</version>

<scope>compile</scope>

</d= ependency>

<de= pendency>

<groupId>org.apache.hadoop</groupId>

<ar= tifactId>hadoop-= client</artifactId>

<version>${cloudera_version}</version>

<scope>compile</scope>

</d= ependency>

<de= pendency>

<groupId>org.apache.hadoop</groupId>

<ar= tifactId>hadoop-= client</artifactId>

<version>${cloudera_version}</version>

<scope>compile</scope>

</d= ependency>


Wor= dCountNew (please modify as needed)
=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


public class WordCountNew {

 

    public static class Map = extends org.apache.hadoop.mapreduce.Mapper<LongWritable, T= ext, Text, IntWritable> {

      private final static IntW= ritable one =3D new IntWr= itable(1);

      private Te= xt word =3D new Text();

 

      public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {

FileSplit fileSplit = =3D (FileSplit)ctxt.getInputSplit();

// System.o= ut.println(value.toString());

String fileName =3D  fileSpl= it.getPath().toString();

String line =3D value.= toString();

StringTokenizer tokeni= zer =3D new StringTokenizer(line);

while (tokenizer.hasMoreTokens()) {

word.set(tokenizer.nextToken());

ctxt.write(word, one);

        }

      }

    }

 

    public static class Redu= ce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWrita= ble, Text, IntWritable> {

      public void reduce(Text key, Iterable<IntWritable> values, Con= text ctxt) throws IOException, InterruptedException {

        int= sum =3D 0;

        for= (IntWritable value : values) {

          sum +=3D value.get();

        }

        ctxt.write(key, <= b>new IntWritable(sum));

      }

    }

 

    public static void main(= String[] args) throws Exception {

org.apache.hadoop.conf= .Configuration hadoopConf =3D new org.apache.hadoop.conf.Conf= iguration();

hadoopConf.set(MapredC= onfEnum.IMPRESSIONS_LOG_REC_SEPARATOR<= /span>.getVal(), MapredConfEnum.PRODUCT_IM= PR_LOG_REC_END.getVal());

hadoopConf.set(MapredC= onfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARAT= OR.getVal(), MapredConfEnum.PRO= DUCT_IMPR_LOG_REC_CACHED.getVal());

hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");


      Job job =3D new Job(hadoopConf);

      job.setJobName("wor= dcountNEW");

      job.setJarByClass(WordCountNew.class);

      job.setOutputKeyClass(Text.class);

      job.setOutputValueClass(IntWritable.class);

      job.setMapOutputKeyClass(Text.class);

      job.setMapOutputValueClass(IntWritable.class);

 

      job.setMapperClass(WordCountNew.Map.class);

      job.setCombinerClass(WordCountNew.Reduce.class);

      job.setReducerClass(Reduce.class);

     

//       job.setInputFormatClass(ZipMultipleLineRecordInputForm= at.class);

      job.setInputFormatClass(org.apache.hadoop.mapreduce.li= b.input.TextInputFormat.class);=


      job.setOutputFormatClass(TextOutputFormat.class);

     

      if (FileUt= ils.doesFileOrDirectoryExist(args[1])){

      org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));

      }

      org.apache.hadoop.mapreduce.lib.input.FileInputFormat.= setInputPaths(job, new Path(args[0]));

    org.apac= he.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));

 

      job.waitForCompletion(true);

      System.out= .println();

    }

}

 




From: Bharati <bharati.adkar@mparallelo.com>
Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Date: Wednesday, May 22, 2013 3:39 = PM
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Re: Eclipse plugin

Hi Jing,

I want to be able to open a project as map reduce project in eclipse instea= d of java project as per some of the videos on youtube. 

For now let us say I want to write a wordcount program and step through it = with hadoop 1.2.0
How can I use eclipse to rewrite the code.

The goal here is to setup the development env to start project as mad reduc= e right in eclipse or netbeans which ever works better. The idea is to be a= ble to step through the code.

Thanks,
Bharati

Sent from my iPad

On May 22, 2013, at 2:42 PM, Jing Zhao <jing@hortonworks.com> wrote:

> Hi Bharati,
>
>    Usually you only need to run "ant clean jar jar= -test" and "ant
> eclipse" on your code base, and then import the project into your=
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bharati.adkar@mparallelo.com> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutori= al, methods  to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug i= n eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered

CONFIDENTIALITY NOTICE
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply email and destroy all copies of the ori= ginal message along with any attachments, from your computer system. If you= are the intended recipient, please be advised that the content of this mes= sage is subject to access, review and disclosure by the sender's Email System Administrator.
--_000_CDC2A0187AC1sanjaysubramanianwizecommercecom_--