Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4EE5F86D1 for ; Tue, 13 Sep 2011 09:38:07 +0000 (UTC) Received: (qmail 2698 invoked by uid 500); 13 Sep 2011 09:38:05 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 2655 invoked by uid 500); 13 Sep 2011 09:38:05 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 2630 invoked by uid 99); 13 Sep 2011 09:38:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Sep 2011 09:38:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bejoy.hadoop@gmail.com designates 209.85.216.176 as permitted sender) Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Sep 2011 09:37:56 +0000 Received: by qyk36 with SMTP id 36so239515qyk.14 for ; Tue, 13 Sep 2011 02:37:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=/AiyqyXQPGJbFy+mv+Xu0OPwe4lK+GgcdW+tj7VGbqY=; b=AmSa2aOQ7zEmXPSyYhV3Qo0WKATrdxeNvdW/CEE2/uNP89cxE8515gVPfvPIZfgLiX AP/RNz31vWkbu+m+7yjCqjdJrTSqtwmPAo1vzDUB6ReanHtwh9ri9/Xj2Z0xIw+RxAuy YM4kzWvjkbymkjrfZ6H4PBAkxAKFVrhv4XTkc= MIME-Version: 1.0 Received: by 10.229.74.67 with SMTP id t3mr2624024qcj.116.1315906655445; Tue, 13 Sep 2011 02:37:35 -0700 (PDT) Received: by 10.229.182.138 with HTTP; Tue, 13 Sep 2011 02:37:35 -0700 (PDT) In-Reply-To: References: <2DF48A15212FD646BE51A3FAC310F981020A87E5@ctsinpunsxuf.cts.com> <2DF48A15212FD646BE51A3FAC310F981020A87E6@ctsinpunsxuf.cts.com> <2DF48A15212FD646BE51A3FAC310F981020A87E7@ctsinpunsxuf.cts.com> <2DF48A15212FD646BE51A3FAC310F981023AC3D0@ctsinpunsxuf.cts.com> Date: Tue, 13 Sep 2011 15:07:35 +0530 Message-ID: Subject: Re: Issues starting TaskTracker From: Bejoy KS To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00163646cf80c162b104accf64c1 --00163646cf80c162b104accf64c1 Content-Type: text/plain; charset=ISO-8859-1 Shreya To add on. From cloudera website you would get images for different VMs like VM Ware, Virtual Box etc. Choose the appropriate one for your use as per your availabe software. To your question, it is definitely possible to run map reduce progarms from Cloudera VM and in fact it is the most comfortable way(at least for me) to test my map reduce code. When you are on cloudera VM to test your plain map reduce code in fact you don't even need to pack your source code into jar, deploy the same and then execute it.(now you might be doing development in windows and deployment and test in linux) To test your code just follow the sequence of steps - Download and install eclipse on the VM (any IDE you are on) - Create your project with Mapper, Reducer and Driver classes (may be a single file also would be fine as per your convenience) - Click on the class that contains your main method, give run as java application - It'd do the job for you. The few things you need to keep in mind are - Use very minimal test data. Larger data volumes would lead to very slow execution due to limited resources.(just use VM to test the logic) - Normally in our driver class we get the input and output directory from command line when we deploy as jars and run the same, but when you run from eclipse just alter the lines of code specifying input and output as For input and output directory in hdfs FileInputFormat.addInputPath(job, new Path("hdfs://localhost/")); FileOutputFormat.setOutputPath(job, new Path(hdfs://localhost/")); For input and output directory in lfs FileInputFormat.addInputPath(job, new Path("")); FileOutputFormat.setOutputPath(job, new Path("")); Hope it helps Regards Bejoy.K.S On Tue, Sep 13, 2011 at 2:40 PM, Bejoy KS wrote: > Hi Shreya > You can copy files from windows to the linux on VM using any ftp > tools like filezilla. > Take a terminal on your linix, type ifconfig , the value given under 'inet > addr:' would be your IP address. > Use this IP address and default port (22) to connect to liux image from > Windows through filezilla. The Cloudera VM has the user name and password as > 'cloudera' itself. > > Hope It helps > > Regards > Bejoy.KS > > > On Tue, Sep 13, 2011 at 2:18 PM, wrote: > >> >> Hi Harsh, >> >> Version of Hadoop - hadoop-0.20.203.0 >> How do I make the process owner same as directory owner >> Directory owner is - Titun >> >> Regards >> Shreya >> >> -----Original Message----- >> From: Harsh J [mailto:harsh@cloudera.com] >> Sent: Monday, September 12, 2011 10:50 PM >> To: mapreduce-user@hadoop.apache.org >> Subject: Re: Issues starting TaskTracker >> >> Shreya, >> >> > I was getting the message owner SYSTEM when I was using default >> > I was getting the message - running as TITUN, but the same error >> >> What user are you actually launching the TaskTracker as? The directory >> owner (user) must be == process owner (user) of the TT, and things >> should be fine! Can you confirm that this isn't the case? What version >> of Hadoop are you using? >> >> P.s. Am really beginning to dislike MS Exchange or your sysadmin's >> mailer settings here :-) >> >> On Mon, Sep 12, 2011 at 9:23 PM, wrote: >> > This e-mail and any files transmitted with it are for the sole use of >> the intended recipient(s) and may contain confidential and privileged >> information. >> > If you are not the intended recipient, please contact the sender by >> reply e-mail and destroy all copies of the original message. >> > Any unauthorised review, use, disclosure, dissemination, forwarding, >> printing or copying of this email or any action taken in reliance on >> this e-mail is strictly >> > prohibited and may be unlawful. >> >> >> >> -- >> Harsh J >> >> This e-mail and any files transmitted with it are for the sole use of the >> intended recipient(s) and may contain confidential and privileged >> information. >> If you are not the intended recipient, please contact the sender by reply >> e-mail and destroy all copies of the original message. >> Any unauthorised review, use, disclosure, dissemination, forwarding, >> printing or copying of this email or any action taken in reliance on this >> e-mail is strictly >> prohibited and may be unlawful. >> > > --00163646cf80c162b104accf64c1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Shreya
=A0=A0=A0=A0=A0=A0 To add on. From cloudera website you would get= images for different VMs like VM Ware, Virtual Box etc. Choose the appropr= iate one for your use as per your availabe software.
=A0=A0=A0=A0=A0 To= your question, it is definitely possible to run map reduce progarms from C= loudera VM and in fact it is the most comfortable way(at least for me) to t= est my map reduce code. When you are on cloudera VM to test your plain map = reduce code in fact you don't even need to pack your source code into j= ar, deploy the same and then execute it.(now you might be doing development= in windows and deployment and test in linux) To test your code just follow= the sequence of steps
  • Download and install eclipse on the VM (any IDE you are on)
    • Create your project with Mapper, Reducer and Driver classes (may be a sing= le file also would be fine as per your convenience)
  • Click on th= e class that contains your main method, give run as java application
  • It'd do the job for you.
The few things you need to keep i= n mind are
  • Use very minimal test data. Larger data volumes would= lead to very slow execution due to limited resources.(just use VM to test = the logic)
  • Normally in our driver class we get the input and output directory from= command line when we deploy as jars and run the same, but when you run fro= m eclipse just alter the lines of code specifying input and output as
=A0=A0=A0=A0=A0=A0 For input and output directory in hdfs
=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 FileInputFormat.addInputPath(job, new Path(&= quot;hdfs://localhost/<full path in hdfs>"));
=A0=A0=A0 =A0= =A0=A0=A0=A0=A0=A0 FileOutputFormat.setOutputPath(job, new Path(hdfs://loca= lhost/<full path in hdfs>"));

=A0=A0=A0=A0=A0=A0 For input and output directory in lfs
=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 FileInputFormat.addInputPath(job, new Path("&= lt;full path in lfs>"));
=A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0 FileOutputFormat.setOutputPath(job, new Pat= h("<full path in lfs>"));

Hope it helps

Regar= ds
Bejoy.K.S


On Tue, Sep 13, 2011 = at 2:40 PM, Bejoy KS <bejoy.hadoop@gmail.com> wrote:
Hi Shreya
=A0=A0=A0=A0=A0=A0=A0=A0 You c= an copy files from windows to the linux on VM using any ftp tools like file= zilla.
Take a terminal on your linix, type ifconfig , the value given un= der 'inet addr:'=A0 would be your IP address.
Use this IP address and default port (22) to connect to liux image from Win= dows through filezilla. The Cloudera VM has the user name and password as &= #39;cloudera' itself.

Hope It helps

Regards
Bejoy.KS


On Tue, Sep 13, 2011 at 2:18 PM, <= Shreya.Pal@cognizant.com> wrote:

Hi Harsh,

Version of Hadoop - hadoop-0.20.203.0
How do I make the process owner same as directory owner
Directory owner is - Titun

Regards
Shreya

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Monday, September 12, 2011 10:50 PM
To: m= apreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker

Shreya,

> I was getting the message owner SYSTEM when I was using default
> I was getting the message - running as TITUN, but the same error

What user are you actually launching the TaskTracker as? The directory
owner (user) must be =3D=3D process owner (user) of the TT, and things
should be fine! Can you confirm that this isn't the case? What version<= br> of Hadoop are you using?

P.s. Am really beginning to dislike MS Exchange or your sysadmin's
mailer settings here :-)

On Mon, Sep 12, 2011 at 9:23 PM, =A0<Shreya.Pal@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of<= br> the intended recipient(s) and may contain confidential and privileged
information.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on
this e-mail is strictly
> prohibited and may be unlawful.



--
Harsh J

This e-mail and any files transmitted with it are for the sole use of the i= ntended recipient(s) and may contain confidential and privileged informatio= n.
If you are not the intended recipient, please contact the sender by reply e= -mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printi= ng or copying of this email or any action taken in reliance on this e-mail = is strictly
prohibited and may be unlawful.


--00163646cf80c162b104accf64c1--